# MetriLLM

> Open-source CLI tool and leaderboard for benchmarking local LLM inference on real hardware. Measures speed (tokens/s, TTFT), memory usage, and response quality (reasoning, math, coding, instruction following, structured output, multilingual). MIT license, npm package.

## Quick Start

Install and run (requires Node 20+ and Ollama running):

```
npm install -g metrillm@latest
metrillm bench
```

Also available via:
- `pnpm add -g metrillm@latest`
- `bun add -g metrillm@latest`
- `brew install metrillm`

Or run without installing: `npx metrillm@latest`

Results are displayed locally and can optionally be shared to the public leaderboard with `--share`.

## What MetriLLM Measures

### Performance (Hardware Fit Score, 0–100)

- **Tokens/sec**: Generation throughput
- **TTFT**: Time to first token (responsiveness)
- **Memory usage**: Host RAM consumed during inference

Weights: 50% Speed + 20% TTFT + 30% Memory efficiency.

### Quality (Response Accuracy Score, 0–100)

14 prompts across 6 categories, auto-evaluated:

- **Reasoning** (20 pts): Logical deduction and analysis
- **Coding** (20 pts): Code generation and problem solving
- **Instruction Following** (20 pts): Constraint adherence
- **Structured Output** (15 pts): JSON/format compliance
- **Math** (15 pts): Numerical computation accuracy
- **Multilingual** (10 pts): Non-English language handling

### Global Score

```
Global = 30% × Hardware Fit + 70% × Quality
```

### Verdict System

- **EXCELLENT**: Global score >= 80
- **GOOD**: Global score >= 60
- **MARGINAL**: Global score >= 40
- **NOT RECOMMENDED**: Global score < 40

### Hardware Profiles

MetriLLM auto-detects hardware tier and adjusts scoring thresholds:

| Metric | Entry (≤8 GB) | Balanced (8–32 GB) | High-End (>32 GB) |
|---|---|---|---|
| Speed (excellent) | 20 tok/s | 30 tok/s | 45 tok/s |
| Speed (minimum) | 3 tok/s | 5 tok/s | 6 tok/s |
| TTFT (excellent) | 1.5 s | 1.0 s | 0.7 s |
| TTFT (maximum) | 20 s | 15 s | 12 s |
| Load time max | 4 min | 3 min | 2 min |

### Disqualification Rules

Automatic NOT RECOMMENDED (score 0) if:
- Token throughput below hardware profile minimum
- TTFT exceeds hardware profile maximum
- Model loading exceeds maximum load time
- Memory delta exceeds 90% of total RAM, or host usage exceeds 95% with model delta >= 10%

### Time Penalty

If a response takes longer than the category time limit (30s, or 90s for coding), the answer is penalized in the quality score.

## Transparency Principles

- **Open Source**: All scoring logic publicly available on GitHub
- **Reproducible**: Same model + same hardware = same results, each run includes a hash
- **No Fine-Tuning**: Models tested as-is from Ollama, no custom prompts
- **No Cherry-Picking**: Every submitted result is displayed without curation

## Thinking Model Detection

MetriLLM detects "thinking" models (e.g., Qwen3) that generate internal reasoning tokens before the actual response. These models are flagged with a THINK badge. Thinking tokens are estimated and reported separately, as they significantly affect tokens/s measurements and benchmark duration.

## Key Pages

- [Leaderboard](https://metrillm.dev/): Live rankings of all benchmark results
- [Methodology](https://metrillm.dev/methodology): Full scoring methodology
- [Specification](https://metrillm.dev/specification): Benchmark data format
- [Models](https://metrillm.dev/models): Individual model pages with stats
- [Hardware](https://metrillm.dev/hardware): Per-CPU benchmark breakdowns
- [Blog](https://metrillm.dev/blog): Articles and analysis
- [Compare](https://metrillm.dev/compare): Side-by-side model comparison
- [Contributors](https://metrillm.dev/contributors): Community contributors
- [My Benchmarks](https://metrillm.dev/my-benchmarks): Personal results lookup

## Source Code

- CLI: https://github.com/MetriLLM/metrillm
- Website: https://github.com/MetriLLM/metrillm-web
- License: MIT