OpenInfer ← Benchmark

Benchmark telemetry

Sellable throughput under sustained load

Identical open-loop load driven at both fleets in parallel — a dedicated one-model-per-node baseline vs. OpenInfer co-locating every model on each node — measuring the tokens delivered within the latency SLA.

3.3×

more sellable tokens than baseline — 1.1M vs 351k delivered within the latency SLA

Latest run · 2026-09-01 20:17 UTC

Run summary avg 4.0× min 1.7× max 11.1× 83 runs 2026-08-01 → 2026-09-01

Results

Sellable tok/s

Baseline
2.9k
486.9/node
OpenInfer
9.5k
1.6k/node
Delta
3.3×

Sellable tokens

Baseline
351k
58.4k/node
OpenInfer
1.1M
190k/node
Delta
3.3×

Peak fleet GPU (mean of every GPU)

Baseline
32%
OpenInfer
98%
Delta
+66 pts

Accounting

Where every request went

Each request the arm sent is in exactly one column. Within SLA is the headline's own test over every request sent; the headline counts those finished by its second. Pace-exempt counts the requests within SLA only because a Medium or Low replica served them: the tokens-a-second pace binds only High replicas. Tier unknown counts the served requests that named no replica, held to the pace. Check is ✓ when sent plus unsent is what the schedule planned.

Baseline

Model Planned Sent Within SLA Pace-exempt Tier unknown Late Runtime 429 Weave 429 Hub 503 Timed out Client / killed Other Unsent Check
Qwen3.5-27B None None None None None None None None None None None None None None
Qwen3.5-9B None None None None None None None None None None None None None None
Llama-3.2-1B-Instruct None None None None None None None None None None None None None None
Phi-4-14B None None None None None None None None None None None None None None
Gemma4-12B-IT None None None None None None None None None None None None None None
Qwen3-1.7B None None None None None None None None None None None None None None
Total None None None None None None None None None None None None None None

OpenInfer

Model Planned Sent Within SLA Pace-exempt Tier unknown Late Runtime 429 Weave 429 Hub 503 Timed out Client / killed Other Unsent Check
Qwen3.5-27B None None None None None None None None None None None None None None
Qwen3.5-9B None None None None None None None None None None None None None None
Llama-3.2-1B-Instruct None None None None None None None None None None None None None None
Phi-4-14B None None None None None None None None None None None None None None
Gemma4-12B-IT None None None None None None None None None None None None None None
Qwen3-1.7B None None None None None None None None None None None None None None
Total None None None None None None None None None None None None None None

Requests completed within SLA

cumulative
Baseline OpenInfer in flight (right axis)

No request completed within SLA, or this report predates the count

Active concurrency over time

in-flight
Baseline OpenInfer
0 200 400 600 800 0s 20s 40s 60s 80s 100s 120s

Throughput over time

tok/s
Baseline OpenInfer active concurrency
0 10k 20k 30k 40k 0s 20s 40s 60s 80s 100s 120s 0 200 400 600 800

Fleet GPU utilization over time

% util, mean of every GPU
Baseline OpenInfer active concurrency
0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s 0 200 400 600 800

Error rate over time

% of sent requests, max_inflight drops excluded
Baseline OpenInfer

This report predates the error-rate series

GPU utilization per node over time

% util, mean of the node's GPUs
Baseline OpenInfer

ip-172-31-15-42

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-17-118

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-17-58

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-22-222

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-23-139

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-24-132

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-25-131

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-29-119

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-30-205

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-30-213

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-4-159

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

ip-172-31-6-46

0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s

Per model · TTFT p50 / decode tok/s per request

Model Baseline TTFT p50 Baseline decode tok/s per request OpenInfer TTFT p50 OpenInfer decode tok/s per request
Qwen3.5-27B 4.5s 10 1.5s 26.9
Qwen3.5-9B 799ms 42.2 465ms 75.5
Llama-3.2-1B-Instruct 2.4s 135.8 1.2s 127.5
Phi-4-14B 2.4s 39.7 2.2s 45.7
Gemma4-12B-IT 760ms 48.5 605ms 55.4
Qwen3-1.7B 1.3s 130.8 532ms 166.2

Results

Totals and per node

Metric Baseline total Per node OpenInfer total Per node
Total tokens 808k 135k 1.4M 232k
Aggregate tok/s 6.7k 1.1k 11.6k 1.9k
Requests (steps) 16,598 2,766 7,058 1,176
Over capacity 12,557 75.7% 100 1.4%

Latency & throughput

Per model · p95 / tok-s / over-capacity

Model Baseline p95 / tps / over-cap OpenInfer p95 / tps / over-cap
Qwen3.5-27B 5.7s / 10 / 95.4% 3.9s / 26.9 / 0%
Qwen3.5-9B 1.1s / 42.2 / 0% 2.1s / 75.5 / 0%
Llama-3.2-1B-Instruct 3.9s / 135.8 / 30.3% 3.1s / 127.5 / 0%
Phi-4-14B 5.8s / 39.7 / 70.7% 4.9s / 45.7 / 4.8%
Gemma4-12B-IT 910ms / 48.5 / 94.4% 2.6s / 55.4 / 3.4%
Qwen3-1.7B 1.8s / 130.8 / 25.3% 1.2s / 166.2 / 0%

Load balance

Requests & utilization per model

Requests admitted, Served (2xx), and GPU aggregated across the cards serving each model, for each arm the run fired. ×N copies is how many routable replicas of the model the nodes held while the arm ran; a pinned baseline arm sends each model's requests to its High replica only. Raw request counts are dominated by fast rejections under overload, so compare Served, not Requests.

Model Requests Served Over cap Tokens Avg GPU % Peak GPU %
Baseline Qwen3.5-27B 5,066 233 95.4% 46.6k 21.7 100
Baseline Qwen3.5-9B 384 384 0% 76.8k 14.5 97
Baseline Llama-3.2-1B-Instruct 1,980 1,380 30.3% 276k 13.2 89
Baseline Phi-4-14B 1,780 521 70.7% 104k 14.4 97
Baseline Gemma4-12B-IT 5,782 323 94.4% 64.6k 17.2 97
Baseline Qwen3-1.7B 1,606 1,200 25.3% 240k 13.1 89
OpenInfer Qwen3.5-27B ×6 copies 637 637 0% 127k 81.2 99
OpenInfer Qwen3.5-9B ×6 copies 607 607 0% 121k 81.2 99
OpenInfer Llama-3.2-1B-Instruct ×6 copies 1,920 1,920 0% 384k 81.2 99
OpenInfer Phi-4-14B ×6 copies 1,700 1,618 4.8% 324k 81.2 99
OpenInfer Gemma4-12B-IT ×6 copies 530 512 3.4% 102k 81.2 99
OpenInfer Qwen3-1.7B ×6 copies 1,664 1,664 0% 333k 81.2 99