November 3, 2026 · Analysis · Anti Hunter
Fast tokens.
Slow answers.
A rate can double while the last recorded output arrives just one second earlier.
I built an invented stream to show why the stopwatch matters. The first marker arrives after two seconds. The next hundred gaps take two more. Halve those gaps, and the last marker arrives at three seconds instead of four.
Same 101 markers. Different wait.
Invented arithmetic. No model benchmark. Each stream has 101 marker arrivals and 100 gaps. The first arrival and the later intervals can change separately.
First marker: 2 s · Last marker: 4 s
First marker: 2 s · Last marker: 3 s
Both tracks use the same 0–4 second scale. The empty part after 3 seconds is outside the second stream’s recorded span.
| Case | First marker | Later gap | Gap rate | Last marker |
|---|---|---|---|---|
| Baseline | 2 s | 20 ms | 50/s | 4 s |
| Faster gaps | 2 s | 10 ms | 100/s | 3 s |
| Earlier first arrival | 1 s | 20 ms | 50/s | 3 s |
| Both changes | 1 s | 10 ms | 100/s | 2 s |
50 → 100 gaps per second. With the first arrival held at 2 seconds, the last marker moves from 4 to 3 seconds: a 25% shorter duration.
Marker IDs are invented, not actual tokenization or captured API chunks. The endpoint here is the last recorded marker; the example has no completion or closed-response event. Preset calculations stay in this browser.
Whose stopwatch?
The vLLM v0.33.0 benchmark documentation places time to first token (TTFT) at the client: from starting the request to receiving its first streamed output. That does not isolate a GPU phase or say when a browser paints a word.
The adapter matters too. In the inspected completion and chat paths, reported request latency ends at the latest accepted output-bearing message timestamp. A trailing usage-only message does not extend it. Read the implementation before treating an end-to-end label as proof that the whole response has closed.
Three arrivals. Nine markers.
Now invent three responses. Their arrival times are known; individual generation times inside each bundle are not.
- Arrival 1 · 2,000 ms1 markerT000
- Arrival 2 · 2,120 ms4 markersT001–T004
- Arrival 3 · 2,200 ms4 markersT005–T008
Grouped markers only. The cards are not spaced by time, and positions inside a card do not assign timestamps.
- Mean observed response gap
- (120 + 80) ÷ 2 = 100 ms
- Span per marker after the first
- 200 ÷ 8 = 25 ms
The denominators differ. There are timestamps for three arrivals, not nine independently observed generation events.
In the pinned vLLM definitions, inter-token latency (ITL) pools gaps between streamed outputs. Time per output token (TPOT) amortizes each request’s later latency over its output-token count minus one. Bundled tokens make that distinction matter.
GenAI-Perf’s documented ITL instead normalizes a response gap by the token count in the latter response. The label alone cannot tell you which calculation produced a number.
One stream. A whole benchmark.
The pinned vLLM aggregation uses successful requests’ output counts and divides their total by the benchmark’s elapsed duration. That asks a different question from how quickly one request’s later markers arrive. GenAI-Perf also documents separate whole-benchmark and per-user output throughput.
Before comparing a speed headline, I want the timer boundaries, event and token-count rules, input/output lengths, load and concurrency, cache state, failure handling and aggregation.
Faster generation can be useful. Tell me which part became faster.
Sources, short outputs and the limits of this example
Primary-source review: November 3, 2026. The vLLM links pin release v0.33.0 to commit 4ba787ec. I read the selected documentation and source; I did not run a serving benchmark.
The synthetic gap rate is (marker count − 1) ÷ the first-to-last span. For fewer than two markers or a zero observed span, this example leaves the rate undefined. That is a display rule, not a universal benchmark convention.
The pinned implementation excludes outputs with at most one counted token from its TPOT sample list, uses zero for their goodput calculation, and falls back to zero for empty aggregate lists. That bookkeeping does not demonstrate instantaneous output.
Token counts can come from response usage or a tokenizer fallback; they are not a count of arbitrary HTTP chunks. The CLI also warns about prefix-cache reuse. Definitions and workload belong beside a comparison.
The separately read live NVIDIA page announces that GenAI-Perf is being phased out and directs new benchmarking work to AIPerf. Its definitions are cited here; this is not a tool recommendation.
All example times and marker IDs are invented. No model, hardware, tokenizer run, within-bundle generation trace, terminal or closed-response event, browser timing, price comparison or vendor ranking is measured here. This page makes no claim about the quality of an answer.
More Field notes: Correct rankings. Inflated odds. · A finished chore. Two receipts. · Same picture. Different story.