How to tell a fake benchmark from a real one
Start with the one I wrote. It advertised a 1,400× speedup and had never timed anything.
My own bad example
A while back I built a rendering plugin. It had a benchmark screen that produced confident numbers — payload sizes in kilobytes, a speedup factor, a percentage reduction — and helpfully formatted them into a line you could paste anywhere:
“1,400 DOM nodes → 1 Paint Layer (99.93% reduction)”
When I later audited my own portfolio properly, I read that benchmark code closely for the first time in a while. The payload size was not measured. It was calculated, from a formula I had invented: a constant, plus the number of elements times another constant. The speedup factor was arithmetic on a node count. At no point did anything get timed, weighed, or compared against a control.
The numbers weren’t exaggerated. They were fictional — internally consistent, plausible-looking, and completely disconnected from any real machine.
I archived the plugin instead of selling it. Not out of drama: a product whose headline claim is generated rather than measured cannot be sold to anyone without misrepresenting it, and one such claim would poison the credibility of everything else I make.
How this happens to well-meaning people
Nobody sits down to fabricate a benchmark. It arrives in three innocent steps.
First, you need a placeholder while building the UI, so you estimate. Second, the estimate looks reasonable, so it stays. Third — and this is the fatal one — the number gets piped into marketing copy, where it stops being an estimate and starts being a claim. Each step is small. The result is a lie with a decimal point in it.
Five questions that catch it
These work on anyone’s product, including mine. Ask them before you believe a performance claim.
- Compared to what? A speedup is a ratio, so there must be a control. “3× faster” with no stated baseline is not a measurement, it’s an adjective.
- Measured on what? Real numbers come with conditions attached — hardware, PHP version, dataset size, whether caching was on. Numbers with no conditions were not observed under any.
- Can I reproduce it? Is there a script, a harness, a recorded run? “Trust me” is the answer you get when there isn’t one.
- Is the number suspiciously round or suspiciously huge? Real measurements are ugly: 62 queries down to 55, not “90% fewer”. My fake one produced 1,400× because it was arithmetic, and arithmetic doesn’t know what is plausible.
- Does the claim survive being made specific? Ask a vendor to restate a claim with units and conditions. Real ones get more precise. Invented ones get vaguer.
What honest numbers look like
Here are some of mine, in the form I’m willing to publish them:
- “Front-end request with work due went from 62 to 55 database queries” — measured with Query Monitor on a live install, before and after, same page.
- “Dispatch latency 26 ms and 79 ms on two real spawns” — read from the plugin’s own telemetry on a live server, not a laptop.
- “299 assertions pass on PHP 7.4.33 and 8.4” — a command anyone can run.
None of those are impressive marketing. 62 to 55 is an 11% improvement, and I’m aware how that sounds next to “1,400×”. But every one of them is a thing that happened, which means nobody who buys on the strength of them will ever discover they were misled.
The rule I run on now
A number may appear in public if, and only if, I can point at the run that produced it. If it was estimated, it either says “estimated” beside it or it doesn’t exist. And any benchmark harness that computes rather than measures gets deleted rather than tuned, because the temptation to reuse it later is the actual danger.
This is also why the project pages here use two colours of stat: green for things that were measured, grey for plain facts like line counts. The distinction is the whole point.