Lab numbers, field data and a real phone

A page scores well and still feels slow. Another page scores badly and nobody has ever complained about it. Both situations are normal, because a synthetic audit, a field dataset and a phone in your hand are three different instruments answering three different questions, and treating any of them as the truth produces work that does not help anybody. The useful skill is knowing which instrument to reach for and knowing what each one gets wrong.

Three instruments, three questions

A synthetic audit runs the page once, on a simulated device, over a simulated network, with an empty cache and usually without the extensions, consent banners and tags that a real visit drags along. Its strength is that it is repeatable: run it before and after a change and the difference is mostly your change. Its weakness is that the device it simulated is not the device anybody owns, and the run it describes never happened to a person.

Field data comes from real sessions and is reported as a distribution, usually with attention on the slower end rather than the average, because the average hides the people having a bad time. Its strength is that it is true. Its weakness is that it explains nothing: it can tell you a quarter of visitors waited four seconds, and it cannot tell you for what. It also lags, because it is aggregated over a rolling window, so the change you shipped on Monday is diluted by three weeks of the old code.

A real phone on a real connection is the only instrument that reports on feel. Scrolling that stutters, a tap that does nothing for half a second, a layout that jumps after the fonts arrive, a keyboard that covers the field you are typing in. None of those appear cleanly in either of the other two.

When each one lies

• The audit lies when your visitors are not on the device it simulated. If most of your traffic is on a fast connection and warm cache, a cold cache on a throttled mid range phone is a worst case, not a typical case, and optimising for it can cost you work that helps nobody.
• The audit lies when run to run variance is bigger than your change. A single run is a sample, and a single sample can move several points for reasons that have nothing to do with your code.
• Field data lies when you have too little traffic. Below a few thousand samples the slow end of the distribution is a handful of sessions, and one person on a train can move it.
• Field data lies when one page or one region dominates. A site wide number that is actually a single popular page tells you nothing about the rest of the site, so segment before you conclude.
• The phone lies when it is your phone, on your wifi, with a warm cache, loading the page you just built and know exactly where to look on. That combination makes everything feel fine.

A routine you can repeat

Five steps, and the discipline is in doing them in order rather than in the tooling:

1. Pick one metric and one page. Not a score, a metric, and the page with the most traffic rather than the one you find most interesting.
2. Take a baseline. Several runs, recorded, with the spread visible. If you cannot say how much your measurements bounce around, you cannot tell a real improvement from noise.
3. Change one thing.
4. Measure again under identical conditions. Same network profile, same cache state, same number of runs.
5. Write down the number and whether you kept the change. A number nobody wrote down gets re argued in three months.

For the baseline, anything that repeats is better than a careful single run. The crude version costs one line:

for i in $(seq 1 9); do
  curl -so /dev/null -w '%{time_starttransfer}\n' https://example.com/
done | sort -n | awk '{a[NR]=$1} END {printf "min %s median %s max %s\n", a[1], a[int((NR+1)/2)], a[NR]}'
# min 0.048 median 0.061 max 0.139

That only covers the server side, but it answers the first question worth answering, which is whether the delay is in producing the response or in rendering it. If the median is already sixty milliseconds, no amount of front end work will help, and you should be looking at something like a slow query and the right index instead.

For the rendering side, the number is less useful than the identity of the element. This reports which element the browser decided was the largest contentful paint, which is the fact that most often changes what you do next:

new PerformanceObserver((list) => {
  const e = list.getEntries().at(-1);
  console.log(
    Math.round(e.startTime),
    e.element ? e.element.tagName : '(none)',
    e.url || '',
  );
}).observe({ type: 'largest-contentful-paint', buffered: true });
// 1840 IMG https://example.com/a/hero.9f1c2a44.avif
// 2210 H1

When the answer is an image, the work is sizing and format. When it is a heading, the work is usually a font or a stylesheet blocking the paint. When it moves between runs, the page has a race in it and the average is meaningless until you fix that.

The score trap

A composite score is a weighted sum of several metrics, each passed through a curve with thresholds. Two consequences follow. A change can move the score without moving anything a person feels, because it crossed a threshold. And a change can help every visitor by a small amount without moving the score at all, because it stayed inside one band.

The practical effect is that chasing the score picks whichever change is cheapest to make, not whichever change is worth making. I have seen an afternoon spent moving a number by three points while the thing visitors actually waited for stayed exactly where it was, which is the trap described from the other side in your entrance animation is delaying your largest paint.

The habit that avoids it is to state the change in user terms before you look at any number. "The headline paints before the font arrives" is a claim you can check. "The score went from 89 to 92" is not a claim about anything. Make the first kind of statement, then use the measurement to confirm or refute it.

How to check it worked

Three checks, in increasing order of what they cost and what they are worth:

# same command, same conditions, after the change
for i in $(seq 1 9); do
  curl -so /dev/null -w '%{time_starttransfer}\n' https://example.com/
done | sort -n | awk '{a[NR]=$1} END {printf "median %s spread %s\n", a[int((NR+1)/2)], a[NR]-a[1]}'
# median 0.058 spread 0.031

Then repeat the audit the same number of times and compare medians, not best runs. Then load the page on a mid range phone, on mobile data, with the cache cleared, and look at it rather than at a number. If all three agree, the change is real. If the audit improved and the phone did not, you changed the measurement rather than the page.

Field data is the slow confirmation. Mark the release date on the chart and expect the line to move over the following weeks rather than the following day, because the window is still full of sessions from the old code.

What to watch out for

• Measuring on the machine that built the site. A local server with a warm cache and no network is the one environment guaranteed to be unrepresentative.
• Changing two things at once. If both ship together and the number improves, you have learned nothing about either, and one of them may be making things worse.
• Optimising a page nobody visits. Sort pages by traffic before you start, and accept that the boring page is usually the one that matters.
• A metric that improved because something stopped loading rather than because it got faster. Check that the feature still works, especially when the improvement is suspiciously large.
• Trusting an audit that ran against a page with different content. A listing page with three items in a test environment is not the listing page with three hundred that visitors see, and the difference can be the entire problem, as it often is with what the responsive images audit is really telling you.

Measuring first pays for itself because the guess is wrong often enough to be expensive, and rigour for its own sake has nothing to do with it. Every performance problem I have worked on had an obvious cause that turned out to be the second or third thing on the list, and the only reason I know that is that somebody wrote the baseline down. Take the baseline, change one thing, measure under the same conditions, and keep the number somewhere the next person can find it. The tooling matters much less than the order.