Your dashboard says average response time is 180ms. Leadership is happy. Then support tickets start: "the app is unbearably slow." Both are true at once — and the reason is that the average is one of the most misleading numbers in all of monitoring.
This guide explains latency percentiles — p50, p95, p99, p99.9 — from first principles. What they mean, why averages lie, how to set SLOs on them, how to alert on them without drowning in noise, and the statistical traps that make naive percentile monitoring quietly wrong.
If you only take one thing away: stop looking at average latency. Look at the tail.
Why The Average Lies
Latency distributions are not bell curves. They are heavily right-skewed: most requests are fast, and a long tail of requests are slow — sometimes 10x to 100x slower. The mean gets dragged toward the fast bulk and hides the tail entirely.
A worked example. Ten requests, in milliseconds:
50, 55, 60, 60, 62, 65, 70, 72, 80, 2000
- Average: 257ms — looks acceptable.
- Median (p50): 63ms — what a typical request actually feels.
- Max: 2000ms — one user waited 2 seconds.
The average (257ms) describes no actual request. It is higher than 90% of requests and 8x lower than the slowest. It is a number that exists nowhere in reality. That is the fundamental problem: averages describe a distribution that latency data does not have.
Worse, averages are easy to game accidentally. Add a few thousand fast health-check pings to your dataset and your average plummets — while every real user still waits exactly as long as before.
What Each Percentile Actually Means
A percentile is a threshold below which a given share of requests fall.
- p50 (median): 50% of requests are faster than this. The "typical" experience.
- p75: 75% are faster. Google uses p75 for Core Web Vitals field data.
- p90: 90% are faster. The beginning of the tail.
- p95: 95% are faster. 1 in 20 requests is slower than this.
- p99: 99% are faster. 1 in 100 requests is slower — your unlucky users.
- p99.9 ("three nines"): 1 in 1,000 requests is slower. At scale, this is thousands of real people per day.
Read p99 = 1,200ms as: "One percent of requests took longer than 1.2 seconds." If you serve 10 million requests a day, that is 100,000 slow experiences every single day — hidden completely behind a healthy-looking average.
Which percentile should you care about?
| Percentile | Use it for |
|---|---|
| p50 | Sanity baseline, capacity trends |
| p90 / p95 | The standard SLO target for most user-facing services |
| p99 | High-value flows: checkout, login, API contracts |
| p99.9 | High-scale platforms where the tail is still huge in absolute terms |
A common rule: set SLOs at p95 or p99, never on the average. The whole point is to bound the bad experiences, not the typical one.
Tail Latency Is The Real User Experience
Here is the counterintuitive part: the more requests a single user-facing action triggers, the more the tail dominates what they feel.
Modern pages fan out. One page load might trigger 20 backend calls (auth, profile, feed, ads, recommendations, telemetry…). The page is only "done" when the slowest of those 20 calls returns. If each call independently has a 1% chance of being slow (p99), the probability that at least one of 20 is slow is:
1 - (0.99)^20 ≈ 18%
So a "1% slow" backend produces an 18% slow page. Your p99 backend latency becomes your users' median experience. This is why tail latency, not average latency, is the number that correlates with churn, abandoned carts, and angry tickets.
For how slow responses translate directly into lost revenue, see the hidden cost of slow websites.
The Statistics Traps (Where Naive Monitoring Goes Wrong)
Percentiles are easy to compute wrong. These are the four traps that bite teams hardest.
1. You cannot average percentiles
This is the big one. If region A has p99 = 200ms and region B has p99 = 800ms, the global p99 is not 500ms. Percentiles are not additive. Averaging them, or averaging them across time buckets, produces a number with no statistical meaning.
To aggregate correctly you must either keep the raw data or use a mergeable structure (histograms, t-digest, HdrHistogram). Most good monitoring backends store histograms precisely so percentiles can be recomputed over any time range or dimension.
2. Pre-aggregated percentiles per host are lost forever
If each server computes its own p99 every minute and ships only that number, you can never reconstruct the fleet-wide p99 — you have thrown away the distribution. Always aggregate histograms, then compute percentiles at query time.
3. Coordinated omission
The nastiest trap. Many load-testing and timing tools only measure requests they actually sent. When the system stalls, they stop sending requests during the stall, so the worst latencies are never recorded. The result: your p99 looks great precisely when the system is at its worst. Tools like HdrHistogram and properly configured load generators correct for this; naive time the request loops do not.
4. Not enough samples
A p99 computed from 50 data points is essentially noise — it is governed by your single slowest sample. You need on the order of hundreds to thousands of samples per window for p99 to be stable, and far more for p99.9. Low-traffic endpoints will have jumpy tail percentiles; widen the window or aggregate routes.
How To Monitor Percentiles In Practice
Always chart multiple percentiles together
Plot p50, p95, and p99 on the same graph. The gap between them is the signal:
- p50 and p99 both low, close together: healthy, consistent service.
- p50 low, p99 high (wide gap): a tail problem — GC pauses, lock contention, cold caches, a slow dependency, noisy neighbors.
- p50 and p99 both rising together: systemic saturation — you are out of capacity.
The shape tells you the cause before you even open a trace.
Segment by dimension
A global p99 hides which segment is suffering. Break percentiles down by:
- Route / endpoint —
/checkoutmay be fine while/searchis on fire. - Region / data center — last-mile and replication differ.
- Device class — covered well by Real User Monitoring.
- Release / deploy — so a regression is attributable to a specific ship.
- Cache hit vs miss — these are two completely different distributions.
Separate client-side from server-side
Server-side percentiles (your app's processing time) and client-side percentiles (what the user's browser measured, including network) tell different stories. TTFB monitoring covers the server side; RUM covers the real-user side. You need both.
Setting SLOs and Alerts On Percentiles
A latency SLO is a percentile target over a window. For example:
99% of
/api/checkoutrequests complete in under 500ms, measured over a rolling 30-day window.
That single sentence defines the metric (p99), the threshold (500ms), the scope (the checkout endpoint), and the window (30 days). It connects directly to error budgets — see the SLO, SLI & error budget guide.
Alert on burn rate, not on instantaneous spikes
Alerting "page me when p99 > 500ms" fires on every transient blip and trains your team to ignore the pager — classic alert fatigue. Instead, alert on error-budget burn rate: page when you are consuming your latency budget fast enough to blow the monthly SLO. This catches real degradation while staying quiet on noise.
A practical multi-window approach:
- Fast burn (page): budget burning >14.4x over 1 hour → something is badly wrong now.
- Slow burn (ticket): budget burning >3x over 6 hours → a creeping regression worth investigating.
Pair latency with error rate
A request that fails fast is not "fast." Always read latency percentiles alongside the 5xx error rate — otherwise an outage where everything returns instant 500s will look like a latency improvement.
A Reference Latency Dashboard
For each critical endpoint:
- p50 / p95 / p99 over time, on one chart, with deploy markers.
- p99 by region and by release.
- Histogram / heatmap of the latency distribution (not just the percentile lines) — this reveals bimodal distributions a percentile line hides.
- Request volume alongside — so you can tell whether a percentile is meaningful or just low-sample noise.
- Error rate on the same time axis.
- Budget burn rate for the latency SLO.
The heatmap is underused and powerful: a bimodal distribution (e.g. fast cache hits + slow cache misses) shows up as two bands, which a single p99 line completely flattens.
Quick Reference
- Never alert or report on average latency for user-facing services.
- p50 = typical, p95/p99 = the SLO targets, p99.9 = matters at scale.
- Tail latency dominates real user experience because of request fan-out.
- You cannot average percentiles — aggregate histograms instead.
- Beware coordinated omission and low sample counts.
- Chart p50/p95/p99 together; the gap diagnoses the cause.
- Alert on error-budget burn rate, not instantaneous spikes.
- Always read latency next to error rate.
How Webalert Helps
Webalert measures latency the way your users experience it — from outside your infrastructure, across regions:
- Multi-region response-time checks so you see p-values per geography, not a single averaged number.
- Response-time alerting with thresholds you can tune to the percentile that matters for each endpoint.
- Outside-in measurement that captures the full path (DNS, TLS, network, server), complementing internal APM percentiles.
- Content validation so a fast response that returns the wrong body still gets caught — see response body validation.
- Status pages and history to communicate and review latency trends over time.
For a satisfaction-oriented single-number companion to percentiles, see the Apdex score guide. For how percentiles fit into reliability targets, see SLOs and error budgets.
Summary
The average is the comfortable lie; the tail is the truth. Latency percentiles — especially p95, p99, and p99.9 — describe what your users actually feel, and they expose the slow experiences that averages bury. Compute them correctly (aggregate histograms, mind coordinated omission, watch your sample counts), chart them together, set SLOs on them, and alert on budget burn rate rather than every spike.
Do that, and the gap between "the dashboard looks fine" and "users say it's slow" disappears.