Self-hosted speed tests need HTTP/1.1 and BBR

I run my own self-hosted speed test. It’s a fork of speedtest-go, the Go backend of LibreSpeed, on a VPS in Amsterdam behind Caddy. My line is 100 Mbps, and I tested it over a cable plugged into the ISP’s modem, without my router. The server is about 45 ms away. The command-line test reached the full line. The browser test showed less than half of it.

Three things were wrong, and none of them was in the code. Except where noted, all the numbers below come from that line, tested in Brave, Chromium, and Firefox.

My fork adds one thing to the stock page. It pings the server every 200 ms during the download and upload, to show the latency under load next to the idle latency. Stock LibreSpeed measures latency only in a separate ping test.

HTTP/2 puts everything on one connection

speedtest-go ships with enable_http2=false, so on its own it serves HTTP/1.1. Caddy turns on HTTP/2 and HTTP/3 by default, so behind Caddy the browser got HTTP/2 without me asking for it.

Over HTTP/1.1 each download request gets its own TCP connection. Over HTTP/2 the browser sends them all through one. What that costs depends on the server’s congestion control, which I cover below.

With cubic, the Linux default, one connection reached 30 to 45 Mbps on my line, so the page showed 36 to 42 Mbps. Six HTTP/1.1 connections reached 94 Mbps.

With BBR the download gap mostly closed, but the ping went up. They go through the same connection as the download and wait behind its data:

Over HTTP/2 the download was 6 to 12 percent lower, and the ping during download was 2 to 4 times higher. That ping measured the page’s own queue, not the line.

The ping problem only matters in a test that measures latency under load. My fork does, and so do fast.com, Cloudflare’s speed test, and Speedtest.net. All three run their test traffic over HTTP/1.1.

Over HTTP/1.1 a browser opens at most 6 connections to one server. If the download took all 6, every ping would wait for a free connection. So my page downloads over 4 and leaves room for the pings.

HTTP/3 depends on the browser

HTTP/3 also puts everything on one connection, but over QUIC on UDP instead of TCP. I ran the test over it in two browsers and got three different results:

A speed test should measure the line. Over HTTP/3 it measured the browser’s QUIC code, which gave a different answer in each case.

Serve LibreSpeed over HTTP/1.1

speedtest-go needs no change, since HTTP/2 is off by default. Turn it off in the reverse proxy instead. In Caddy, set the protocols in the global options:

{
	servers {
		protocols h1
	}
}

The option isn’t per site. If the same Caddy serves other sites on port 443, they lose HTTP/2 too. To keep HTTP/2 for them, run the speed test on its own server. With only h1, Caddy also stops serving HTTP/3.

Keep the grace times

LibreSpeed ignores the first 1.5 seconds of the download and the first 3 seconds of the upload, while the connections speed up. These are the grace times. With time_auto, a fast test ends after 6 to 9 seconds, so that start is a large share of it.

My settings.json set time_dlGraceTime and time_ulGraceTime to 0, so the test counted the start. The page’s average was 49 Mbps with both at 0 and 62 Mbps with the defaults. Leave them out of settings.json.

Use BBR on the server

With HTTP/1.1 and the default grace times, the page still showed 62 Mbps, and one connection reached only 30 to 36 Mbps.

When a cubic connection loses packets, it slows down, and at 45 ms it takes seconds to speed up again. The test ended before the connections caught up. BBR doesn’t treat every lost packet as a sign to slow down, and it filled the line within a few seconds. With BBR the page’s average rose from 62 to 84 Mbps, and one connection reached 57 to 90 Mbps.

To switch, run as root:

echo tcp_bbr > /etc/modules-load.d/bbr.conf
echo 'net.ipv4.tcp_congestion_control = bbr' > /etc/sysctl.d/60-bbr.conf
modprobe tcp_bbr
sysctl -p /etc/sysctl.d/60-bbr.conf

BBR applies to new connections and only on the sending side, so on the server it speeds up the download, not the upload. It also raised the latency under load from about 60 ms to between 120 and 150 ms, while the idle latency stayed at about 44 ms. The test now fills the line, and packets wait in the queue of the ISP’s modem. That’s bufferbloat on my line, not a problem with the server.

Results

After all three fixes, the test in Brave showed 100 to 104 Mbps down and 103 to 111 Mbps up, the full line.

Router QoS made the ping worse

My router, a TP-Link AC1200, has a QoS setting that limits the speed. I had it at 88 Mbps in both directions, a bit under the line, to keep its queue short. Tested over Wi-Fi and HTTP/1.1, in Firefox:

On the upload, QoS worked. The router holds my outgoing packets back to 88 Mbps, so no queue builds up behind it, and the ping stayed at 50 ms.

On the download, the packets have already crossed the ISP’s line by the time they reach my router, and it dropped whatever went over 88 Mbps. BBR doesn’t slow down for those drops, so the router kept dropping, and some of the lost packets were the pings. TCP waits at least 200 ms before it resends a lost packet, and doubles the wait each time the resend is lost too, so those pings took 300 ms to 3 seconds. With QoS off, the pings waited in a queue instead. That raised the median, but the worst case fell from 3 seconds to 0.4.

SQM (smart queue management) should fix both directions. It also limits the speed a little below the line, but it gives each connection its own queue. A ping doesn’t wait behind the download, and when the router has to drop packets, it drops them from the download. My router doesn’t have SQM, so I haven’t measured it.

Summary

• Serve the test over HTTP/1.1. Over HTTP/2 the download shares one connection. With cubic that cost more than half the speed, and with BBR the pings during download were 2 to 4 times higher. HTTP/3 results depend on the browser.
• Check what your reverse proxy turns on. Caddy enables HTTP/2 and HTTP/3 by default, even when the backend has HTTP/2 off.
• Keep the default grace times, so the test skips the start.
• Use BBR on the server. With cubic, a short test ends before the connections reach full speed.
• Against bufferbloat, use SQM on the router, not a QoS speed limit.