Fix bufferbloat with WireGuard and cake

Bufferbloat is when a download or upload fills a queue somewhere on the line, and every other packet waits in it. Calls stutter and games lag while something else downloads. The usual fix is SQM on the router. It limits the speed a bit under the line, so the queue builds in the router instead of the ISP’s modem. It also gives each connection its own queue.

That stops working once the traffic goes through a VPN. If you run your own WireGuard server, you can shape the traffic at both ends of the tunnel instead, with cake. It’s a traffic shaper built into Linux.

My line is 100 Mbps, and the idle ping to my speed test server is about 46 ms. I measured in Firefox with my own speed test, which pings the server every 200 ms during the download and upload. I describe it in Self-hosted speed tests need HTTP/1.1 and BBR.

Router SQM can’t see inside the tunnel

The router sees everything that goes through the VPN as one connection, so it can’t tell the connections inside it apart. Over the VPN, with the router’s SQM on, the median ping during download rose from 45 to about 85 ms.

Shape at both ends

Cake on the server’s WireGuard interface shapes the download. Cake on the laptop’s shapes the upload. Both get the packets before WireGuard encrypts them, so cake can give each connection its own queue.

On the server, as root:

tc qdisc replace dev wg0 root cake bandwidth 82mbit

That applies to all connected VPN clients.

On the laptop, NetworkManager runs the scripts in /etc/NetworkManager/dispatcher.d/ when a connection goes up or down. This one sets cake on the tunnel while I’m on my home 5 GHz Wi-Fi. On other networks, including the slower 2.4 GHz one at home, it removes cake, since 85 Mbps would be the wrong limit there:

#!/bin/sh
# Shape upload on the WireGuard tunnel with cake, only on the home line.
# NetworkManager runs this with the interface in $1 and the action in $2.

TUNNEL=client
LIMIT=85mbit
# NetworkManager connection names of the home line, one per line.
HOME_CONS="Home Wi-Fi 5G"

case "$2" in
	up | down) ;;
	*) exit 0 ;;
esac

# The tunnel is down, so there is nothing to set.
[ -d "/sys/class/net/$TUNNEL" ] || exit 0

if /usr/bin/nmcli -t -f NAME con show --active | grep -qxF "$HOME_CONS"; then
	/usr/sbin/tc qdisc replace dev "$TUNNEL" root cake bandwidth "$LIMIT"
else
	/usr/sbin/tc qdisc del dev "$TUNNEL" root 2>/dev/null
fi
exit 0

nmcli con show lists your connection names. Save the script as /etc/NetworkManager/dispatcher.d/90-client-cake, owned by root, with chmod 755. NetworkManager skips scripts that aren’t executable or that others can write to.

Results

Over the VPN, with SQM off on the router:

With cake on both ends, I got the same on two different routers.

Pick the limits

Cake has to be the slowest point on the line, or the queue builds somewhere else. WireGuard adds about 4 percent to every packet, and cake on the tunnel doesn’t count it. So the limits have to be well under the line.

Start about 20 percent under the speed you get without the VPN. Then run a test that shows the ping under load, such as Waveform’s bufferbloat test with the VPN on. Raise the limit a few Mbps at a time until the ping under load starts to rise, then go back a step. On my line that was 82 Mbps for download and 85 for upload.

Limitations

• It costs speed. I got 84 Mbps, against 97 with SQM on the router.
• Only traffic through the VPN is shaped. A TV that downloads outside it still fills the modem’s queue, and the ping goes up for everyone.
• Wi-Fi has to be faster than cake’s limit. Far from the router, the Wi-Fi link became the slowest point, and the worst ping during download reached 1,165 ms.
• Android can’t set cake for upload without root. The server still shapes the phone’s download traffic.

Summary

• Router SQM can’t separate connections inside a VPN tunnel.
• With your own WireGuard server, cake on both ends of the tunnel keeps the ping at the idle level in both directions, on any router.
• If your router has SQM, keep it on too, with its limits set above cake’s. It shapes the devices outside the VPN, and the VPN ping stayed at the idle level with it on.
• Set the limits under the line, and find them with a test that shows the ping under load.