I came across a blog where one tailscale engineer explained how he managed 12.5x throughput gain by switching to Tailscale’s peer relay. This is interesting.
A couple months back Tailscale introduced Peer Relays that allows two devices in a tailnet to connect to each other in case a direct connection cannot be established due to some reason. Most of the time, its because of NAT. Earlier Tailscale used DERP servers as fallback. All of the traffic is E2E encrypted over Wireguard. But these are global servers with shared resources so the QoS is generally not that good. What is interesting here is tailscale couldn’t establish a direct connection between his homeserver in US and his node in Delhi. I have a tailscale client running on EC2 instance in us-east-1 and my home server (behind Jio CGNAT). This is what I got:
tailscale ping 100.80.219.97
pong from homelab (100.80.219.97) via DERP(iad) in 743ms
pong from homelab (100.80.219.97) via DERP(iad) in 237ms
pong from homelab (100.80.219.97) via DERP(iad) in 241ms
pong from homelab (100.80.219.97) via DERP(iad) in 239ms
pong from homelab (100.80.219.97) via DERP(iad) in 235ms
pong from homelab (100.80.219.97) via 18.234.64.135:41641 in 329ms
Apparently, tailscale establishes direct peer-to-peer connection on default port 41641 and DERP appears only occasionally (usually on the very first ping of a new session or after a brief disconnect/reconnect). Instead of using the relay closest to my home server (Bangalore), it selects one closest to the instance in Ashburn (iad), which he explains as minimum combined latency.
Just like televisions and computers, it took quite a time for internet to be widely accessible in India. And with the scarcity of IPv4 addresses in late 90s and the infamous Domain Name Business, by mid-2010s when 4G was launched in India, IPv4 addresses were already running out. This led to wider adoption of IPv6 addresses. I have a JioFibre and my IPv4 address is likely shared by a lot of other people. Most ISPs use layers of NAT to distribute address spaces internally to their customers. For Tailscale to establish connection, it needs to punch through every NAT sitting in between my device and the internet. The most problematic among them is symmetric ones. Punching holes through a symmetric NAT is PITA.