Tailscale performance updates reduce memory use, increase throughput, and speed startup with multi-queue, writev, and netmap caching.
111 comments
For really high bandwidth cases, things like DPDK are the long term best choice and are primarily userspace, for good reasons. Kernel mode is not the pure benefit it once was (if it ever was).
Separately, wireguard itself has a problem that the crypto suite it uses is not supported by hardware accelerators. So if we want to get into the hundreds of gigabits range, we will possibly need to switch packet formats entirely. (But, wireguard also needs to update to support post-quantum so maybe they’ll fix both problems at the same time and we can join in.)
Might as well switch to IPsec. Everything is already in place, including hardware acceleration. But most people won't, and we'll live in a world with duck-tape hacks built around a compromised Wireguard-ish layer.
Storytime: 15 years ago I co-founded a startup to build easy-to-use infrastructure management for AWS. At that time, it did not have cross-region VPC peering or routing, so you couldn't easily and safely have apps that communicate between regions.
So I started working on creating an overlay network. My idea was to use IPsec, it even has an RFC that documents its kernel interface. So that when a host wants to send a packet to the secure network, the kernel goes to my userspace daemon, that in turn goes to the central server that provides it the key for the given host pair.
And it turned out that the interface lacked a crucial part - on-demand key negotiation for incoming packets. It had this for _outgoing_ packets, but not incoming. The only sane way to make it work was to create a proactively updated database of all the hosts.
Well, I did that. It also did not work (tm). I found so many issues with broken MTU handling, broken NAT traversal, etc.
I eventually gave up on that idea and started working on a simple TUN/TAP-based overlay. I almost made everything work, but our startup got acquired by AWS, and this line of work was abandoned.
I don't think there's anything necessarily complicated about MLKEM that would make this too difficult.
Switching to IPSEC loses you other WireGuard benefits; the wins don't end at "just one carefully curated set of cryptography primitives", but extend into things like DoS prevention and a design that admits to processing incoming frames without dynamic allocation.
Things like Google’s PSP already achieve that while still being hardware acceleratable: https://github.com/google/psp
Post-quantum negotiation will kinda suck but you don’t have to sacrifice everything.
Has hardware-offload (for AES et al) got faster still, or that keeping CPU busy in the data path for 100gbps workloads is not ideal, or something else? The WireGuard website claims ChaPoly is at least as fast as hardware-accelerated AES. And that it can be further sped up with SIMD.
https://www.wireguard.com/known-limitations
> now we’re on to the next order of magnitude together
Curious: Is this work currently in progress? If so, what's more that's still lined up? The previous GRO/GSO(/LRO, too?) improvements were incredibly impressive (even to u/majke, https://news.ycombinator.com/item?id=35567268).
Thanks.
What is referred to by the term "hardware-offload"? For NICs:
> ConnectX NICs offload and accelerate encryption/decryption at speeds up to 400Gb/s.
* https://www.nvidia.com/en-us/networking/ethernet-adapters/
There have been MACsec implementations at 800Gb/s for several years:
* https://www.rambus.com/blogs/rambus-launches-800g-macsec-mul...
* https://semiengineering.com/the-evolution-of-ethernet-to-800...
And 1.6T/3.2T as well:
* https://www.rambus.com/security/protocol-engines/macsec-ip-3...
I haven’t used WireGuard but I have easily doubled OpenSSH performance by switching back to AES.
It's quantum-resistant if you define a PSK. You still have to distribute the key out of band, but you already have to do that anyways for the public keys of the peers.
It is slow. It cannot achieve speeds of greater than 1Gbps on clients systems (Windows & Mac), where you'd normally see it being used. On Linux, it struggles to achieve 10Gbps even when using a synthetic large packet benchmark [1]. With an IMIX benchmark, it would not be competitive whatsoever.
This problem is fixable. WireGuard achieves higher performance (Kernel vs Userspace implementation) and IPsec implementations can achieve 100Gbps/400Gbps (DPDK/XDP). Zero-copy networking.
From this blog post, I can say Tailscale still seems to not have the appetite for that, which is a shame.
>IPsec
Remember that at least one LPE CVE associated to kernel IPSec implementation has been discovered (copy.fail), which means that whatever gains you get from this vpn tunneling, is lost by breaking the basic user security system guarantee.
You are better off not using a VPN at all rather than using kernel crypto
And if any tailscale employees are reading this - https://github.com/tailscale/tailscale/issues/15724 please fix this too. Regular users not using some sort of enterprise saas DNS (whatever their thing is?) deserve DNS privacy too.
That said, note that if you run your own DNS server on your tailnet, the regular UDP DNS is automatically private because it’s carried over Tailscale. That’s the most common setup for non-SaaS DNS servers. DoH doesn’t really add anything in that arrangement. (And it’s more fiddly because you need to get and refresh a TLS cert.)
So it's not that simple: it's impossible for Tailscale to use any existing kernel or accelerated WireGuard implementation. They could derive inspiration, but a kernel module for Linux won't fix Windows & Mac. With that said, I feel they have enough funding to maintain a few platforms (:
If they would have taken that advice, tailscale instances would have been pwned by copy.fail
WireGuard takes 30 mins to an hour to set up, you'll need to configure port forwarding, DDNS, create keys for each device, and add them to each device manually. But you have 100% control.
Performance-wise, I haven't noticed a difference. My internet connection maxes out way before Tailscale hits any performance limits.
Tailscale wins for convenience.
- I was able to get my partner onto the tailnet by telling her to install an app and login. She doesn't know or care what wireguard is, but she can now access some of my self hosted services on her phone.
- I'm able to easily dynamically register machines to the tailnet, such as CI jobs
- I'm able to self host a DNS resolver and have it just work for devices connected to the tailnet
I'm sure I could achieve these goals with plain wireguard, but I feel like I was able to outsource significant complexity to tailscale instead.
Just the other day I was able to set my sister up with access to my Plex server and the ability to piggyback on my UK internet connection to stream BBC/Channel 4 content from Australia. It took all of 5 minutes to get it working.
I've even got a backup wireguard server running on a Pi 1b. Works fine. We currently run wireguard on our router (and it seems more and more routers are supporting it).
There are keys to configure for each client, but once you have the configuration for one client, the rest come very quickly and easily.
I should add that I don't have any experience with Tailscale, but compared to OpenVPN and other VPN solutions, Wireguard is lightweight, simple, and easy to setup/configure.
We use it on all our mobile devices (phones, laptops, tablets) to tunnel our traffic through our home network with all the filtering it offers (along side access to private services we host).
Once you install it you can then do everything from a UI to onboard new users.
It supports OIDC so you can avoid mangling with password and delegate that to an iDP such as google, github etc.
That being said, I do understand the appeal of Tailscale and have used it.
Tailscale allowed me to setup everything at home and just plug them to their network in 5 mins.
Assume we have devices a, b and c, which are basically in different segments of the same network and have nice pings to each other. We're trying to ssh from a to c, but NAT traversal isn't possible. Both a and c can do NAT traversal to b.
Instead of a going all the way over to the DERP in WAW and then back to c again, it could go a->b->c instead.
In my experience, DERPs are pretty slow and have high latency (compared to not going off-network at all), but there's no real way to avoid them if everything you have is behind some sort of NAT, even if some of them are trivially traversable (think "devices can do UPNP").
Note: I'm a tailscale employee
What I'm thinking of is something far more opportunistic; the way to piggyback on an existing peer (or a set of such) as a pseudo-relay, if and only if network conditions allow, and this is actually something that is worth doing in the given situation.
I'm not about to switch back to the configuration management hell that was pure wireguard but it would be really lovely to not have to connect/disconnect when I want to use tailscale
Read the full thread on Hacker News →
Related stories
- We're Making Tailscale Fastertailscale.comHacker News · 2 points · 8 days ago
- We're Making Tailscale Fastertailscale.comHacker News · 1 points · 8 days ago
- Hacker News · 1 points · 8 days ago
- Hacker News · 124 points · about 8 hours ago
- Hacker News · 67 points · 10 days ago
- Using Tailscale without using Tailscaletailscale.devLobsters · 39 points · over 3 years ago