Tailscale performance updates reduce memory use, increase throughput, and speed startup with multi-queue, writev, and netmap caching.

249 points•yarapavan•7 days ago•111 comments•

111 comments

apenwarr7 days ago
(Tailscale cofounder) I see a few comments here that using kernel wireguard would make it faster; it’s not really that simple. In fact, for a while (and we wrote a blog post about it), our optimizations made wireguard-go faster than kernel wireguard because it was better optimized. They adopted some of those improvements and now we’re on to the next order of magnitude together.

For really high bandwidth cases, things like DPDK are the long term best choice and are primarily userspace, for good reasons. Kernel mode is not the pure benefit it once was (if it ever was).

Separately, wireguard itself has a problem that the crypto suite it uses is not supported by hardware accelerators. So if we want to get into the hundreds of gigabits range, we will possibly need to switch packet formats entirely. (But, wireguard also needs to update to support post-quantum so maybe they’ll fix both problems at the same time and we can join in.)

xingped7 days ago
Hey why do you hard code certain android apps to be excluded from Tailscale with split tunneling without giving users any way to disable split tunneling for these apps? It doesn't matter how you think VPN does or does not affect these apps, it's really awful anti-user behavior.
dobremeno7 days ago
I haven't heard of this behavior before. Could you elaborate or link to some evidence of it?
alex7o7 days ago
I think it is the opposite google allows some apps from opting out of VPN.
Gathering66786 days ago
care to elaborate?
wahern7 days ago
Wireguard with PQ won't be Wireguard, anymore. It'll just be a rehash of IKE+IPsec. What made Wireguard better was the very simple handshake and minimal state, but no PQ algorithms can support that simplicity because the keys are too large and/or not as simple to use as ECC.

Might as well switch to IPsec. Everything is already in place, including hardware acceleration. But most people won't, and we'll live in a world with duck-tape hacks built around a compromised Wireguard-ish layer.

cyberax7 days ago
IPSec suffers from the "it can do everything" syndrome.

Storytime: 15 years ago I co-founded a startup to build easy-to-use infrastructure management for AWS. At that time, it did not have cross-region VPC peering or routing, so you couldn't easily and safely have apps that communicate between regions.

So I started working on creating an overlay network. My idea was to use IPsec, it even has an RFC that documents its kernel interface. So that when a host wants to send a packet to the secure network, the kernel goes to my userspace daemon, that in turn goes to the central server that provides it the key for the given host pair.

And it turned out that the interface lacked a crucial part - on-demand key negotiation for incoming packets. It had this for _outgoing_ packets, but not incoming. The only sane way to make it work was to create a proactively updated database of all the hosts.

Well, I did that. It also did not work (tm). I found so many issues with broken MTU handling, broken NAT traversal, etc.

I eventually gave up on that idea and started working on a simple TUN/TAP-based overlay. I almost made everything work, but our startup got acquired by AWS, and this line of work was abandoned.

tptacek7 days ago
I don't think this really follows. Negotiation would be problematic, but you can just version the protocols and do WireGuard v1 and WireGuard v2, with v2 in a PQ configuration. As long as the configuration about which to use is static, you're not running up against the IPSEC problem.

I don't think there's anything necessarily complicated about MLKEM that would make this too difficult.

Switching to IPSEC loses you other WireGuard benefits; the wins don't end at "just one carefully curated set of cryptography primitives", but extend into things like DoS prevention and a design that admits to processing incoming frames without dynamic allocation.

apenwarr7 days ago
(Tailscale cofounder) I’m a little more optimistic; wireguard was always going to need a v2 eventually, that’s just the nature of cryptography. It’ll always be simpler than IPsec as long as it avoids live negotiation (you need to specify your suite up front for each node you talk to; v1 and v2 are the only suites) and continues to go over UDP instead of IPsec’s “it’s a different transport protocol!” madness.

Things like Google’s PSP already achieve that while still being hardware acceleratable: https://github.com/google/psp

Post-quantum negotiation will kinda suck but you don’t have to sacrifice everything.

rurban7 days ago
But then consider ipsec being backdoored by the NSA. As it came out of the Snowden leaks.
lausobo7 days ago
Any chance the Apple TV can get a performance boost? Specially as an exit node. It always gives very little bandwidth. Thank you!
ls6127 days ago
Apple TV has an OS level issue where it will kill apps that do too much background activity at random, the Tailscale guys can’t really change that.
ignoramous7 days ago
> wireguard itself has a problem that the crypto suite it uses is not supported by hardware accelerators. So if we want to get into the hundreds of gigabits range, we will possibly need to switch packet formats entirely

Has hardware-offload (for AES et al) got faster still, or that keeping CPU busy in the data path for 100gbps workloads is not ideal, or something else? The WireGuard website claims ChaPoly is at least as fast as hardware-accelerated AES. And that it can be further sped up with SIMD.

https://www.wireguard.com/known-limitations

> now we’re on to the next order of magnitude together

Curious: Is this work currently in progress? If so, what's more that's still lined up? The previous GRO/GSO(/LRO, too?) improvements were incredibly impressive (even to u/majke, https://news.ycombinator.com/item?id=35567268).

Thanks.

throw0101c6 days ago
> Has hardware-offload (for AES et al) got faster still, or that keeping CPU busy in the data path for 100gbps workloads is not ideal, or something else?

What is referred to by the term "hardware-offload"? For NICs:

> ConnectX NICs offload and accelerate encryption/decryption at speeds up to 400Gb/s.

* https://www.nvidia.com/en-us/networking/ethernet-adapters/

There have been MACsec implementations at 800Gb/s for several years:

* https://www.rambus.com/blogs/rambus-launches-800g-macsec-mul...

* https://semiengineering.com/the-evolution-of-ethernet-to-800...

And 1.6T/3.2T as well:

* https://www.rambus.com/security/protocol-engines/macsec-ip-3...

mxey7 days ago
> The WireGuard website claims ChaPoly is at least as fast as hardware-accelerated AES.

I haven’t used WireGuard but I have easily doubled OpenSSH performance by switching back to AES.

10000truths7 days ago
> But, wireguard also needs to update to support post-quantum

It's quantum-resistant if you define a PSK. You still have to distribute the key out of band, but you already have to do that anyways for the public keys of the peers.

wisemang7 days ago
Why do public keys need to be distributed out of band?
iscoelho7 days ago
In my opinion, this is Tailscale's largest issue.

It is slow. It cannot achieve speeds of greater than 1Gbps on clients systems (Windows & Mac), where you'd normally see it being used. On Linux, it struggles to achieve 10Gbps even when using a synthetic large packet benchmark [1]. With an IMIX benchmark, it would not be competitive whatsoever.

This problem is fixable. WireGuard achieves higher performance (Kernel vs Userspace implementation) and IPsec implementations can achieve 100Gbps/400Gbps (DPDK/XDP). Zero-copy networking.

From this blog post, I can say Tailscale still seems to not have the appetite for that, which is a shame.

[1] https://tailscale.com/blog/more-throughput

TZubiri7 days ago
>(Kernel implementation)

>IPsec

Remember that at least one LPE CVE associated to kernel IPSec implementation has been discovered (copy.fail), which means that whatever gains you get from this vpn tunneling, is lost by breaking the basic user security system guarantee.

You are better off not using a VPN at all rather than using kernel crypto

iscoelho7 days ago
By that logic, we should avoid TCP as the Linux kernel implementation has had plenty of CVEs. Thankfully our expert critical thinking helps us acknowledge that as silly.
boomer_joe7 days ago
Yes. Just fucking stop doing userspace wireguard on linux https://github.com/tailscale/tailscale/issues/426 - issue has been open for 6 years (and is locked now, lol), btw.

And if any tailscale employees are reading this - https://github.com/tailscale/tailscale/issues/15724 please fix this too. Regular users not using some sort of enterprise saas DNS (whatever their thing is?) deserve DNS privacy too.

apenwarr7 days ago
(Tailscale cofounder) That’s a good callout on DoH support, thanks.

That said, note that if you run your own DNS server on your tailnet, the regular UDP DNS is automatically private because it’s carried over Tailscale. That’s the most common setup for non-SaaS DNS servers. DoH doesn’t really add anything in that arrangement. (And it’s more fiddly because you need to get and refresh a TLS cert.)

iscoelho7 days ago
Tailscale's netstack is barely even WireGuard and they aren't compatible whatsoever. It's all marketing at this point.

So it's not that simple: it's impossible for Tailscale to use any existing kernel or accelerated WireGuard implementation. They could derive inspiration, but a kernel module for Linux won't fix Windows & Mac. With that said, I feel they have enough funding to maintain a few platforms (:

lokar7 days ago
Kernel networking is not automatically faster then userspace.
TZubiri7 days ago
> Just fucking stop doing userspace wireguard on linux - issue has been open for 6 years

If they would have taken that advice, tailscale instances would have been pwned by copy.fail

fitblipper7 days ago
I used to LOVE tailscale. Then I put wireguard on my home network exposed to the internet with a dynamic DNS provider and it immediately became irrelevant. Not only is raw wireguard more stable (I don't have to fight the DNS issues on my mobile phones) it feels faster and is amazingly simple to set up.
tristanj7 days ago
Tailscale takes two minutes to setup and you can add more devices with zero configuration.

WireGuard takes 30 mins to an hour to set up, you'll need to configure port forwarding, DDNS, create keys for each device, and add them to each device manually. But you have 100% control.

Performance-wise, I haven't noticed a difference. My internet connection maxes out way before Tailscale hits any performance limits.

Tailscale wins for convenience.

mnahkies7 days ago
I started with plain wireguard then migrated to tailscale, for my use case:

- I was able to get my partner onto the tailnet by telling her to install an app and login. She doesn't know or care what wireguard is, but she can now access some of my self hosted services on her phone.

- I'm able to easily dynamically register machines to the tailnet, such as CI jobs

- I'm able to self host a DNS resolver and have it just work for devices connected to the tailnet

I'm sure I could achieve these goals with plain wireguard, but I feel like I was able to outsource significant complexity to tailscale instead.

ctippett7 days ago
I followed a similar trajectory for similar reasons. I was playing with wireguard around 2020 when I learned of Tailscale and since then haven't looked back.

Just the other day I was able to set my sister up with access to my Plex server and the ability to piggyback on my UK internet connection to stream BBC/Channel 4 content from Australia. It took all of 5 minutes to get it working.

PorciiVorbesc7 days ago
Care to share your setup? I did some research into self hosting my own wireguard for my nuc and rpi, before ultimately settling on Tailscale because of how much simpler and plug-and-lay it was to add/remove devices compared to self hosting wireguard, not dealing with certificates, maintenance, etc.
fitblipper7 days ago
I have an openwrt router running wireguard. I use it to provision the peer keys and routes. I also use openwrts cloudflare ddns which is super simple to setup. Any new client I want to add I jump into the wireguard interface in the GUI, go to the peer tab, and it does everything for me there.
davidee7 days ago
There are no certificates to share with Wireguard. Nothing to rotate if you don't want to. Once it works, it works.

I've even got a backup wireguard server running on a Pi 1b. Works fine. We currently run wireguard on our router (and it seems more and more routers are supporting it).

There are keys to configure for each client, but once you have the configuration for one client, the rest come very quickly and easily.

I should add that I don't have any experience with Tailscale, but compared to OpenVPN and other VPN solutions, Wireguard is lightweight, simple, and easy to setup/configure.

We use it on all our mobile devices (phones, laptops, tablets) to tunnel our traffic through our home network with all the filtering it offers (along side access to private services we host).

kureikain5 days ago
I used https://wgportal.org/latest/

Once you install it you can then do everything from a UI to onboard new users.

It supports OIDC so you can avoid mangling with password and delegate that to an iDP such as google, github etc.

bmurphy19767 days ago
Get a Unifi system. It's built in. Works great!
havaloc7 days ago
I use an Island Router with Wireguard server built in, it handles DDNS and even the base Island router is beefy enough to give me up to 954mbps or so of Wireguard to right inside my home network. It took 2 minutes to set up.

That being said, I do understand the appeal of Tailscale and have used it.

UltraSane7 days ago
My favorite thing about Tailscale is how it lets you SSH/RDP INTO servers without having to open any ports.
pammf7 days ago
It all depends on the use case…I have two raspberry units running as exit nodes back in my home country, one in my mother’s place and another in my in-laws’. They have regular internet providers routers, and at least one of the routers wouldn’t even be able to properly support port forwarding.

Tailscale allowed me to setup everything at home and just plug them to their network in 5 mins.

miki1232117 days ago
Re: Tailscale and speeds, I wish Tailscale had a better story about relay / DERP flexibility.

Assume we have devices a, b and c, which are basically in different segments of the same network and have nice pings to each other. We're trying to ssh from a to c, but NAT traversal isn't possible. Both a and c can do NAT traversal to b.

Instead of a going all the way over to the DERP in WAW and then back to c again, it could go a->b->c instead.

In my experience, DERPs are pretty slow and have high latency (compared to not going off-network at all), but there's no real way to avoid them if everything you have is behind some sort of NAT, even if some of them are trivially traversable (think "devices can do UPNP").

jaxxstorm7 days ago
you appear to be describing a peer relay: https://tailscale.com/docs/features/peer-relay

Note: I'm a tailscale employee

miki1232116 days ago
Yes, except peer relays require the peer not to be behind NAT instead of being behind traversable NAT.

What I'm thinking of is something far more opportunistic; the way to piggyback on an existing peer (or a set of such) as a pseudo-relay, if and only if network conditions allow, and this is actually something that is worth doing in the given situation.

dust-jacket6 days ago
This all sounds great but until they stop it eating all my iOS battery I can't leave my phone connected continuously, which is such a pain.

I'm not about to switch back to the configuration management hell that was pure wireguard but it would be really lovely to not have to connect/disconnect when I want to use tailscale

Read the full thread on Hacker News →

Related stories