Cloudflare K2: serverless event streams

Cloudflare K2: serverless event streams blog.cloudflare.com

Cloudflare K2 is a serverless event streaming service built directly on top of R2 object storage for high-scale data movement and long-term retention. By decoupling producers and consumers at the edge, K2 enables…

134 points•elffjs•about 6 hours ago•50 comments•

#cloudflare #serverless #event #serverless event #streams #event streams #serverless event streams #streaming

50 comments

psanfordabout 4 hours ago

Object store is quickly becoming the new core data substrate. Lets build kafka, but on s3. Lets build github, but on s3. It feels like we going to see more and more "object-store first" systems in the next few years.

I am excited about this future. Give me stateless servers and a storage bucket over having to manage systems with disks any day.

I do wonder if we will see an expansion of the s3 api to support more of these use cases. S3 added a janky file append operation to their new express-one-zone bucket type, and limited to 10k total file append operations. I wonder what else we will get in the next few years.

shyeabout 1 hour ago

It makes perfect sense: it's a extremely reliable, infinitely-scalable, strongly consistent (in some implementation), cheap, and extremely simple to use key-value store. As such, it transparently solves a lot of the problems distributed systems have to be engineered around.

As long as you can run on a CSP, and can engineer around the high-ish latency (most business cases can), it's extremely expensive to try engineer around it.

khazitabout 2 hours ago

I still think S3 is underutilized.

The range of things you can do with blob storage and a (very simple) auth model are surprisingly broad.

We recently replaced our Docker container registry with S3 using a tiny tool [1] we built in-house. I think that even with current capabilities, we can still model a lot services as a very thin layer over object storage.

[1]: https://github.com/Simple-Observability/grue

monster_truckabout 2 hours ago

From the opposite side of things, I feel the same about OPFS. Finally having something that performant that multiple workers can operate against in a browser is such a massive boon. You could plug something like that in to it locally with markedly less bullshit than you would have had to do previously with the other APIs.

deanputneyabout 1 hour ago

I'm a little surprised the Docker CLI doesn't support this directly, since gcr.io works this way. Maybe there's a thin layer between the bucket and the CLI? Neat that you worked it out for the generic case.

tomjen3about 2 hours ago

That sounds really cool — I tend to agree with you that S3 and similar are underutilized, but I remember that essentially all of the providers charge for bandwidth measured in gigabytes and I'm like no. My consumer line is measured in megabits per second and if I have to pay for my data usage the way it's paid for in data centers it would be far more expensive. Somehow consumer ISPs, who have to pay for the lines, are cheaper than than the cloud providers.

necubiabout 4 hours ago

Definitely agree that every data system that doesn't need <100ms latency is moving to object storage.

> I do wonder if we will see an expansion of the s3 api to support more of these use cases

This is actually an area where I think we have a big leg up on folks building on top of S3. My team (which built K2) sits next to the R2 team, and we have the opportunity to co-evolve the products in mutually beneficial ways.

madjam00243 minutes ago

Came across this the other day which lets you run a etcd compatible API/Kubernetes on top of S3

https://github.com/t4db/t4

Haven't tried it yet but looks nice for simpler K8s deployments

6thbitabout 3 hours ago

doesn't that make egress fees egregious? or still cheaper than disks?

cjabout 3 hours ago

You can download objects from S3 from EC2 without traversing the public internet using things like gateway endpoints [0] which avoids s3 egress fees. But doesn't avoid egress fees from EC2 to the end user.

[0] https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpo...

vmg12about 3 hours ago

This all depends on the cloud you are building on and if the data ever leaves the datacenter.

wg013 minutes ago

Has anyone built anything on top of D1 + Durable objects for multi tenancy? How has been the experience?

vira2833 minutes ago

I wonder how many of data infra startups are wrapper on top of S3.

Also, the boundary between OLTP and OLAP is blurring every day.

For folks who want an off shelf version of this you may be interested in https://github.com/viggy28/streambed

(Disclaimer: I am one of the committers)

alexpotato8 minutes ago

LinkTree is a billion dollar business and it's basically a key value store.

addisonjabout 3 hours ago

Congrats on the launch! Stream/event-based system are really powerful, but they are also just pretty complex, in no small part because the modeling of streams for most people today is really modeling Kafka topic/partitions which has a whole bunch of foot-guns and complexity. Making the individual stream really cheap and easy is a big simplification, especially if flexibly consuming a stream for both ordered and unordered use-cases is made simple. This looks to work for unordered, be curious to see how the managed dividing the work for the mentioned key-based ordering.

necubiabout 4 hours ago

I'm the author of the post and tech lead for K2. Happy to answer any questions!

e1gabout 1 hour ago

I'm doing some testing, and while my writes are <1s, I'm seeing E2E delivery latency of p95=2.5s / p99=7.5s. Is this roughly expected?

samtpabout 2 hours ago

After reading the post, still don't fully understand when you would use CF Queues vs K2. Can you help to elaborate a bit more?

necubi26 minutes ago

Sure! There are definitely some overlapping use cases, and we've seen folks using/abusing queues for use cases that are more appropriate to something like k2.

Queues are great when you have a unit of work that needs to be completed, retried, and tracked individually. For example, a shop might need to call a payment processor API that can fail or timeout, and retry it until it succeeds, while polling on the frontend for the state of that particular message. With a queue, you can insert a message tracking that payment, and have a queue processor that keeps getting sent it until it succeeds or has failed too many times.

In a queue each item is its own thing that's important to someone, and queues give you APIs to interact with that particular item.

K2 is for moving large volumes of data around. Pricing is per GB, not per message. Records are produced and consumed in bulk, and what matters is that all records are processed, but no one is querying the state of a particular record. K2 also supports multiple consumers for the same record, and long term retention. For example, all of your applications may emit events when things happen, and those events need to be read by an alerting system, a system that durably stores them, and a system that uses them to build ML features.

aniketsauravvabout 4 hours ago

Great product, congratulation on launch. Are you using this internally in any way?

necubiabout 4 hours ago

Yes! We originally built K2 to serve as the ingestion layer for Basin Pipelines [0], our stream processing product. We have a number of other teams building new products on top of it at the moment which I can't talk about yet :)

[0] https://developers.cloudflare.com/basin-pipelines/

Read the full thread on Hacker News →

50 comments

Related stories