S3 Is the Future, S3 Is the Past

S3 Is the Future, S3 Is the Past btrblocks.com

Better building blocks for the cloud: storage infrastructure rebuilt for modern SSDs, fast networks, and AI-era workloads.

79 points•tkhattra•5 days ago•119 comments•

#past #future #modern #cloud #infrastructure #blocks #storage #rebuilt

119 comments

simonw3 days ago

One thing I find notable about S3 today is that, while it used to drop in price reasonably often, there hasn't been a price drop in a full decade:

  2006-03-14  $0.150/GB-month
  2010-11-01  $0.140/GB-month
  2012-02-01  $0.125/GB-month
  2012-12-01  $0.095/GB-month
  2014-02-01  $0.085/GB-month
  2014-04-01  $0.030/GB-month
  2016-12-01  $0.023/GB-month

Today it's still $0.023/GB-month.

OrangeDelonge3 days ago

I guess inflation is the price reduction we get. 23 cents in 2016 is 32 cents in 2026

magicalhippo3 days ago

Just be happy they keep the GB as large as they used to...

someonebaggy3 days ago

And that's by official inflation numbers. If you go by how much prices of food and rent increased i think you get a number more like 50 or 60 cents

charcircuit3 days ago

Egress pricing of $0.09/GB has been around for over a decade too.

For reference transit cost has dropped over the years and is now about at $0.000247/GB or free if there is a peering agreement with the network the data is sent to.

someonebaggy3 days ago

It's the lock-in cost. They don't want you to move all your data out, they want it to remain trapped. Europe forced them to allow a one-time free exit, but you have to negotiate it with their support, so they're hoping nobody uses it.

otterley3 days ago

Egress pricing pays for the AWS network infrastructure that is incredibly reliable (hardware failures happen regularly yet almost no one notices) and allows it to operate at scale sufficient to absorb even the largest DDoS attacks.

This isn’t just AWS BTW; all tier 1 cloud providers recoup their costs this way.

kondro3 days ago

That's true for the base S3 product, but there are a lot more storage tiers than there used to be with cheaper pricing. All the way to $0.99/TB for Deep Archive.

mey3 days ago

Not that deep archive isn't a valid option for the appropriate work load, but it has different effective costs.

AtlasBarfed3 days ago

Well there was also a period of low inflation in them during that time.

God why am I defending Amazon?

layoric3 days ago

This is absolutely the case for nearly all of AWS products. Some new services have filled lower price gaps, but I previously looked at EC2 and other service prices in the past and late 2016 is where it all seemed to stop getting price drops. The M5+ upgrades for example all came with price increases along with the performance gains.

KAdot3 days ago

Every new EC2 generation is typically slightly more expensive, but the gains in the compute performance are 15-25% or even higher depending on your workload. The same dollar buys you a lot more compute power than a decade ago.

439203 days ago

Even prior to the current AI-driven hardware shortages, hard drive prices stopped decreasing in about 2020: https://backblazeprod.wpenginepowered.com/wp-content/uploads...

richieartoul3 days ago

Nice article. I agree that it is a bit of a shame that everything is forced to be so S3-centric (and I say that as someone whose work helped motivate a lot of people to do that), but right now its an unfortunate reality of running software in the cloud because cloud networking and SSDs are so expensive that you really are required to use S3 if you want a system that can handle "big data" scale workloads cost effectively

zokier3 days ago

Right now it is not even clear how to interface with SSDs even on a single host, there has been all sorts of attempts to move away from the traditional plain block device model. NVMe has extensions for KV, ZNS, and FDP, all which offer different characteristics. And then there are of course open channel SSDs and some others too. I kinda expect the future foundational IO interface to be more S3-like than block-device or unixy filesystem-like.

jauntywundrkind3 days ago

I really wish reviewers would harp on FDP (Flexible Data Placement) and perhaps KV support. FDP supposedly somewhat ate ZNS as a spec, allegedly, but there might be gaps, reasons to keep ZNS.

There's only a small little mention, if we are lucky, on the couple drives that have it (expensive enterprise flagships). It should be a regular sticking point, whether it's there or not. Without pressure it's not going to get regularly available, it feels like.

FDP is so simple. Declare a number for what pool of data you want to write into. Data of the same pool gets written to the same storage such that you can wipe it latter together. It has huge wins though against write amplification! Massive wins. For so close to free.

Some day I want to own a FDP drive. And then I can finally start using the tokio/io-uring support that I contributed! https://github.com/tokio-rs/io-uring/issues/380

The NVMe-KV is more radical. Still worth putting some pressure on, but your drive as KV, as object store, feels harder. Side note, really enjoyed this ceph nvme-kv offload post thing, my favorite tech write up in a while! https://ceph.io/en/news/blog/2026/for-whom-the-door-bell-tol...

someonebaggy3 days ago

FDP relies on putting logic on the drive side, then they can upcharge for drives with this feature and still give you limited control. If you go the other direction instead, you have MTD devices which gives the host system kernel full control over data placement, page erasing and error correction, and for this reason they need specialized filesystems. These devices usually aren't attached over PCIe as they use controllerless raw flash interfaces instead, but in principle they could be.

someonebaggy3 days ago

Why would it be S3-like? S3 is a very general abstraction on storage. If there's room below the current abstractions, it's below, not above - Linux has drivers for raw flash devices (mtd devices) which gives Linux full control of the program/erase cycle and responsibility for wear-leveling.

thadt3 days ago

I would assume “S3-like” in the sense that objects are immutable, large writes, separate metadata storage, etc. Patterns that fit modern SSDs better than the abstractions of block devices.

hadlock3 days ago

I finally stopped using (S)FTP when i realized that all the ftp clients now have native support for S3. It turns out if you have a business partner who wants to use sftp, 99.9% of the time of they upgrade their client, the upgrade has S3 support, and then you can just use modern tooling on your side. And when they finally automate, they can also use modern tooling.

oasisbob3 days ago

DDEX choreography, which is used to distribute musical recordings and other assets within the music industry to sites like Spotify followed this same evolution.

The standards say SFTP. Most everyone ignores that part and have been using S3 buckets by bilateral consensus instead for years.

jiggawatts3 days ago

Had the same experience with trying to use the built-in SFTP support in Azure Storage accounts.

It turned out that all of our peers supported blob storage better than SFTP, which has some “show stopper” problems like forced outages caused by mandatory host key rotations.

0xCMP3 days ago

The unfortunate part of that graph is that I think it's not updated for today's prices given the memory shortage/crunch we're experiencing that's driven up the prices for all kinds of memory.

But also while it should be faster than it is, how would an S3 designed around SSDs look differently to an API user? I would think the API is basically the same.

pugz3 days ago

That's S3 Express One Zone. Directory buckets are SSD-backed and regular buckets are HDD-backed. Some of the API differences off the top of my head:

- Directory entries are no longer returned in sorted order in ListObjectsV2

- There's an AppendObject API

- There's a RenameObject API

WatchDog3 days ago

It's hard to imagine how these API differences can be explained by the different underlying block device. I don't see any good reason you couldn't support these operations on a HDD.

I suspect it's more to do with the fact that with One Zone is a clean rewrite of large parts of the application stack that makes up S3.

S3 is made up of hundreds of microservices[0], there probably isn't anyone at Amazon that actually understands the whole system. Refactoring it to support these features probably requires coordination between a lot of different teams. They might have petabytes of metadata, making a change to how metadata is persisted probably requires a massive risky data migration.

[0]: "All in, S3 today is composed of hundreds of microservices" - https://www.allthingsdistributed.com/2023/07/building-and-op...

alexjurkiewicz3 days ago

It would probably look a lot like existing S3 concepts that have explicit hot and cold tiers. For example Intelligent Tiering, or Glacier.

Read the full thread on Hacker News →

119 comments

Related stories