Here’s a product feature which the world is going to need a whole lot more of over the coming months and years: default hard budget caps. I’m talking about the …
313 comments
It was, unfortunately, a nightmare. There were tons of tickets and even threats of lawsuits from customers whose service got cut off hard at the worst possible time due to organic growth/going viral/big event/nobody knew about the limit/etc. Not only did they lose all the leads and revenue they would have gotten from that bump, but they also pissed off their own existing users who suddenly couldn't use the service either.
Generally speaking, it's much better to use alerts instead of hard limit. Even in the worst case (hackers pwn your credentials and mine Bitcoin or whatever) the rest of your business is unaffected and you can negotiate with the billing department at comparative leisure.
This is all assuming you have humans operating the service. If you're letting AI agents yolo infra in prod, you have a whole series of new problems.
Probably for companies and opportunistic individuals with a high risk tolerance. As for me, just give me hard budget caps. It shouldn't be a choice between supporting one of the approaches, when you could let each client pick what they want.
Want an alert? Configure one.
Want a hard limit? Configure one.
(just make the configuration easy and visible)
1000 years ago, before CI/CD took over folk would have launch/go-live events, where teams would do a good checklist. There were pre-launch meetings to review, which included load checking.
Heck, have it do the equivalent of send people a DocuSign equivalent PDF to sign acknowledging the risk before enabling it. Would it still stop pissed off people? Probably not. Would it help with the risk of lawsuits, very possibly.
As a customer, the big number is scary and causes panic but for the provider… customers constantly fail to pay bills, providers are constantly writing off bills because it just isn’t worth the cost to chase, if a customer says “hey that usage was a mistake” it’s usually worth it to write it off to save the relationship. If you write off a big bill that wouldn’t have been paid anyway, the customer will perceive you as wonderful and benevolent and be loyal for life when they are ready to spend their money.
With tokens though the actual cost being incurred is much, much higher. If your service is just a wrapper around tokens, and a customer incurs $10k of usage that you paid OpenAI $5k for, it becomes much more difficult to write off.
Google Cloud is one of the few services that actually pursues unpaid bills even on their high margin services.
I recently loaded up on prepaid api credits for gemini and it somehow triggered some billing shenanigans in my linked accounts where it said I had a negative balance (from the credits), and they were going to discontinue my services. I had to reset some settings to sort it out, mainly using their chat ai and mine (because theirs gave me right status info, but wrong conclusions).
It's pretty messy across like aistudio.google.com, and their typical console, and google workspace business account. I'd be so fucked if they froze my account, I'd rather just pay openrouter to access credits in the future.
It works best when coupled with spam of other irrelevant alerts, to maximize the chance of higher earnings.
Queue lengths, request sizes, response wait duration, message payload size, authentication attempts, allocation rates -- there's always some upper number beyond which the system is so messed up you'd rather it crashes.
> An argument against this is that businesses don’t want their hosted applications to start throwing errors because some budget was exceeded. I expect that most businesses and individuals would prefer errors to a surprise $10,000+ bill.
Indeed. If you want a surprise $10,000 bill that's still not an argument against a hard cap -- just set it at $9,999,999 instead, or wherever you don't want the surprise bill. There's always a number that indicates something has gone insane. There's always a sensible upper limit to any operation.
Ooh. It could even literally be "hard limit is 3x of previous month's bill", and you can adjust the multiplier.
That's much less likely to be set, forgotten, and gotten limited by, but it's unlikely to bankrupt customers.
When video models just came out, I wanted to make a short clip. Gemini didn't have enough control back then, so I started in Studio. I had $10 on my account. Try, try, try, not good, retry, altogether maybe 20-30 retries with 4 choices for a 10 second video.
Wake up next morning to an email from Google - my Studio account is frozen because of negative balance.
I check it, it's at -$160. Not financial ruin, but a painful sum for a 10-second video I didn't even use in the end.
I don't think there was (or is) a setting in there that says "stop everything when I'm in the negative".
Almost a year later, it's still as bad. Trying to make some music via Lyria, I need to wait 10-12 hours for the API charge to show in my costs. Now I had the bitter lesson, I wait after every major novel experiment to see how much it costed me.
Coding is solved, my ass.
I can subscribe to your service for a specific fee on a monthly basis ($20/month say), and take the risk of losing that month's fee if your service or I make mistakes, or I can choose to drop another $20 mid-month, or anything for convenience.
Saying that the computer will "control" the billing and can run haywire tells me that I don't want to be anywhere near your pile of bad incentives.
It’s such a basic thing, not having it has to be deliberate to make you accidentally spend more than you’d like.
It was a necessary feature when I was learning. I was amazed when I moved up to the big guys and this wasn't an easy option.
This is unlikely. The big cloud providers routinely cancel bills based on accidental usage. The real explanation is just that it's technically difficult to calculate all usage in real time and shut down systems immediately as soon as a certain cap is reached.
That is not an evidence alone. They also don't cancel bills in many cases. Most people also just swallow the damage and pay. Like gym subscriptions where people just pay because cancelling was too hard and they postpone the canceling because it takes so much effort. And they also think it is their fault of not cancelling, because it took that much of effort.
> The real explanation is just that it's technically difficult to calculate all usage in real time and shut down systems immediately as soon as a certain cap is reached.
That is trivial to add these days. Cloud providers usually say that "we care about your business and don't want to make your systems go down unexpectedly if you did not pay your bills".
I was a member of AWS customer advisory board for years. Two dozen of largest enterprises and unicorns and so on. Nobody wanted the billing department "inline" of systems controls.
AWS provides all the controls you need to flip the switch yourself, and provides the CDK / TF / etc. patterns for it. Plenty GitHub projects to host your own light switch service.
Don't relinquish the switch.
And of course, two dozen enterprises trump hundreds if not thousands of other small businesses right? Did AWS think of, oh I don't know, a 'feature flag' or somesuch thing that enterprises and unicorns are so fond of to provide this feature to those who don't have VC money to flush?
But it still does and can only claim having those after building them.
Their customers have been asking for that hard cap feature for a long long time, not building that feature while saying customers have all the tools to build it themselves is just a deliberate misdirection at this point.
Read the full thread on Hacker News →
Related stories
- Hacker News · 3 points · about 2 months ago
- Lobsters · 10 points · over 4 years ago
- DEV Community · 12 points · 2 months ago
- Hacker News · 6 points · 2 months ago
- DEV Community · 0 points · about 2 months ago
- 'You Can See Everything' Trailervariety.comHacker News · 2 points · about 2 months ago