All of those browser and computer use agents (available from OpenAI, Anthropic, Meta, and X.ai already) represent an interoperability workaround already.

IMO it's a fundamental tension. AI does not fit well inside a product. It works best outside of it. But that means the product - not its functionality, but the business around it - becomes irrelevant. Which none of the product vendors want.

Also the motivation behind CLIs and SDKs, right? And the way agents write Python regularly, these days, I am hopeful what can be offloaded to deterministic systems will be.

I feel like this is a bit too-optimistically describing an outcome which is unlikely, given how consumers have been slowly losing the same war for decades already.

The elephant in the room and the really "tricky part" involves powerful advertising and copyright-holding companies. They will (continue to) interpret that kind of user-agency as an existential threat. Their businesses revolve around deciding and knowing what you actually see and do with your computer, and their toolkit isn't just technical, but also economic and political.

That's why we already have problems with DRM, "felony contempt of business model" under the DMCA, browser fingerprinting, and everybody surrendering to arbitrary JS code controlling their computer. We capitulate not just to see an article or talk to a friend, but also to avoid our technical details getting into a secret un-appeal-able "suspicious" blacklist.

Not sure if companies would ever want this. They rely on their dark patterns and intentional manipulations to get customers to do what they want.

That basically describes the Google shopping ribbon that appears with searches for products. When Google was good, you'd say a website doesn't exist if you can't find it in the 1st page of Google. The shopping ribbon provides offers from websites that aren't on the first page of my search result. Although I don't think you can pay without visiting the seller's website, this means that these are effectively stores that are more visible through the APIs they use to talk to Google's recommender engine. These suggestions are also no doubt personalized.

This also describes adsense, or Amazon or AliExpress, so it is clearly a successful model. If there is much left of other models. The only thing we have to hopeful about is that the interface is standardized so commodified SWE can be used to make your own Amazon or AliExpress.

I'm currently building something like this, basically converting every service to a standardized API that you can consume in a structured manner. Not focused on AI, but focused on normal users with special requirements on usability.

Basically the ad blocking approach has failed, and one needs to recognize that an allow-list based approach is needed when consuming internet-based offerings. Pick the cherries out of each website, leave the trash behind, and then build a custom per-website GUI around the things that you actually want to consume.

You shouldn't need LLM and AI to do such things. Even if such things might be useful to some people, I might want to use the API without LLM/AI and just to read the API documentation and then to make my own, and some other people might want to do similarly. In the specific case of e-commerce, I had been making my own specification of a file format for e-commerce (which is independent of the protocol), which is not intended to use with LLM/AI, but instead is intended to avoid many kind of dishonest business while also being flexible and that you could make and use your own software or some other implementaiton instead of being forced to use their UI/UX.

> from the company perspective how do they, for example, have a tool call that gives the customer a discount, without it getting used when it shouldn’t?

I might be misunderstanding your point but I think you're in agreement with the gp, as in they are arguing for a locked down interface behind which the AI sits, narrowed to use within whatever particular operations the user can access, as opposed to a wide open AI interface with access to any operation (in principle).

So in your example there's no way to ask the AI for any old discount because the interface for doing so is locked down to just the discounts you can access, although the AI may be able to apply a discount you didn't ask for dynamically, if allowed, to give the customer a better experience.

Of course both versions can be implemented securely, it's just probably in general a smaller attack surface if you provide a tighter interface first.

> have a tool call that gives the customer a discount, without it getting used when it shouldn’t?

without comment on the rest of your post, a lot of discounts are just dark patterns, buy 2 for $10 or one for $6 is one of the oldest dark patterns around, and a lot of people fall for it, when they really didn't need that much whatever it was they were buying. In fact there are laws outlawing these practices coming into force for various types of products (alcohol, sugar) .

Other types of discounts like loyalty are more vague personally I think are a minor dark pattern. Negotiated discount on bulk purchases b2b are a different class, but it could be argued that you should only get the discount that the seller saves on logistics.

I'm just pointing this out because its very normalized, 'signup and get 3 months free', 'buy 2 get one free' etc, people dont bat an eyelid at, the same as advertising , but they are all just the earliest versions of dopamine hacking and dark patterns that consensus now is coming around to say probably isn't the best thing when scaled beyond a simple one to one interaction. Why are snapchat dopamine hacking with streaks but your local coffee shop stamping your card for your 10th coffee free different?

> how do they, for example, have a tool call that gives the customer a discount, without it getting used when it shouldn’t?

One possibility is that the company and the user both have AI agents, and that these are able to negotiate with each other. Then I talk to my local customized agent, which knows my preferences, and it explains the required context if the company’s agent does something that won’t make sense to me. But the company’s agent is still privy to the details required to offer discounts, say.

You prevent abuse of the API by models the same way you prevent abuse by humans: you have server side checks.

They can want anything they like, if customers want to use agents and they don't provide APIs they will lose out.

> I don't know if his solution (""We should all go insane building interlocking evaluation and optimization pipelines, instead.") would be the long-term solution.

TFA could explain this one part better I think. The whole process proposed is real with lots of stuff in the literature, but by definition NOT a long-term solution in the sense that this process actually has no end. None of the approaches can get you a static answer for a moving target/platform.

So the "interlocking pipelines" for eval/opt would not be some stepping stone you can throw away, and they aren't something you'd run periodically. They'd basically be always on forever and spending 10-100x on system complexity and on tokens. Unless of course you're ready to freeze everything else about the whole system forever (including the backend model, and the whole nature of the "average" context window, the plugins/other prompts in the mix, etc).

Are most people in position to freeze requirements/platform forever? Not really, because if they were they'd just build a fairly static system and probably have limited use for AI. Are most people in a position to just casually accept 100x complexity/cost? Not really, that's the "it's not yet webscale" kind of advice that sounds good but isn't necessarily reasonable for average use-case or average org. Since specializing your own locale for this is usually a mistake.. the likely future direction is eval/optimization as a service

But I got real lucky once!

If turn the temperature down for tests, they won’t match production behaviors. If you turn it down too far in production, the output will just be bad.