Better prompt caching for GPT‑6

Better prompt caching for GPT‑6 openai.com

3 points•mehrdadrad•8 days ago•2 comments•

#gpt #prompt #caching #prompt caching #better prompt #better prompt caching

2 comments

The biggest continuing limitation I see is that the cached input has to be at least 1024 tokens. This is terrible. It means a lot of good prefixes that are smaller will go uncached for no good reason. The threshold should have been 128.

OutOfHere8 days ago

Guide: https://developers.openai.com/api/docs/guides/prompt-caching

Read the full thread on Hacker News →

2 comments

Related stories