Fathom: Per-query read depth for sparse decoding over offloaded KV caches

Fathom: Per-query read depth for sparse decoding over offloaded KV caches arxiv.org

When agentic sessions run to a million tokens with many sessions resident at once, the KV cache and the index that ranks it live in host memory, and the scan that ranks all n keys for a top-k step becomes the traffic…

1 points•vivekkalyanaran•1 day ago•0 comments•

#offloaded #depth #decoding #sparse #caches #fathom #query #sparse decoding

0 comments

No comments yet.

Read the full thread on Hacker News →

0 comments

Related stories