VRAM for local LLMs: why memory bandwidth sets your tokens per second

DEV Community·2 points·axrisi·about 20 hours ago·dev.to

VRAM for LLMs is a bandwidth problem: every token streams the whole model from memory. Bandwidth per tier, the 20x offload cliff, and what fits in 16, 24 or 48 GB.

Read the full article at dev.to →

#bandwidth #llms #memory #vram #local #tokens #memory bandwidth #local llms

Related stories

Show HN: Zerikai Memory: local code-memory MCP server, now with Jevgithub.com
Hacker News · 1 points · about 14 hours ago
Show HN: Cross-platform governed memory for a fleet of agentsgithub.com
Hacker News · 1 points · 8 days ago
Show HN: Ox – A local agent that uses the internet for youopenox.ai
Hacker News · 2 points · 7 days ago
Show HN: SelMem – selective reconstructive memory for LLMsgithub.com
Hacker News · 1 points · 6 days ago
Show HN: StarSkirmish, an arena where LLMs create StarCraft Brood War botsstarskirmish.com
Hacker News · 3 points · 4 days ago
Show HN: Local-coder – Build a team of coding agents with local modelsgithub.com
Hacker News · 6 points · 10 days ago

Related topics

high61 video374 low40 vram4 transoceanic4 codec11