Training NanoGPT in 39.9 Seconds | Hacker News - But Better

Training NanoGPT in 39.9 Seconds hyperstition.cc

How I brought NanoGPT training from 73.889 to 39.914 seconds with ANVIL II, sampled softmax, and sparse updates to the bigram and trigram tables.

2 points•Mizza•2 days ago•1 comment•

#training #nanogpt #training nanogpt #softmax #brought #anvil #sampled #sparse

1 comment

Mizza2 days ago

Less mathy discussion on Twitter, that also covers some of the speedup that comes from outside the optimiser https://x.com/classiclarryd/status/2104688738354028962

Massive result

Read the full thread on Hacker News →

Related stories

OpenAI halts frontier-model training amid string of agent misalignment incidentsarstechnica.com
Ars Technica · 0 points · 2 days ago
Training NanoGPT in 39.9 Secondsgithub.com
Hacker News · 2 points · 2 days ago
Axon Training Pod (ATP) VR for training police officersaxon.com
Hacker News · 2 points · 9 days ago
We built DeepL's next-generation LLMs with FP8 for training and inference (2025)deepl.com
Hacker News · 1 points · 3 days ago
OpenAI pauses training of its ‘most capable models’theverge.com
The Verge · 0 points · 4 days ago
Microsoft exec called AI scraping the “largest theft of labor in human history”arstechnica.com
Ars Technica · 0 points · 13 days ago