* Edit, I did however just notice literally all of this is just another vibeslopped banger, so I am not quite sure how much fun there really is, feels more like coding for the sake of keeping the wheel turning.

I extensively used minGPT for home experiments on transformer architecture. It is great for learning!

However, if you want to scale the experiments up at home you need to go faster. Karpathy made optimized https://github.com/karpathy/nanoGPT, but it is tuned for "8XA100 40GB node in about 4 days of training".

13s is a bit overkill here (my machine builds that project in 30s). But it gives some space for experimentation with architectures that don't have optimized primitives.

But it is a byte predictor. You can train it on any file.