PSSA: a non-transformer language model written from scratch in Rust. Recurrent state-space layer, episodic memory, self-modifying weights, CUDA training, and a full terminal dashboard. - Sparticle6...
38 comments
For what I understand PSSA belongs to a line of research for LLMs with scalable architecture because you don't need to load the full KV in memory to generate a single token:
[0] https://aclanthology.org/2023.findings-emnlp.936/
https://arxiv.org/abs/2503.00392
https://papers.nips.cc/paper_files/paper/2023/hash/6ceefa7b1...
Edit: the claimed reason of "updates" also seems to not exist in code. If the author is going to use LLM for this, the very least they can do is to ask it to check properly before publishing.
> The implementation language is a detail, and a Python port is welcome.
I guess the post title could have dropped "in rust"
You can test the whole thing for free on a GPU with Google Colab. Test both the transformer and your new architecture on a larger dataset. Something that maxes out the GPU for an hour each run.
Also, the readme mentions keeping the same optimizer schedule which sounds nice at first but they are completely different architectures. The loss is high on the transformer, did you try raising the learning rate on it?
In general I’m interested in parameter efficient architectures. I don’t think transformers are optimal, and indeed many improvements have been made to vanilla transformers. But if you have an idea for something better you need to show it.
That said I don't know what's wrong with using a Rust AI framework like Candle.
Read the full thread on Hacker News →
Related stories
- PSSA: A non-transformer language model written from scratch in Rustai-news-site-cyan.vercel.appHacker News · 2 points · about 2 months ago
- Hacker News · 2 points · about 2 months ago
- Lobsters · 13 points · over 2 years ago
- Lobsters · 6 points · over 4 years ago
- Rust fact vs. fiction: 5 Insights from Google's Rust journey in 2022opensource.googleblog.comLobsters · 81 points · over 3 years ago
- rust-glancer: An alternative LSP for Rust with focus on low memory usagerust-glancer.github.ioLobsters · 25 points · 3 months ago