Complex KDA:Understanding and Enhancing the Expressivity of Kimi Delta Attention | Hacker News - But Better

Complex KDA:Understanding and Enhancing the Expressivity of Kimi Delta Attention arxiv.org

Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that composing two delta-rule transitions…

2 points•jul8234•9 days ago•0 comments•

#delta #expressivity #attention #complex #kda #understanding #enhancing #kimi

0 comments

No comments yet.

Related stories

Delta (zed.dev) – Help understanding ToSzed.dev
Hacker News · 2 points · 10 days ago
Show HN: MoA – Attention kernel proven minimal before code was writtengithub.com
Hacker News · 1 points · 9 days ago
Inferact's Kimi K3 Megakernel hits 700 tok/s on TPU, Beating GPUinferact.ai
Hacker News · 5 points · 7 days ago
Newborn girls and boys show different attention patterns within hours of birthmedicalxpress.com
Hacker News · 8 points · 8 days ago
Sex differences in attention are present at birthcam.ac.uk
Hacker News · 3 points · 8 days ago
Transformer: A Novel Neural Network Architecture for Language Understandingresearch.googleblog.com
Lobsters · 4 points · about 9 years ago