Scaling Laws for Looped Mixture of Experts | Hacker News - But Better

Scaling Laws for Looped Mixture of Experts arxiv.org

Looped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity expands total capacity at fixed active…

2 points•matt_d•about 4 hours ago•0 comments•

#experts #scaling #mixture #looped #laws #looped mixture #scaling laws #moe

0 comments

No comments yet.

Related stories

Scaling Laws for Neural Language Models (first "scaling laws" paper from 2020)arxiv.org
Hacker News · 2 points · 7 days ago
Field Notes on Scaling MoE Expert Parallelism with DeepEPnousresearch.com
Lobsters · 2 points · 8 months ago
Runtime Dynamic Compression of Mixture of Experts [pdf]timdettmers.com
Hacker News · 1 points · 8 days ago
Kalshi loses again as judges rule prediction markets must obey gambling lawsarstechnica.com
Ars Technica · 0 points · 4 days ago
Scaling out Postgres, Part 1: Partitioningpetrohi.me
Lobsters · 6 points · about 14 years ago
Scaling to 100k Usersalexpareto.com
Lobsters · 18 points · over 6 years ago