Field Notes on Scaling MoE Expert Parallelism with DeepEP

Lobsters·2 points·jado·8 months ago·nousresearch.com

A worklog documenting the journey of scaling expert parallelism to achieve high-throughput pretraining.

Read the full article at nousresearch.com →

#parallelism #expert #scaling #expert parallelism #deepep #notes #field #moe

Related stories

Scaling Laws for Neural Language Models (first "scaling laws" paper from 2020)arxiv.org
Hacker News · 2 points · 6 days ago
Show HN: Ledge.sh – Runnable Markdown Notesledge.sh
Hacker News · 76 points · 1 day ago
Ask HN: How have NLP researchers adapted since LLM disrupted their field
Hacker News · 1 points · 10 days ago
PostgreSQL to SQL Field Notes: Date and Timedevblogs.microsoft.com
Hacker News · 1 points · 5 days ago
Scaling to 100k Usersalexpareto.com
Lobsters · 18 points · over 6 years ago
Scaling out Postgres, Part 1: Partitioningpetrohi.me
Lobsters · 6 points · about 14 years ago

Related topics

notes45 field27 moe5 expert9 scaling32