Scaling Marin's Open LLM Pretraining

Marin started as an academic project with one full-time researcher, training medium-sized language models on Google’s TPU Research Cloud. This month, the team finished pre-training a 535B-parameter mixture of experts model on 18T tokens, using GPUs from the Huang Foundation, in the largest live-streamed pre-training run to date. This required scaling everything: compute, data, focus, and people. We ran thousands of smaller models to understand scaling, built systems for turning trillions of raw tokens into training mixtures, learned how to train sparse models efficiently, built infrastructure for running reliably across large and heterogeneous clusters, and found people and organizations excited to support open development at this scale. Open Athena is hosting this event at Laude Labs to share the research, engineering, and organizational work that went into scaling Marin, and what we've learned (and are still learning) from the run. Light bites and drinks will be served for the evening. Agenda: 6:00pm - Door Opens 6:20pm - Welcome Remarks with Percy Liang 6:30pm - Fireside Chat with David Hall and Andy Konwinski 6:45pm - Presentation by Will Held 7:20pm - Freeform Conversations 8:00pm - Event Ends We welcome the greater academic, engineering, and research community to join us.