Scaling Marin's Open LLM Pre-Training
At Open Athena's first community meetup, Will Held walked us through Marin's pre-training process, particularly the hero training run, which wrapped up this week.
Post-Training in the Open
We’re excited to share more about our progress post-training our mixture of experts models. Our latest milestone is the release of Marin 67B-A2B “Snowball” v0.1, a mixture of experts model trained on ten trillion tokens. We’ve built a fully open post-training stack, code, and data, and post-trained a model to GPT-OSS-level quality, putting Snowball nine to fifteen months behind the open-weights frontier for similarly sized models.
There is no posttraining data commons
The LLM revolution ran on a pretraining commons that humanity built for other reasons. Posttraining consumes tasks, not documents, and no such commons exists. If the open models community wants one, it will have to be built deliberately.
Meet our team: Tim O'Donnell
In this Q&A, Tim O’Donnell reveals his most memorable AI turning point, explains how Open Athena acts as a “third place,” and shares his favorite town for peaceful bike rides.
Expert parallelism at hero scale
Expert Parallelism made a 535B-A23B model practical to train. We compare FSDP, fixed-capacity EP, and results from a scaling ladder.
Open Athena selected as an ARIA Activation Partner
Open Athena has been selected as one of ARIA's new Activation Partners, pairing our engineers with ARIA-funded research teams to build open scientific models, datasets and benchmarks.
Meet our team: Jesse Rusak
In this Q&A, Jesse Rusak talks about the evolution of AI, the ambition of OA's mission, and how elephant seals are helping climate research.
Curating 25 Trillion Tokens for LLM Pretraining
How Marin deduplicates and decontaminates permissively licensed data, scores quality, tags topics, and learns how often to sample each data bucket.
Marin 535B-A23B launch note
A look back at Marin's first year at Open Athena and forward at the 535B-A23B hero run, the largest model the team has ever trained.
Marin's 535 billion parameter model training run launched with support of The Jen-Hsun and Lori Huang Foundation GPU gift
Open Athena announces the launch of Marin's largest training run yet—a 535B parameter large language model—with a generous gift of compute from The Jen-Hsun and Lori Huang Foundation.
Meet our team: Alex Merose
In this Q&A, Alex Merose lays out the machine learning project that changed his career, the value of doing basic research in the open using tools at the frontier of AI, what Open Athena has in common with Bell Labs, and why we should think like mapmakers rather than soldiers.
Meet our team: Will Held
In this Q&A, Will Held discusses AI’s bitter lesson, why he thinks laptops are dead in the age of agents, and the linguistic history lesson hiding in the phrase “pardon my French.”
A 1B standard Transformer rivals Evo 2 40B on variant effect prediction
How MarinDNA combined data curation, hyperparameter transfer, scaling, and data-mixture experiments to build an efficient 1B alignment-free genomic language model with balanced performance across genomic regions.
Scaling Samudra: Our journey towards better ocean emulators
We scaled Samudra, our neural ocean emulator, to train on 16x more data on the same hardware — cutting training time from four days to four hours. We found out systems performance tuning is a humbling process.
I argued with the father of open source for 2 years. Now the AI fight is the same — only bigger.
David Siegel, founder and chairman of Open Athena, writes in Fortune about the importance of open source in AI.
Research Software Engineering in the Age of Agentic Tooling
With the rise of AI coding agents, the role of research software engineers is changing. Instead of building libraries and using them to write bespoke code, RSEs must shift their focus both upstream and downstream of the software creation process.
Meet our team: Betsy Cannon
In this Q&A, Betsy Cannon explains what drew her to this work, her cookie-related introduction to LLMs, and what a project on electron density can learn from one about ocean modeling.
Open Development of Frontier AI
Open development is key to making frontier-level artificial intelligence legible, reusable, and public.
Cluster Scheduling with Iris
Training a frontier-level LLM requires significant resources across a variety of providers and accelerator types. Our Iris scheduling system has allowed us to effectively make use of these precious resources using a custom global scheduler; in the months since rollout, our sustained concurrent TPU usage has roughly doubled.
Preparing for the AI Future with Ethics in Mind
At a recent panel discussion, our COO and CSO Jared Crooks explained why openness is key to understanding AI, and dug into the importance of embedding ethics in this new technology.
Improving our LLM Pretraining Efficiency
How Marin pretraining became more efficient through Mixture of Experts, higher expert sparsity, MuonH, PKO, and routed expert normalization.
Scaling Laws That Extrapolate 300× Past the Fit
Delphi is an open scaling suite ranging from 3e18 to 1e23 FLOPs. A pre-registered forecast from its scaling law predicted the loss of the largest run within 0.2%, extrapolating 300× past the largest run used in the fit.
Mixture of Experts Quantile Balancing: Validated at 32B-A5B (1e22 FLOPs) Scale
Quantile Balancing (QB) is a hyperparameter-free load balancer for Mixture of Experts models, introduced by Jianlin Su. We validated it on a 32B-A5B (1e22 FLOPs) Marin run over 326B tokens: zero hyperparameters, zero loss spikes, and no need for leading dense layers, auxiliary losses, or capacity overload factors.
Problems with Chinchilla Approach 2
The Chinchilla paper's "Approach 2," fitting parabolas to IsoFLOP curves, turns out to have some subtle biases that can add up. We show these can lead to non-trivial errors in compute-optimal allocation (around 6.5% of total compute for Llama 3, worth over $1M in GPU time), especially when IsoFLOP grids aren't perfectly centered or symmetric. The paper also proposes a reparameterization of "Approach 3" that makes direct parametric fitting simple and stable. You can even run it in 70 lines of JavaScript.
