Meta says Prometheus cut training interruptions by 99%

Meta published a deep dive on Prometheus, its gigawatt-scale AI supercluster in New Albany, Ohio. The first phase is complete and was used to train Meta's Muse family of models. Meta is now scaling it toward roughly 1 million GPUs and says it already holds hundreds of thousands across several connected buildings.

The headline number is reliability. At the start in mid-2025, interruption rates peaked at nearly 1,000 events per 8,000 GPUs per day. A year of engineering has cut that by 99%. For the Muse Wonder training runs Meta reports more than 85% goodput for pre-training and more than 90% for reinforcement learning, and it has since raised the mean time between interruptions to more than 54,000 GPU hours.

Meta credits embedded Nvidia engineers, screening of more than 600,000 NVLink connections, fast firmware rollouts and spare GPUs inside each rack. The cluster runs two Nvidia platforms now with Vera Rubin arriving soon.

A second site, Hyperion in Louisiana, is being designed with these lessons, and Meta says it will pay the full cost of its grid connection.

Our take

At this scale, the winner is not the one with the most GPUs but the one whose GPUs keep working. Reliability engineering is turning into a competitive moat.