Dwarkesh Podcast ·

Analysis: from 2019 to 2025, gains in pretraining compute efficiency came mostly from data improvements rather than model improvements

Breaking down 6 years of pretraining progress into data vs model improvements

Lead Source

Dwarkesh Podcast — Analysis: from 2019 to 2025, gains in pretraining compute efficiency came mostly from data improvements rather than model improvements ↗

How this story grew

Coverage · 0 Discussion · 0

Sep 9Sep 10

Discussion

Rohin Govindarajan

@dwarkesh_sp @arimorcos @dorialexander @sarahookr

Related stories

South China Morning Post: China says its AI compute capacity rose 177% YoY to 2,185 eflops by the end of June, and is targeting 9,800 eflops by 2030 via ~$532B in IT infrastructure spend

The Register: OpenAI says new monitoring and security safeguards will add 20% compute overhead to monitored inference workloads; costs will not be passed on to customers

The Register: Revolut says it cut AI coding inference costs by ~90% in less than a year by making self-hosted GLM models its default over commercial tools like Claude Code

Financial Times: AlphaSense: mentions of open models in US earnings calls and conferences rose 6x YoY in August and September; open models hit 56% of Vercel tokens in August

Microsoft AI: Microsoft unveils MAI-Base-2, a MoE model with 2.8T total and 80B active parameters, claiming 4x-12x more pretraining FLOPs-efficiency than DeepSeek-V4-Pro-Base

Discuss with AI

ChatGPT ↗ Claude ↗

← Back to TechSnif