Software Development

This post draws practical lessons from How AI Impacts Skill Formation by Judy Hanwen Shen and Alex Tamkin at Anthropic. For a summary of the study, see AI Makes You Faster But Dumber?. For a critical analysis, see The Skill Tax of AI Coding. The Two-Line Summary Anthropic’s study found that developers who used AI assistance scored 17% lower on a mastery quiz, but developers who used AI to ask questions and build understanding scored nearly as well as those who coded without AI at all. ...

This post is a critical analysis of How AI Impacts Skill Formation by Judy Hanwen Shen and Alex Tamkin at Anthropic. For a summary of the study’s findings, see my companion post AI Makes You Faster But Dumber?. Why This Study Matters Let me start with what’s genuinely praiseworthy: Anthropic, an AI company that directly profits from increased AI usage, published a rigorous study showing a meaningful downside of using their own technology. That takes institutional integrity. The study itself, a pre-registered, randomized controlled trial with screen recordings and careful evaluation design, is methodologically stronger than most work in this space, which tends toward observational surveys or anecdotal evidence. ...

This post is based on How AI Impacts Skill Formation by Judy Hanwen Shen and Alex Tamkin at Anthropic, and the accompanying blog post. The Headline That Writes Itself “AI makes coders dumber.” You can practically hear the hot takes forming. And on the surface, the numbers seem to support it: in a randomized controlled trial, developers who used AI assistance scored 17% lower on a mastery quiz compared to those who coded by hand. That’s the equivalent of nearly two letter grades. ...