AI reduces code mastery in study

An Anthropic study suggests that using AI to program can speed up a task, but make it harder to learn what you have just done. Participants who relied on an AI assistant performed worse when tested on how well they understood the code.

The difference was clear: people who used AI answered 50% of the questions correctly, compared with 67% of those who programmed without assistance. The study involved 52 software engineers and measured their understanding just minutes after they finished the task.

What the study measured

The participants, mostly junior developers, had experience with Python but were unfamiliar with Trio, a library for creating programs that perform multiple tasks at once. They had to learn how to use it and program two new functions.

Half worked with an AI assistant integrated into the programming platform. The tool could read their code and generate a complete solution if asked. The other half completed the tasks by hand, using the same introductory material.

Afterward, everyone answered a questionnaire covering four skills:

• Debugging: finding errors and understanding why they occur.
• Code reading: understanding what a program does.
• Code writing: choosing or producing the correct way to solve a problem.
• Concepts: understanding the library's principles and design.

The test focused especially on debugging, code reading and concepts. These are the skills someone needs to review AI-generated code and spot problems before putting it into production.

Faster, but not significantly so

The group that used AI finished the task about two minutes earlier, on average. However, the difference was not large enough to be considered statistically significant.

The strongest result appeared in learning. The AI group scored 50% on the questionnaire, compared with 67% for the group that programmed without help. Anthropic describes the difference as equivalent to nearly two academic grades and says it was especially large on the debugging questions.

That matters because generating correct code in the moment does not guarantee that you will know how to review it later. If an application fails, someone has to identify the problem, understand its consequences and decide how to fix it.

Not every way of using AI produces the same result

The study does not show that AI necessarily reduces learning in every situation. It also found that some participants used it to understand the work, not just to delegate it.

The best results appeared among those who:

• Asked for explanations after receiving code.
• Asked questions about the concepts behind the solution.
• Combined code generation with comprehension checks.
• Tried to solve errors on their own before asking for help.

By contrast, participants with the worst results tended to hand over the complete writing or debugging process to AI. Some finished faster and found almost no errors, but they also did not have to confront the problems that help reinforce a concept.

One case sums up the difference well: several participants spent up to 11 minutes, 30% of the available time, writing queries for the assistant. Asking questions was not the problem by itself. The difference was whether the questions sought an immediate solution or an explanation that allowed them to keep thinking.

What changes for you

If you use AI to learn programming, copying a working answer can make you faster, but it does not necessarily give you mastery. For repetitive tasks or skills you already control, assistance can save time. When you are learning something new, the cost may be losing the practice you need to spot errors later.

A safer way to work is to ask AI to explain the code, compare several solutions, try to predict the result before running it and solve some errors without help. You can also ask for questions or hints instead of the complete answer.

For companies, the finding raises a concrete problem: junior developers may produce more code while developing less ability to supervise it. That is especially concerning when AI writes an increasing share of systems that affect customers, money or security.

The study is preliminary and has important limitations. The sample was small, the questionnaire was administered immediately after the task, and it does not show what will happen to learning over the long term. It also cannot tell us whether the same effect appears in other professions or with more autonomous tools, such as agents that modify an entire project.

The central idea is not choosing between programming with AI or without it. It is distinguishing between using AI as a shortcut and using it as support for learning. AI can accelerate skills you already have, but it is still unclear how to prevent that speed from slowing the development of the skills you will need to check, correct and control what it produces.