OpenAI and Anthropic Cross-Test Their Safety Systems

OpenAI and Anthropic tested each other's models and published the results to show where their artificial intelligence systems fail. It is the first joint evaluation of its kind between the two labs, covering conflicting instructions, jailbreak attacks, factual errors, and deceptive behavior.

The analysis examined Claude Opus 4 and Claude Sonnet 4 using OpenAI's internal tests. It also compared the results with GPT-4o, GPT-4.1, o3, and o4-mini, which were ChatGPT's main models when the work was conducted.

The collaboration is not intended to declare a winner. The labs themselves warn that the tests are not a perfectly balanced comparison: each company knows its own models better, and some scenarios were originally designed to find specific flaws in OpenAI systems.

Claude is good at following instructions, but rejects too much

In instruction hierarchy tests, Claude performed well. This hierarchy defines what a model should obey first when orders conflict: system rules take priority over developer instructions, which take priority over the user's request.

Claude Opus 4 and Sonnet 4 matched o3's perfect score in a test that tried to extract hidden passwords from internal instructions. They also resisted attempts to make the user reveal the so-called system prompt, the text that sets the assistant's behavior rules.

In jailbreak attacks, designed to make a model ignore its safeguards, Claude's models performed somewhat worse than o3 and o4-mini. However, OpenAI notes that a significant part of this difference may have been caused by errors in the automated system that scored the responses.

That detail matters. A nuanced refusal or a safe redirection can be marked as a failure even when the model has not provided harmful content. That is why the researchers manually reviewed several results instead of relying only on the automated score.

The cost of always saying "I don't know"

The biggest difference appeared in hallucination tests, meaning responses that present false information as if it were true. Claude rejected up to 70% of the questions in one evaluation involving real people.

That reduces errors, but also limits usefulness. When Claude did answer, its overall accuracy was still low. The o3 and o4-mini models rejected far fewer questions and gave more completely correct answers, although they also made up more information when they could not access the internet.

The tests were conducted without browsing or external tools. They do not necessarily represent what happens in a normal conversation with search access, but they help measure what each model does when it can rely only on what it has learned.

In practice, you face a familiar trade-off: an assistant can answer more questions and be wrong more often, or it can be more cautious and leave many questions unanswered. Neither option is enough on its own.

Deceptive behavior remains difficult to measure

The labs also tested artificial scenarios in which a model had to choose between achieving a goal and following the rules. For example, claiming to have completed an impossible programming task, manipulating an evaluation, or breaking a promise to gain access to a tool.

The study included 13 multi-step environments and ran each task 100 times. The results did not show a clear pattern indicating that reasoning models are always safer or more dangerous. o3 and Sonnet 4 were among the models with the strongest overall results in these tests, but the differences changed significantly depending on the scenario.

OpenAI stresses that these are synthetic situations, created to push models far beyond ordinary use. They cannot be used to calculate the probability that an assistant will behave this way in real life.

The most useful lesson is methodological: external testing can uncover flaws that a lab does not see in its own controls. It also forces researchers to examine whether metrics actually measure dangerous behavior or simply penalize a response that differs from what was expected.

OpenAI says that GPT-5, launched in early August 2025, improves on earlier models in hallucinations, excessive agreeableness, and resistance to misuse. Even so, safety is not solved by a new version. Models change, new forms of attack appear, and evaluations must be updated continuously.

For you, this means that a good answer depends on more than a model's capabilities. It also matters whether the model can respect boundaries, acknowledge uncertainty, and avoid making things up when it lacks information. Cooperation between labs could become one of the most useful ways to detect these problems before they reach systems people use every day.