OpenAI tests models by replaying past chats

OpenAI says it has deployed a new simulation approach to check how a model might behave before it’s released.

The core idea is simple: instead of guessing, the system replays previous conversations and runs them through a candidate model. OpenAI says this replay is done in a privacy-preserving way, so the goal is to learn from the pattern of real past interactions without exposing sensitive content.

OpenAI also reports that the method improved estimates of undesired behavior rates in GPT-5-series “Thinking” deployments. That matters because safety teams and engineers often need more than a surface-level read of how a model will act once it’s in the wild. Better estimates can help narrow down what could go wrong before the model sees broader use.

There’s another layer to it too. The company says the approach is meant to catch problems before rollout, including for agentic tool-using deployments. In plain terms, that’s the kind of setup where a model doesn’t just answer questions—it may take steps, use tools, and follow through on tasks. Those situations can make edge cases easier to miss if testing only covers basic chat behavior.

This kind of pre-release simulation doesn’t guarantee everything will be perfect. But it’s a practical step toward testing models against a broader slice of prior conversation patterns, while keeping the process focused on safety outcomes.

Source: OpenAI