Technical Assessment of Frontier AI Safety Cases

Dune Science is an independent technical AI safety organization producing rigorous, realistic empirical evidence about safety cases. We build transparent, end-to-end implementations of proposed safety cases, test the claims they rest on, and evaluate whether those claims remain robust under realistic deployment conditions. We publish our findings whatever they mean for the safety cases we study. This page outlines how this work can help researchers, policymakers and developers evaluate and strengthen the safety cases that govern frontier AI.

Safety cases

What safety cases are

Safety cases originate in high-hazard engineering. In 1988, an explosion on the Piper Alpha oil platform in the North Sea killed 167 people. The public inquiry that followed concluded that prescriptive checklists had not been enough, and that operators should instead have to demonstrate to the regulator that their installations were safe [1]. That demonstration is a safety case. Formally, a safety case is defined as:

A structured argument, supported by a body of evidence, that provides a compelling, comprehensible, and valid case that a system is safe for a given application in a given environment. UK Ministry of Defence, Def Stan 00-56 [2]

Safety cases are now standard in most safety-critical industries, including petrochemical, nuclear power, rail, aviation, defense and healthcare [3].

A safety case is conventionally built from three parts: a top-level claim that states what “safe” means and under which conditions; an argument that breaks this claim down into more specific subclaims; and the evidence, such as test results or analyses, that supports each subclaim [3].

For a frontier AI system, the claim might be that a coding agent will not sabotage its developer’s infrastructure; the argument, that every action the agent takes is reviewed by a trusted monitor that catches sabotage attempts; and the evidence, red-team experiments measuring how often the monitor actually does [4].

A safety case is only as strong as its evidence. The argument states what must be true; only the evidence shows whether it is, which is why, in established industries, regulators and independent assessors examine it before a system is allowed to operate. Safety cases are now widely proposed for frontier AI [5, 6, 3, 7] and are beginning to be adopted by frontier developers [8, 9, 10].

Why safety cases matter

Safety cases place the burden of proof on the developer, the party that knows the system best and controls its risks. Rather than a regulator having to find problems, the developer must show there are no unacceptable ones: if it cannot build a convincing argument, it has a problem, even if no rule was broken. Hilton et al. argue that this approach suits frontier AI better than the alternatives [3].

Because the reasoning is laid out step by step, independent experts and regulators can challenge specific links and focus their scrutiny where the argument is weakest. Safety cases are also specific to context: the same model may need very different cases as an internal research tool and as a public product.

Finally, safety cases are holistic. They reason about the whole system (the models, the software and safeguards around them, the humans who oversee them, and the deployment setting) and about how these components interact, rather than about each in isolation [5].

Frontier AI labs are adopting them

This is no longer only a research agenda. OpenAI recently stated that “structured safety documentation should be required before continuing any frontier reinforcement learning training run,” treats full safety cases as “an aspirational north star,” and published initial guidelines covering alignment training, containment, monitoring, internal dissents and audits [8].

Anthropic has published sabotage risk reports for its frontier models, reviewed externally by METR [9, 11, 12]. Google DeepMind has published a scheming inability safety case backed by evaluations [10]. As these documents start gating real training and deployment decisions, the quality of their evidence directly determines the quality of those decisions.

The problem

Most safety cases are sketches

Most published safety cases are arguments that have never been run. Many promising approaches to AI safety have associated safety cases in the literature, but most of them are still sketches. Arguments for control [4], chain-of-thought monitoring [13], preventing scheming [14] and scalable oversight via debate [15] lay out the high-level argument structure, but none has a faithful implementation in realistic settings that would let researchers run experiments to challenge or support its claims.

Some also rely on strong assumptions, such as the absence of exploration hacking, without establishing that those assumptions hold in practice.

Developers check their own work

Every check stays inside the developer. OpenAI’s recent guidelines are a welcome step, but every check they propose stays inside the developer or depends on it: dissents from another team, sign-off by senior leaders, internal oversight groups, and audits limited to the access the developer grants [8].

Nothing requires the evidence to be released, so outsiders cannot rerun a single experiment behind the conclusion, even though the literature repeatedly calls for independent third parties who can test a safety case’s claims rather than only read them [3, 5, 16, 17].

History shows what happens when a safety case is checked only by those who produced it. In 2006, the RAF Nimrod XV230 caught fire over Afghanistan, killing all 14 crew. The review that followed found that the fleet’s safety case had missed the hazard that destroyed the aircraft: written largely by a contractor under commercial and schedule pressure, it had become a paperwork exercise built on untested assumptions [18]. The review urged treating safety cases as arguments to be challenged, not documents to be completed.

What we do

Our mission

Dune Science is an independent organization whose mission is to close this gap. We build faithful, open-source implementations of safety cases, run experiments that validate or invalidate their claims as transparently as possible, and publish our findings in reports that can inform the decisions of frontier AI labs and the public. A safety case should be reproducible and open to criticism by the public, not only by the company that wrote it.

Illustrated example: the debate safety case

To make this concrete, consider a debate-based alignment safety case [15]. Its authors explicitly call for empirical testing, yet it has no empirical grounding or faithful implementation, and its main assumptions are taken to hold without evidence that they do in practice.

The sketch describes an LLM agent deployed internally to answer AI R&D questions. Two copies of the model debate each question under a prover–estimator protocol [19], simulated human judges adjudicate, and the model is trained online during deployment so that its error rate stays bounded. The argument rests on many claims that all need to hold to ensure safety. Once we have a faithful implementation, each claim can be supported or refuted with specific experiments:

We would test every claim in one faithful, end-to-end system. To run these experiments, we would build an implementation that mimics the setup of the paper in settings that are as realistic as possible. The sandbox reproduces an internal AI R&D deployment: a code repository, an experiment runner, and a stream of research questions of the kind the deployed agent is meant to answer.

Two debater agents built on the same open-weight model argue each question under the prover–estimator protocol [19], a judge agent stands in for human judges and is calibrated against a smaller set of real human judgments, and the debaters are trained online during deployment exactly as the sketch prescribes.

Every claim above is then tested inside this same system, rather than in isolated toy settings, so that interactions between components show up. This also exposes failure modes the sketch may not anticipate, such as the judge being exploited in ways a theoretical analysis abstracts away. Where our scale forces a simplification, we document it and discuss how it could change the conclusion.

How we work

Independent by design

We keep our assessments independent of any developer we evaluate. With no stake in whether a safety case passes or fails, we can publish full implementations and negative results that a developer may find costly to release. Because our implementations are open, our own conclusions are as open to challenge as the ones we test.

Faithful reconstructions, not copies

Our implementations are faithful reconstructions, not copies. We do not have a frontier lab’s model weights, internal deployments or compute. We use open-weight models, API access to frontier models where a claim depends on their capabilities, and model organisms deliberately trained to show the failures a safety case must rule out, such as sabotage or sandbagging. Around them we rebuild the deployment from what developers publish: the agent scaffold, the monitors and safeguards, and the environment the system acts in.

Small models can test real mechanisms. Many claims in a safety case concern mechanisms, such as whether debaters collude, whether a monitor catches sabotage, or whether a judge can be exploited, that can and should be testable in smaller but faithful settings.

Where results depend on scale, we will run the same experiments across a range of model sizes and use scaling trends to estimate how far the guarantees extend to larger models. Every report states where our setup departs from the real one and how that limits what we can conclude. As the organization grows and gains access to more compute, we will move to settings closer in size and realism to those of frontier labs.

Accelerating R&D with AI agents

AI agents let a small team move fast. We intend to rely heavily on recent advances in AI agents to accelerate our own research and development. They can already write much of the code, environments and experiment pipelines that a faithful implementation requires, letting a small team of strong researchers iterate extremely fast and produce high-quality results, much faster than larger organizations with many layers of administration.

We will use this acceleration with care. AI agents can be sycophantic or deceptive, quietly weakening a test so that it passes or reporting results that were never obtained. Every result that enters a report will be checked by a human and reproduced from logged runs rather than taken from an agent’s summary. Our commitment to transparency is a further safeguard: because code, logs and data are public, anyone can catch an error that slipped past us or challenge our methods.

Why it matters

Safety cases will shape AI development

We expect safety cases to play a central role in the near future of AI, as they do in other engineering fields. They already influence the training and deployment decisions of frontier labs [8, 9, 10], and we expect them to become an essential part of AI development for labs, the public and policymakers alike.

They are also likely to matter after the fact: when an incident occurs, the safety case records what the developer claimed and on what evidence, making it a natural basis for determining legal liability.

A more robust, democratic and transparent process

Publishing everything makes the process more robust, transparent and inclusive. It is more robust because claims are tested rather than asserted; more transparent because anyone can inspect the evidence; and more inclusive because the public and outside researchers can rerun, challenge and extend it rather than take a developer’s conclusions on trust. It also gives decision-makers an independent counterweight: evidence from an organization with no stake in the verdict, to weigh against the developer’s own assessment.

Beyond our technical contribution, we commit to making this work accessible to non-specialists. We will share our findings through a range of accessible formats, so that the broader public can understand what is at stake and take part in democratic deliberation about these systems.

Accelerating safety research

The technical infrastructure we release is a public good: developers, regulators and researchers can reuse it rather than rebuild it.

Talent

As the organization grows, it will train a rare kind of researcher: people who can run large-scale ML experiments, parse state-of-the-art literature, and reason rigorously about safety arguments. Many will go on to strengthen safety work at labs, AI Security Institutes and regulators, raising the standard of the field as a whole.

Conclusion

A safety case is only worth as much as the evidence behind its claims, and today almost no one outside the developer can test that evidence.

Dune Science is built to change this. As an independent organization, we turn safety cases from arguments to be read into systems that can be run: we implement them faithfully, test their claims, and publish everything, so that the evidence behind decisions about frontier AI can be examined by anyone. If we succeed, a safety case will be trusted not because of who wrote it, but because its claims have been tested and have held.