Click to Play Audio

Download PDF

Click to Play Audio

Download PDF

Click to Play Audio

Download PDF

Underwriting
Superintelligence

Rune Kvist, Rajiv Dattani, Brandon Wang

July 15, 2025

About us

About us

Insurance Unlocks Secure AI Progress

We’re navigating a tightrope as Superintelligence nears. If the West slows down unilaterally, China could dominate the 21st century. If we accelerate recklessly, accidents will halt progress, as with nuclear power.

Insurance, standards, and audits together create skin in the game for quantifying, communicating, and reducing AI risks so we can balance this tightrope. We call this the “Incentive Flywheel.”

Benjamin Franklin first discovered the Incentive Flywheel, when fires threatened Philadelphia’s growth. He gathered neighbors and founded America's first fire insurance company. They created volunteer fire departments and established the first building safety standards.

Since then, this Flywheel has been at the heart of balancing progress and security for new technology waves like electricity and the automobile. 

But the Incentive Flywheel won’t appear fast enough on its own for AI: we need to jumpstart it. This essay outlines 25 actions entrepreneurs and policymakers must take by 2030 across agents, foundation models, and data centers.

Markets are a uniquely Western solution to risks. The Incentive Flywheel adapts faster than regulation, accelerates rather than slowing down progress, and has more teeth than voluntary commitments.

I. Security as Progress

I. Security as Progress

Benjamin Franklin and the Incentive Flywheel

Houses in Philadelphia in the 1700s had a bad habit of burning down. Made of wood and packed closely together, fire caught easily and spread quickly, killing many. Homeowners could not assess their own fire risk. And did not bear the full cost of their negligence. Ad-hoc volunteer responses failed. A single uncontained fire would often destroy entire city blocks.

As the population of Philadelphia grew tenfold in the 1700s, residents were building houses faster than the systems meant to contain them.

Tired of the fires, Philadelphia-resident Benjamin Franklin took it upon himself to found the Philadelphia Contributionship in 1752, America’s first mutual fire insurance company. This insurance company designed and funded a flywheel of incentives through market-based governance: insurance, standards (building codes), and audits (fire inspections). The goal was to lower insurance payouts from fire damage. Together, the flywheel created incentives to build houses made of brick, spread best practices and invest in better fire buckets — enabling cities to grow without putting their citizens at increased risk of death.

Tired of the fires, Philadelphia-resident Benjamin Franklin took it upon himself to found the Philadelphia Contributionship in 1752, America’s first mutual fire insurance company. This insurance company designed and funded a flywheel of incentives through market-based governance: insurance, standards (building codes), and audits (fire inspections). The goal was to lower insurance payouts from fire damage. Together, the flywheel created incentives to build houses made of brick, spread best practices and invest in better fire buckets — enabling cities to grow without putting their citizens at increased risk of death.

Franklin’s solution shows up again and again through history and offers a blueprint for how markets can speed up secure adoption of new technology
1
.

Progress vs Security: A False Dichotomy 

Progress vs. Security: A False Dichotomy 

AI capabilities are advancing at breakneck speed, from preschool-level intelligence in 2020 to undergraduate-level in 2024

2

. Many researchers expect superhuman AI as early as 2028

3

, potentially creating “a country of geniuses in a datacenter.

4

” AI could help us develop cancer treatments that can reduce mortality by 95%; on the other hand, AI hackers can already find holes in our cyber security defenses

5

, and the same capabilities could soon enable terrorists to create synthetic bioweapons. Epochs are increasingly measured in months or weeks.
To capture AI’s benefits and increase America’s competitive lead, we must move forward with haste. The civilization with the most powerful AI will be economically, culturally and militarily superior. In the last two years, the US lead over China in AI capabilities has shrunk from 2 years to less than 8 months.

6

 Accelerationists are correct that most regulation will slow AI progress down, risking our already tenuous lead.
On the other hand, cautious voices are also correct that we may race right into the wall, unless we steer with care. Racing for market share is fine when the risks are small (buggy software rarely kills people). But we don’t want medical device manufacturers or nuclear power plant operators to move fast and break things. AI will quickly get baked into critical infrastructure and could enable dangerous misuse. The territory is sufficiently uncharted that any incentives for cutting corners should make us nervous.

7

Speed of AI progress is often presented — incorrectly — as being in tension with AI security.

8

  • Progress requires security. An accident could cause significant damage, and threaten America’s lead in AI. Nuclear's promise of abundant energy died for a generation after accidents like Three Mile Island and Chernobyl accelerated public backlash and regulatory scrutiny. The same will be true if AI causes major harm — courts and voters will shut AI progress down.

  • Security powers progress. ChatGPT was created using an AI alignment technique called RLHF that made systems more steerable — and thus more useful. Steerable, reliable AI systems are simply more valuable.

II. The Incentive Flywheel for AI: Insurance, Standards, and Audits

II. The Incentive Flywheel for AI: Insurance, Standards, and Audits

A market-based incentive flywheel of insurance, standards and audits is a proven way to accelerate secure AI progress. The flywheel should lead, regulatory solutions should follow. It is:

A market-based incentive flywheel of insurance, standards, and audits is a proven way to accelerate secure AI progress. The flywheel should lead, regulatory solutions should follow. It is:

  • More secure than voluntary commitments from AI companies: the rapid pace of AI progress and associated catastrophic risks mean that AI companies’ voluntary commitments will not inherently create security. The flywheel will align incentives and create accountability.

Faster than regulation: major pieces of regulation, created by bureaucrats without technical expertise, move at glacial pace.

10

 Patchwork regulation across 1,000+ state bills

11

 will create a burden on AI developers that slows adoption and investment. The flywheel will move faster and create the confidence necessary for adoption.

Figure 1: The Incentive Flywheel of Market-based Governance

INCENTIVE FLYWHEEL

FIRE SAFETY

FIRE SAFETY

CAR SAFETY

CAR SAFETY

AI

AI

INSURANCE

INSURANCE

Benjamin Franklin’s mutual fire insurance funded fire brigades and inspections

Benjamin Franklin’s mutual fire insurance funded fire brigades and inspections

Three major insurance

groups founded Insurance Institute for Highway Safety (IIHS)

Three major insurance

groups founded Insurance Institute for Highway Safety (IIHS)

Insurers can

cover AI failures

and fund safety research

Insurers can

cover AI failures

and fund safety research

Standards

Standards

Incentive-aligned fire inspections to determine eligibility

Incentive-aligned fire inspections to determine eligibility

IIHS double-checks manufacturer-submitted self-tests

IIHS double-checks manufacturer-submitted self-tests

Independent oversight

of organizations

and technology

Independent oversight

of organizations and technology

Audit

Audit

Created auditable standards (brick construction, no large shade trees)

Created auditable standards (brick construction, no large shade trees)

IIHS create testing benchmarks that get incorporated into law

IIHS create testing benchmarks that get incorporated into law

Can build auditable standards with clear risk threshold and best practices

Can build auditable standards with clear risk threshold and best practices


The market mechanics are already taking shape:

Insurance pays when things fail, motivating insurers to demand proof that risk is well-managed and fund research into lowering risks even further. This is more than risk transfer: insurers often require best practices to be followed before offering insurance. This is happening today with customer service companies starting to buy insurance against AI hallucinations that cause financial losses, like the one suffered by Air Canada where a chatbot hallucinated the airlines refund policy.

12

 As insurers accurately assess risk through technical testing, coverage becomes both a credible safety signal and protection if AI fails. This enables faster adoption with confidence. Another example emerged in late 2023 when Microsoft announced a “Copilot Copyright Commitment” effectively insuring customers against copyright violations.

13

  Within months OpenAI, Anthropic, and Amazon had all implemented versions of this copyright insurance for code-generation models, while Adobe, Canva, and Getty offered similar for image generation.

14

  These promises have now become a competitive vector, with many enterprises requiring them of vendors, thereby forcing smaller vendors to purchase these policies from insurers. This incentivizes the development of safeguards and monitoring required by the insurer.
Audits assess risk independently to let insurers price that risk accurately and inform insurance eligibility. The best AI developers already test their products and share the results with customers for assurance. Independent evaluations give buyers and insurers confidence to adopt and accurately price risk. Again, insurers often run or fund audits because they need risk data. This incentivizes robust audits.
Standards boost communication in markets. They distill complex topics into easy-to-understand certificates and metrics. They clarify best practices and what it means to take ‘due care’ that informs liability and audits. For AI model developers, defining specific risks and thresholds at which system mitigations are required is becoming common practice. For example, as AI models get better at biology, we need to prepare for new biothreats.

15

 For data centers, initial work has defined threats and thresholds for Security-Levels, and further work is ongoing to build an auditable standard. Standards are often funded by insurers. This incentivizes the reduction of damages.

Once this flywheel is spinning, investing in security will enable AI companies to grow faster by enabling confident customer adoption. Standards and audits help enterprise risk teams distinguish hype from reality, just as bond ratings help (1) investors act with confidence and (2) governments and regulators oversee financial institutions.

Historical Blueprint: Fire & Car Safety

This is not a new model.

When electricity created new fire hazards around the turn of the 20th century, Chicago Fire Underwriters' Association and the Western Insurance Union funded Underwriters Laboratories (UL) to research risks, certify products, and develop safety standards. The lightbulbs and toasters in your house today are almost certainly UL certified and marked today.

When demand for cars increased after WWII, the insurance industry established the Insurance Institute for Highway Safety (IIHS) in 1959, nearly a decade before federal government action. IIHS ratings and premium discounts created direct incentives to adopt seatbelts and airbags before they became mandatory. Deaths per mile plummeted 80% while driving surged 200%.

This Flywheel reduced risks, letting entrepreneurs build governance capacity long before government intervention.

Skin in the game is the driving force at play. Financial markets rely on risk assessments, like Moody's assigned AAA bond ratings to toxic mortgage securities before 2008, because they were paid by issuers, not affected by losses. Insurance is therefore the necessary skin in the game: when insurers misprice risk, they go bankrupt.

Even within AI, this idea of using market-based approaches to manage AI development, or using insurance, standards, and audits is not new.

16

 The rest of this piece explores in more detail how to get the flywheel turning faster, including investments and decisions required by players in the ecosystem.

Agents, Foundation Models, and Data Centers

The Incentive Flywheel secures AI progress across all three critical layers of AI development:

  1. Applications represent the majority of AI agent deployments in the real-world today. Enterprises must adopt AI agents to maintain competitiveness domestically and internationally.

  1. Foundation model developers are racing to build superintelligence. They must build the confidence of their customers and stakeholders including the public to earn the right to continue investing in, and deploying these capabilities.

  1. Data center developer infrastructure is critical for the application and model developers. They must build the confidence of their customers and stakeholders (including governments) to earn the right to scale investments to trillions of dollars and protect what could become the most valuable asset in the world.

Figure 2: Applying the Incentive Flywheel across AI development

Agentic


applications

Agentic


applications

foUndation modelS

foUndation modelS

Data
centers

Data
centers

INSURANCE

INSURANCE

Insure AI agent failures, across hallucinations, brand risk, security breaches

Insure AI agent failures, across halluci-nations, brand risk, security breaches

Insure similar AI agent failures, and catastrophic risks 

Insure similar AI agent failures, and catastrophic risks 

Insure confidentiality of model weights, integrity against sabotage, and availability

Insure confiden-tiality of model weights, integrity against sabotage, and availability

Standards

Standards

Safeguards, monitoring, testing against enterprise risk categories (data & privacy, security, customer safety etc)

Safeguards, monitoring, testing against enterprise risk categories (data & privacy, security, customer safety etc)

Define risks, testing frequency (during development and after), third-party oversight, at a system, org, and model level

Define risks, testing frequency (during development and after), third-party oversight, at a system, org, and model level

Set risk management best practices including tracking supply chain and ongoing threat monitoring

Set risk management best practices including tracking supply chain and ongoing threat monitoring

Audits

Audits

Application developers across their systems, and their specific enterprise deployments

Application developers across their systems, and their specific enterprise deployments

Model developers organizationally, and technically assess models and systems regularly 

Model developers organizationally, and technically assess models and systems regularly 

Throughout build-out process and operation

Throughout build-out process and operation

Standards in particular are central to leading global AI development. Industry standards shape technological development, and they’re increasingly a central geopolitical battleground. China recently boosted its presence in international standards bodies

17

; the US recently renamed a core AI group to Center for AI Standards and Innovation (CAISI).

18

 Private industry is central here: for example, China’s Huawei is now the largest provider of 5G equipment in the world.

19

 AI must be built on American infrastructure.

III. Incentive Flywheel Before Regulation

III. Incentive Flywheel Before Regulation

Faster Than Legislation 

In the absence of market-based approaches, many people are looking to legislation and regulation to fill the gap. The result of this is a broad range of legislation is working its way through various states and countries. Over 1,000 state-level AI bills have been introduced in the U.S. in 2025 alone.

20

 The bills are mostly well-intentioned, and those who are cautious about AI are right about their core concern. However, disconnected regulation creates a burdensome patchwork of divergent compliance regimes that will hold up investment and adoption.

Crafting comprehensive laws like the EU AI Act takes longer than it took for AI capabilities to advance from preschool to undergraduate level intelligence. In the last two years, two factors have completely changed the regulatory premise: token costs have dropped by more than 99% while open-source alternatives have emerged.

Markets leading regulation is a more effective way to satisfy all parties. For most types of risk, insurers are incentivized to develop and quickly iterate on core safety measures. Those risks can then be codified into fewer, more simplified pieces of regulation once proven (e.g. mandating airbags). Market-based governance prices in risk changes in real-time and insurance rates adjust monthly based on new data, enabling markets to clear the fog. 

Other markets show how even in scenarios with moral hazard (e.g. car accidents) or catastrophic risk (e.g. nuclear and terrorism) markets can carry the majority of the risk, supported with government mandates and backstops. For nuclear, the Price-Anderson Act of 1957 codified that the private sector cover losses up to $16 billion, with the government covering truly catastrophic excess costs.

21

 Three Mile Island's cleanup in 1979 was fully covered by private insurance. Tail risk can be difficult to price, but the fact remains: someone is always implicitly underwriting that risk, with or without insurance.
Other markets show how even in scenarios with moral hazard (e.g. car accidents) or catastrophic risk (e.g. nuclear and terrorism) markets can carry the majority of the risk, with government mandates and backstops supporting. For nuclear, the Price-Anderson Act of 1957 codified that the private sector cover losses up to $16 billion, with the government covering truly catastrophic excess costs.

21

 Three Mile Island's cleanup in 1979 was fully covered by private insurance. Tail risk can be difficult to price, but the fact remains: someone is always implicitly underwriting that risk, with or without insurance.

Only the government can deal with certain national security risks (e.g. ensuring international proliferation of standards, secure critical infrastructure, and national defense). In these areas governments should lead, partnering with the market to support the development of technologies and deployment as needed.

More Secure Than Voluntary Commitments

The accelerationist approach correctly identifies that markets excel at experimentation, learning, and adaptation. Capital chases promising ideas and bad products disappear over time (e.g. FTX crashed, while Coinbase thrives). However, the nascent AI markets suffer from market failures to prevent secure-by-default outcomes. Misaligned incentives and speed of progress mean companies do not face the consequences of cutting corners, while customers and investors lack the information to accurately assess security. There is a missing market to address these challenges:

Figure 3: Summary of why voluntary commitments are insufficient for secure AI progress

Challenges with AI companies' voluntary commitments

Challenges with AI companies' voluntary commitments

Incentive problems

Incentive problems

Incentive
problems

Companies that invest more in security testing get to market slower and are beaten by competition. This has led to a streak of AI models being deployed without sufficient testing leading to roll-backs and broken commitments.

22
Companies that invest more in security testing get to market slower and are beaten by competition. This has led to a streak of AI models being deployed without sufficient testing leading to roll-backs and broken commitments.

22

Information
problems

Information problems

Buyers lack credible information to distinguish risky AI products from secure ones, so they don't reward security investments. AI companies themselves often are unsure how risky their products are and so are unwilling to take on liability. Modern AI systems surprise even their creators. No-one can currently verify whether an AI data center can withstand state-level attacks.

Buyers lack credible information to distinguish risky AI products from secure ones, so they don't reward security investments. AI companies themselves often are unsure how risky their products are and so are unwilling to take on liability. Modern AI systems surprise even their creators. No-one can currently verify whether an AI data center can withstand state-level attacks.

Insufficient time

Insufficient time

Insufficient
time

AI progress moves too fast for markets to naturally weed out bad actors before significant harm occurs.

AI progress moves too fast for markets to naturally weed out bad actors before significant harm occurs.

Catastrophic risks

Catastrophic risks

Catastrophic
risks

Even where companies could be punished for causing harm, potential costs may exceed their balance sheet, so they'll never bear the full cost of catastrophic failures.

Even where companies could be punished for causing harm, potential costs may exceed their balance sheet, so they'll never bear the full cost of catastrophic failures.

IV. The Path Forward: Building AI's Incentive Flywheel

IV. The Path Forward: Building AI's Incentive Flywheel

IV. The Path Forward: 25 Immediate Actions

The Flywheel Is Already Emerging

We must develop the mechanisms to secure AI progress faster than we have for any technology before. Car safety standards took more than 40 years to bring down the rate of death.

23

 Artificial superintelligence has significantly higher potential for both good and harm than cars, and it could be less than 4 years until it is upon us.

24

 However, there are three reasons to be optimistic: all players profit from faster adoption resulting from more security. History provides a blueprint we can learn from. And there are early signs of the flywheel emerging today.

Established insurers like Munich Re (Est. 1880)  have teams dedicated to addressing generative AI risks. Cyber insurance companies like Coalition and Resilience (both valued at $1B+) have proven how to bundle insurance with deep technical expertise. Organizations like METR, Transluce, Haize Labs, and Virtue AI are pushing the technical evaluation frontiers. AI labs coalesce around “Frontier AI Safety Commitments" and share information and best practices through the Frontier Model Forum, while NIST has published AI Risk Management frameworks.

At the same time, an intellectual ecosystem is emerging. Jack Clark and Gillian Hadfield proposed regulatory markets; and more recently Gillian explored the role of insurance in regulatory markets; Dean Ball has suggested private AI governance with audits; Miles Brundage has written about how the triad can align incentives.

25 Immediate Actions to Accelerate the Flywheel

This emerging progress is insufficient. The scale and risk of this opportunity demands that we collectively invest $100+ billion over the next 5 years to create insurance policies, strengthen technical auditing capabilities, and develop industry standards. This investment will represent only a single digit percentage of spend on AI progress. By 2030, global AI data centers alone are projected to require $5 trillion of investment

25

, while enterprise AI spend

26

 is forecast to reach $500 billion. Comparatively, cybersecurity budgets are ~6% of IT spend and airports spend ~20% of their budgets on security.

27

 

Below are 25 actions required in the coming years. Most can be led by private industry. The actions focus on what we will need in the coming years, but we can get started with much less. The place to start is insuring the near-term harms that already have clear liability, or where contractual indemnity can be established. For example in the case of agents: hallucinations, IP infringement, bias, harmful outputs. Insuring these risks with AI-specific insurance will incentivize data-collection across risk types, research into standards, and adherence to these best practices from developers. Insuring million dollar risks will pave the way to insuring the billion dollar risks. 

Figure 4: 25 Immediate Actions to Accelerate the Flywheel

Agentic


applications

Agentic
applications

Foundation
models

Foundation
models

Data
centers

Data
centers

INSURANCE

INSURANCE

1. Insurance leaders to update policies to explicitly include or exclude AI risks from coverage, providing $500bn+ of AI coverage by 2030.

28

 Avoid a repeat of cyberinsurance, where insurers left customers unsure of whether cyber attacks were covered by their insurance policies and trillions of dollars of exposure remain without protection

29

 2. Insurers to require that AI security best practices from standards are adopted to be eligible for insurance, like insurers did to drive adoption of early Multi Factor Authentication 3. Industry to fund data collection of AI incidents to support underwriting and standards similar to insurer-funded PERILS tracking natural catastrophe data

30

1. Insurance leaders to update policies to explicitly include or exclude AI risks from coverage, providing $500bn+ of AI coverage by 2030.

28

 Avoid a repeat of cyber-insurance, where insurers left customers unsure of whether cyber attacks were covered by their insurance policies and trillions of dollars of exposure remain without protection

29

 2. Insurers to require that AI security best practices from standards are adopted to be eligible for insurance, like insurers did to drive adoption of early Multi-factor Authentication 3. Industry to fund data collection of AI incidents to support underwriting and standards similar to insurer-funded PERILS tracking natural catastrophe data

30

9. Industry convening to build $25bn+ coverage

31

 for foundation model developers, syndicated across the industry (akin to autonomous vehicles or aerospace), with data sharing and building risk modelling infrastructure 10. Government insurance mandates and backstops for catastrophic risk scenarios, to give insurers and model developers confidence (akin to Price-Anderson Act for nuclear energy) 11. Clarification of legal liability to create clarity and accountability (akin to Price-Anderson Act) 12. Scale funding from $150m today to $500m+ annually for research into technical auditing of developers 13. Require adherence to a single industry standard
9. Industry convening to build $25bn+ coverage

31

 for foundation model developers, syndicated across the industry (akin to autonomous vehicles or aerospace), with data sharing and building risk modelling infrastructure  10. Government insurance mandates and backstops for catastrophic risk scenarios, to give insurers and model developers confidence (akin to Price-Anderson Act for nuclear energy)  10. Clarification of legal liability to create clarity and account-ability (akin to Price-Anderson Act for nuclear energy) 11. Scale funding from $150m today to $500m+ annually for research into technical auditing of developers 12. Scale funding from $150m today to $500m+ annually for research into technical auditing of developers 13. Require adherence to a single industry standard
9. Industry convening to build $25bn+ coverage

31

 for foundation model developers, syndicated across the industry (akin to autonomous vehicles or aerospace), with data sharing and building risk modelling infrastructure  10. Government insurance mandates and backstops for catastrophic risk scenarios, to give insurers and model developers confidence (akin to Price-Anderson Act for nuclear energy)  11. Clarification of legal liability to create clarity and accountability (akin to Price-Anderson Act) 12. Scale funding from $150m today to $500m+ annually for research into technical auditing of developers 13. Require adherence to a single industry standard
20. Industry leaders to provide new coverage with limits above $10bn

32

 for specific AI data center risks including weight exfiltration, and integrity against state-level threats 21. Government insurance mandates and backstops for catastrophic risk scenarios, to give insurers and data center developers confidence (akin to Price-Anderson Act for nuclear energy)
20. Industry leaders to provide new coverage with limits above $10bn

32

 for specific AI data center risks including weight exfiltration, and integrity against state-level threats 21. Government insurance mandates and backstops for catastrophic risk scenarios, to give insurers and data center developers confidence (akin to Price-Anderson Act for nuclear energy)

Standards

Standards

4. Create an adoption-focused standard that 50%+ of AI agent deployments are certified against, like SOC 2 is for enterprise software

5. Clarify legal liability for AI agents and agent developers to create accountability and strengthen incentives

7. Create an adoption-focused standard that 50%+ of AI agent deployments are certified against, like SOC 2 is for enterprise software


8. Clarify legal liability for AI agents and agent developers to create accountability and strengthen incentives

14. Consensus around Frontier AI Safety Commitments and adoption across the industry, with stronger public, auditable commitments 

15. Stronger commitments to information sharing practices with third parties including governments and auditors to ensure they have sufficient access to e.g. model internals to conduct rigorous testing and employees have whistleblower protections

16. Clarify legal liability for AI models and agents, and developers to create accountability for entity-level risks and strengthen incentive

17. Consensus around Frontier AI Safety Commitments and adoption across the industry, with stronger public, auditable commitments 

18. Stronger commit-ments to information sharing practices with third parties including governments and auditors to ensure they have sufficient access to e.g. model internals to conduct rigorous testing and employees have whistle-blower protections


19. Clarify legal liability for AI models and agents, and developers to create account-ability for entity-level risks and strengthen incentive

14. Consensus around Frontier AI Safety Commitments and adoption across the industry, with stronger public, auditable commitments 

15. Stronger commitments to information sharing practices with third parties including governments and auditors to ensure they have sufficient access to e.g. model internals to conduct rigorous testing and employees have whistleblower protections

16. Clarify legal liability for AI models and agents, and developers to create accountability for entity-level risks and strengthen incentives

22. Development of an auditable standard for Security-Level 4 and 5 building on existing frameworks and threat analyses

33

25. Develop- -ment of an auditable standard for Security-Level 4 and 5 building on existing frameworks and threat analyses

33

Audit

Audit

6. Auditors to build executive-level dashboards tracking real-time agent deployments and risks in the real world - similar to Basel III banking dashboards 

7. Significant scale-up of technical auditing expertise and capacity from <1,00 deployments in 2024 to 1,000,000+ in 2027, powered by productionized testing and monitoring infrastructure building on examples like Inspect and Docent

8. Technical auditing expertise combining the rigor of PwC with the cutting edge AI capabilities of startups like Haize Labs to develop evaluations applicable to real-world harms, including multi-turn and multi-agent evaluations, and that robustly assess risk including using model internals and checking for sandbagging as needed. These tests are essential to ensure companies cannot cheat tests, like VW did with emissions


6. Auditors to build executive-level dashboards tracking real-time agent deployments and risks in the real world - similar to Basel III banking dashboards 

7. Significant scale-up of technical auditing expertise and capacity from <1,00 deployments in 2024 to 1,000,000+ in 2027, powered by productionized testing and monitoring infrastructure building on examples like Inspect and Docent

8. Technical auditing expertise combining the rigor of PwC with the cutting edge AI capabilities of startups like Haize Labs to develop evaluations applicable to real-world harms, including multi-turn and multi-agent evaluations, and that robustly assess risk including using model internals and checking for sandbagging as needed. These tests are essential to ensure companies cannot cheat tests, like VW did with emissions


17. Auditors to build an ‘AI risk dashboard’ that can be shared with American intelligence as well as enterprises, tracking real world risks and evaluation techniques, e.g. like Iran’s use of OpenAI for an influence campaign

18. Significant scale-up of technical auditing talent outside model developers from <100 researchers to 500+ by 2027

19. Auditing of organizational and system-level risks (as opposed to only model-level) to e.g. prevent state-affiliated misuse or information gathering. Build on best practices from defense, cyber, financial audits

14. Auditors to build an ‘AI risk dashboard’ that can be shared with American intelligence as well as enterprises, tracking real world risks and evaluation techniques, e.g. like Iran’s use of OpenAI for an influence campaign


15. Significant scale-up of technical auditing talent outside model developers from <100 researchers to 500+ by 2027


16. Auditing of organi-zational and system-level risks (as opposed to only model-level) to e.g. prevent state-affiliated misuse or information gathering. Build on best practices from defense, cyber, financial audits

23. Auditors to build a ‘data centers risk dashboard’ that can be shared with American intelligence and integrated into defense planning 

24. Audits that can integrate financial, physical security, and technological expertise into a single actionable approach 

25. Audits that oversee the entire build process (e.g. tracking the chip supply chain to prevent espionage) alongside operations

23. Auditors to build a ‘data centers risk dashboard’ that can be shared with American intelligence and integrated into defense planning 

24. Audits that can integrate financial, physical security, and technological expertise into a single actionable approach 

25. Audits that oversee the entire build process (e.g. tracking the chip supply chain to prevent espionage) alongside operations

Building The Movement 

This is merely a starting point. The fog around AI’s trajectory calls for a need to experiment with incentives quickly, fail, learn, and adapt. As evidenced by the rapid advances of AI research and application development, the stakes have never been higher, the timelines never more compressed. Now is the time to act.

Applying the incentive flywheel to underwrite secure AI progress needs the technologist’s ingenuity, the actuary’s carefulness, the business leader’s pragmatism, the economist’s incentive analysis, the legal scholar’s historical grounding and the researcher’s willingness to explore unusual futures. 

The authors are building the incentive flywheel right now. If you are interested in contributing, reach out at rk@aiuc.com.

We are grateful to Nat Friedman, Daniel Gross, Dan Hendrycks, Charlie Songhurst, Geoff Ralston, Judd Rosenblatt, Gillian Hadfield, Nabeel S. Qureshi, Ivan Vendrov, Nathan Labenz, Sean McGregor, Chris Painter, Sanmi Koyejo, John Loeber, Miles Brundage, Willem Van Lancker, Steven Adler, Herbie Bradley, Nate Rush, Vishal Maini, Rudolf Laine, Alan Chan, Salen Churi, Eric Ries, and Henna Dattani for helpful feedback and discussion about these issues. Thanks to Stephen Thomas and Andrew Ettinger for editorial support. 

We are grateful to Nat Friedman, Daniel Gross, Dan Hendrycks, Charlie Songhurst, Geoff Ralston, Judd Rosenblatt, Gillian Hadfield, Nabeel S. Qureshi, Lauren Wagner, Ivan Vendrov, Nathan Labenz, Chris Painter, Sanmi Koyejo, John Loeber, Miles Brundage, Matija Franklin, Willem Van Lancker, Steven Adler, Herbie Bradley, Vishal Maini, Rudolf Laine, Alan Chan, Sean McGregor, Matija Franklin, Salen Churi, Eric Ries, and Henna Dattani for helpful feedback and discussion about these issues. Thanks to Stephen Thomas and Andrew Ettinger for editorial support. 

Footnotes

1

Table 2 below outlines examples across fire risk, car safety and AI.

5

Sean Heelan used OpenAI’s o3 model to find a zero day in the Linux Kernel’s SMB implementation, https://sean.heelan.io/2025/05/22/how-i-used-o3-to-find-cve-2025-37899-a-remote-zeroday-vulnerability-in-the-linux-kernels-smb-implementation/

9

For example, the EU is considering pausing their flagship EU AI Act before it has even come into effect. https://www.dlapiper.com/en-gb/insights/publications/ai-outlook/2025/the-european-commission-considers-pause-on-ai-act-entry-into-application

11

AirCanada’s customer service chatbot hallucinated their refund policy. Courts found that companies are responsible for the promises their AI agents make: https://www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-know

19

Examples: Microsoft shut down their AI chatbot Tay in 2016 after it spewed racist and Nazi ideology (link); Google’s Gemini outputted photos of people of color in Nazi uniforms in 2024 (link); OpenAI rolled back a overly sycophantic version of ChatGPT in April 2025 (link); Google committed to publishing safety papers significant AI model releases, but shipped Gemini 2.5 Pro without the promised safety documentation (link)

29

Sandbagging refers to AI systems deliberately changing behaviour when they know they are being evaluated