TL;DR
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A live experiment called the Firmulate Crucible League tests AI models’ ability to manage a simulated business crisis. Results show that management quality, including decision-making and trust, is a distinct and critical measure of AI performance, not just response accuracy. For more insights, see the original analysis.
The Firmulate Crucible League, conducted in July 2026, has evaluated AI models’ management capabilities during a simulated crisis within a small software company. The experiment assigns real financial consequences and trust standards, revealing that management quality is a separate and vital dimension of AI performance, beyond traditional chat or coding benchmarks. For a detailed analysis, see the original analysis.
The experiment involved five AI models competing in a realistic business scenario, where they faced multiple crises, customer negotiations, and ethical considerations. The models were scored on a 100-point scale, with the top performer, GPT-5.6-SOL, achieving a score of 95, and the lowest, Opus 4.8, scoring 73. Despite all models successfully identifying crises and resisting manipulation attempts, only two managed to close a key €55,000 deal, highlighting that effective management involves more than just accurate diagnosis.
One notable finding was that models that read and reference internal documents performed better in closing deals, emphasizing the importance of retrieval accuracy. Moreover, the experiment enforced a strict trust standard: any breach, such as providing false information or bypassing approvals, capped the score, underscoring that trustworthiness is as critical as technical competence. The models’ ability to refuse manipulative requests was consistent, but their capacity to complete managerial tasks varied significantly, with some adding extensive analysis yet failing to escalate or finalize decisions properly. This highlights the importance of evaluating AI management skills, as discussed in this management test.
Why Management Skills Are the Next Benchmark for AI
This experiment demonstrates that traditional AI benchmarks—focused on language quality or coding—do not capture the full scope of AI usefulness in real-world management. The ability to triage crises, prioritize actions, communicate effectively, and uphold trust are essential for deploying AI in operational roles. The findings suggest that future AI evaluation should incorporate management-like tasks to ensure models can handle complex, consequential decisions reliably, not just produce plausible responses.
AI management decision-making tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limitations of Current AI Benchmarks and the Need for Real-World Testing
Historically, AI performance has been measured through coding challenges, chat responses, or benchmark datasets that do not reflect the messiness of real business environments. The Firmulate experiment bridges this gap by simulating a company’s worst week, with real financial stakes, customer interactions, and ethical dilemmas. Prior to this, no standardized test has evaluated AI’s capacity to manage organizational consequences, making this a pioneering step toward practical AI deployment.
“Traditional benchmarks miss the core of management: trust, decision-making under pressure, and the ability to follow through on commitments.”
— Thorsten Meyer, creator of the experiment
Unclear Aspects of AI Management in Real-World Deployments
It remains unclear how well these results translate to actual business environments outside the controlled simulation. The experiment’s scope, while comprehensive, does not yet confirm how models will perform over longer periods, in larger organizations, or under different regulatory and cultural constraints. Additionally, the impact of ongoing learning and adaptation in live settings is still to be explored.
Next Steps for Evaluating and Deploying AI Management Capabilities
The ongoing experiment will finalize its results in July 2026, providing a detailed ranking of models on management tasks. Future work includes integrating these evaluation methods into enterprise AI procurement processes, developing standards for trust and escalation, and expanding testing to larger, more complex organizations. Researchers and companies will also explore how to train models explicitly for management tasks, emphasizing trustworthiness and decision completion.
Key Questions
How is this experiment different from traditional AI benchmarks?
This experiment assesses AI’s ability to manage a simulated business crisis, focusing on decision-making, trust, escalation, and execution, rather than just language quality or coding accuracy.
Why is trustworthiness emphasized in the evaluation?
Because in management roles, trust is critical. The experiment penalizes any breach of trust, such as providing false information or bypassing approval processes, ensuring models are reliable and honest.
Can these results predict AI performance in real companies?
The experiment offers valuable insights but is still a simulation. Real-world deployment will require further testing, especially over longer periods and in diverse organizational contexts.
What should companies consider before using AI for management tasks?
Beyond response quality, companies should evaluate whether the AI can read organizational documents, escalate issues appropriately, and maintain trust under pressure—key factors highlighted by this experiment.
When will the final results be available?
The complete rankings and analysis are scheduled for release in July 2026, providing a new benchmark for AI management capabilities.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
