Safe Autonomous Organizations Deployed in the Real World

Far fewer people have seen what happens when frontier models are given real autonomy, real customers, real money, and real consequences.

The goal isn’t to showcase what AI can do. It’s to discover where it breaks, how it recovers, and what today’s leading models are genuinely capable of when operating over extended periods with minimal human intervention.

Drawing on his work building evaluations at Andon Labs, he’ll discuss where AI is now, where it was a few months ago, and what to expect given current trajectories.

Expect a deep, practitioner-level session with lots of Q&A and real world anecdotes!

This session will move beyond benchmarks and leaderboards to explore questions practitioners are increasingly facing:

• Real-World AI Deployment: What actually limits autonomous agents today?
• Rapid Evolution: How quickly are those limits moving?
• Practical Insights: What failures only appear after hours or days of autonomous operation?
• Future Trajectories: What evaluation techniques matter before putting AI into production?

Expect an honest, technically grounded discussion based on real-world experiments, practical lessons, and observations from the leading edge of AI capability development, with plenty of time reserved for audience Q&A.

Callum Sharrock works at the forefront of AI evaluation and safety, developing real-world assessments for frontier language models through his work at Andon Labs.

Rather than measuring performance on static benchmarks, Callum evaluates how autonomous AI systems perform when entrusted with running real businesses, interacting with customers, managing operations, making decisions, and adapting to changing environments over extended periods.

His work brings him into close collaboration with frontier labs, including Anthropic, while evaluating models from the industry’s leading providers. His focus is understanding the capabilities, limitations, and safety considerations of increasingly autonomous AI systems before they are deployed into the real world.

Callum brings a rare practitioner perspective from the intersection of frontier model evaluation, autonomous agents, and AI safety, making him uniquely positioned to separate today’s reality from tomorrow’s speculation.

• Expert Presentation: Direct insights from the frontlines of AI evaluation.
• Interactive Q&A: Ample time dedicated to answering your specific technical and operational questions.
• Networking: Connect with fellow practitioners, builders, and AI enthusiasts.