Joel Becker | Reconciling Impressive AI Benchmark Performance

AI agents can now autonomously complete coding tasks that take humans with relevant expertise multiple hours with 50% reliability — and this capability is doubling every 4-7 months. Yet when METR ran an RCT around ~March 2025 (when frontier AIs had around a 1 hour time horizon), experienced open-source developers took 19% longer to complete tasks when they were allowed access to frontier AI tools relative to when AI tools were disallowed.

This talk will introduce both papers, attempt to reconcile apparently contrasting conclusions, and discuss research frontiers and directions coming out of the attempted reconciliation.

Link to time horizon paper

Link to developer productivity paper

Details:

Time: 12:00 pm - 1:00 pm PT

Location: Gates Computer Science Building, Room 119, 353 Jane Stanford Way, CA 94503

You can find additional event details, speaker bio, and information about upcoming seminars on our website.

Know what’s Happening Next — before everyone else does.

Get App

Host or Publisher Stanford Digital Economy Lab