Hi all, wanted to share a cool project me and two buddies have been working on. Essentially we took a Toyota Corolla, installed a Comma 4, hooked it up to GPT-6 Astra/Claude Fable/Grok 4.6/Sol, gave them MCP tools to control the Toyota, and then we let them try driving a cone course (in an empty parking lot at low speed). They can observe the cameras/telemetry, request steering/speed, and stop the car. The goal was to see how good frontier LLMs might be just out-of-the-box at a task like driving a real physical car (inspired by recent such demos in robotics).

The best model was GPT-6 Astra which fully finished the course on its second attempt (which was pretty shocking to us). We gave each model up to 3 attempts on the course in the same chat conversation (to incentivize in-context learning). Fable also improved quite a bit from 9% to 45% by its 3rd attempt.

We also have a trace viewer at our website (for example see https://drivingbench.com/trace/gpt-6-astra/2/ for Astra's successful attempt), and everything is open source (traces, harness/code at https://github.com/aditya-ramabadran/drivingbench_harness_v1, we have a report at https://drivingbench.com/report/ on how we did everything, some of the things that went wrong or that we could do better, etc).

Obvious disclaimer: the models clearly aren't good enough to drive on an actual road yet. Also we did this at super low speeds in an open parking lot with a human always ready to brake at any time. Please use the harness/code/etc at your own risk. Would love to answer any questions or take any feedback for v2!