Analyzing Frontier Model Progress with My Favourite Game: Prince of Persia

Analyzing Frontier Model Progress with My Favourite Game: Prince of Persia blog.priyan.in

I first played Prince of Persia in 1995 on an IBM PC XT. I still go back to it from time to time: the rotoscoped animation, the way the p...

63 points•msephton•5 days ago•43 comments•

#prince #persia #progress #analyzing #frontier #favourite #game #analyzing frontier

43 comments

criemen4 days ago

A German magazine (c't) ran a Asteroids programming contest in 2008 - create a client with access to the emulator output of the game that provides keyboard inputs to control the game.

The highest scoring submission that won the contest had a high score of around 137k. Last week, I had GPT-6 Astra, Sol and Luna implement and hill-climb on this task, as I wanted to see how big the difference in smartness is. Luna implemented something, but never exceeded ca. 20k points, with a large variance. Sol got something in the area of the humans implementation.

Astra, which finished fastest, had a highscore of around 1.7Mm when the game seemed to fairly reliably crash. On the way, it disassembled parts of the ROM to extract information about the game.

I didn't do a ton work to document and measure the specifics, but it was very impressive.

looperhacks4 days ago

If I remember correctly, back then all "good" entries in the contest solved the game by syncing with the RNG and could basically predict where and how new asteroids would spawn. The difference between entries was planning ahead movement and residence against control issues due to the contest setup (like inputs getting delayed). I wonder how Astra managed so much better, I thought the problem was already solved

criemen4 days ago

Thanks! I went back, and had a look at both the contest rules and the highest scoring submission again. Turns out I benchmarked against infinite games, whereas the contest measured a 5min time limit. So, Astra solved a different problem.

Comparing strategies, though, Astra does something pretty different from the highest scoring submission. It doesn't predict the RNG, instead it reacts to the visuals on-screen, planning ahead by estimating velocity of all objects on screen. Unlike the winning solution, it actively flies the ship, whereas the winning solution basically only rotates and teleports. So Astra behaves more like a regular "perfect" player would, rather than one that breaks the PRNG.

NitpickLawyer4 days ago

> On the way, it disassembled parts of the ROM

You should check the logs, there's probably a bunch of retro gaming forums that got hacked behind the scenes :D

Kuyawa4 days ago

I am going to try to build the original Prince of Persia using Swift as it is the best language for building a macOS game. Being a classic 2D cinematic platformer, Apple's native frameworks provide exactly what I need without the overhead of a massive cross-platform engine, and I love simplicity

So here goes my weekend, flag this and get a life

andsoitis4 days ago

Share it when you're done! Even if it isn't complete complete.

Kuyawa4 days ago

Already started and will be ready soon, check back in an hour

https://github.com/kuyawa/prince

gambiting4 days ago

The real miracle of AI is that it could work with these prompts at all

"now char coming but movement everything wrong ... prince is not in floor"

I'm a games developer and I have no idea what OP is asking for here.

tehlike4 days ago

Instead of fixing a broken version, maybe each model should have started from scratch

jablongo4 days ago

yea for a real comparison between models this would have been better, but I have a feeling he just wanted to play the best possible version of PoP.

tehlike4 days ago

That may still be a new model. Having an existing broken engine might bias the new model in iterative design vs figuring out what's wrong in the first place.

msephton4 days ago

I agree this would have been more interesting.

smokel4 days ago

The prompts provided are atrocious. It's amazing that the LLMs actually built something useful.

> My first prompt was simple: "in this original code there 6502 assembly code for prince of persia, use the save level files and try to do it in c# console."

Do what in the what now in the console?

gcanyon4 days ago

"now char coming but movement everything wrong ... prince is not in floor"

I'm assuming a language barrier on the part of the author. I wonder if the models would do better being prompted in the author's main language.

cavemandaveman4 days ago

Or having another model proofread the prompts and write clearer instructions

aksss4 days ago

The prompts were awful and really distracted from an otherwise cool project concept. Also, testing the next model by iterating on the first model’s crappy foundation. Why not start from the original each time?

opiotrek4 days ago

That was my same thought, these are barely coherent sentences in English.

daemonologist4 days ago

Yeah, I'm thinking the nigh-unreadable AI-speak we get these days makes a lot more sense if this is what they're training on. Or maybe the author has translated the prompts from another language?

InsideOutSanta4 days ago

It's so confusing how the actual article is in English, but the prompts are just gibberish. But LLMs are pretty good at deciphering gibberish; I often put our CEO's absolutely atrocious E-Mails into ChatGPT and tell it to explain wtf he wants from me.

Also, I feel like the LLMs would have done better if they had started from scratch each time, rather than being burdened by the output from the previous attempt.

Read the full thread on Hacker News →

43 comments

Related stories