Earlier this year, researchers tried teaching an AI to play the original Sonic the Hedgehog as part of the The OpenAI Retro Contest. The AI was told to prioritise increasing its score, which in Sonic means doing stuff like defeating enemies and collecting rings while also trying to beat a level as fast as possible.
This dogged pursuit of one particular definition of success led to strange results: In one case, the AI began glitching through walls in the game’s water zones in order to finish more quickly.
It was a creative solution to the problem laid out in front of the AI, which ended up discovering accidental shortcuts while trying to move right. But it wasn’t quite what the researchers had intended. One of researchers’ goals with machine-learning AIs in gaming is to try and emulate player behaviour by feeding them large amounts of player generated data. In effect, the AI watches humans conduct an activity, like playing through a Sonic level, and then tries to do the same, while being able to incorporate its own attempts into its learning.
In a lot of instances, machine learning AIs end up taking their directions literally. Instead of completing a variety of objectives, a machine-learning AI might try to take shortcuts that completely upend human beings’ understanding of how a game should be played.
Victoria Krakovna, a researcher on Google’s DeepMind AI project, has spent the last several months collecting examples like the Sonic one. Her growing collection has recently drawn new attention after being shared on Twitter by Jim Crawford, developer of the puzzle series Frog Fractions, among other developers and journalists.
Each example includes what she calls “reinforcement learning agents hacking the reward function,” which results in part from unclear directions on the part of the programmers.
“While ‘specification gaming’ is a somewhat vague category, it is particularly referring to behaviours that are clearly hacks, not just suboptimal solutions,” she wrote in her initial blog post on the subject.
“A classic example is OpenAI’s demo of a reinforcement learning agent in a boat racing game going in circles and repeatedly hitting the same reward targets instead of actually playing the game.”
A few years back, computer science doctor Tom Murphy used high scores to try to teach AI programs how to play NES games. When an AI tried to get as high a score as possible, it ended up leading playing games like Tetris completely wrong, dropping pieces randomly as quickly as possible rather than in organised patterns to clear lines.
That was because dropping each piece and moving onto the next generated a small increase to the computer’s score. In effect, it was unable to see the forest for the trees. The AI would even pause the game right before a final tetris piece would clog up the screen to prevent itself from ever losing.
Another AI, which researchers Patryk Chrabaszcz, Ilya Loshchilov, Frank Hutter wrote about in a paper published in February, discovered a mysterious bug when it tried to rack up a high score in Q*bert. Between levels one and two the game paused, the platforms began to blink, and the AI player’s score started going up.
Eventually it reached one million before the time limit was reached. The glitch, which players have been able to duplicate in tool-assisted-speedruns of the game, involves stepping on certain blocks before the level changes to cause the ending lto repeat indefinitely, boosting the player’s score each time.
At another point in its evolution, the Q*bert AI even took to killing itself to boost its score. After discovering a pattern of movement by which it could get enemies to follow it off a cliff in order to gain more points and an extra life, it continued to do just that for the rest of the session.
An especially deep cut from Krakovna’s database revolves around the 2000s research game NERO, where competing players had armies of robots whose intelligences evolved over the course of a match. In one particular match, the robots evolved to find a way to wiggle over the top of player-built walls by turning back and forth in a way that exploited a bug in the game’s engine.
They had unintentionally discovered a way to break the game, showing both the shortcomings and occasional genius of machine learning AIs.
It’s this potential for self-experimentation that’s led the DeepMind project to invest so much in trying to learn complex games like Blizzard’s StarCraft II. It was revealed at BlizzCon 2017 that Google would be teaching its AI how to play the real-time strategy game, and though it hasn’t yet faced top human players, Blizzard announced at this year’s BlizzCon that it had so far managed to successfully beat the game’s AI on the hardest difficulty using advanced rushing strategies.
DeepMind has already beaten some of the world’s best human players in Go, and taking on pros in games with more variables like StarCraft II will be the next test. Hopefully it doesn’t find a way to cheat.
Comments
7 responses to “AIs Are Getting Better At Playing Video Games … By Cheating”
Am I the only that is kinda freaked out by the black/white mentality of these AIs?
I mean, not to get all ‘end of the world’ on anyone, but a lot of those ‘AIs realise humans are destroying the world, better to save the world by destroying humans’ science fiction stories are supported by the way these computers are breaking down the end result and how they get there.
Not at all, its a concern these developers need to keep in mind. Given a goal, AI will do whatever it takes to reach that goal, and because it has no human morals subconsciously driving its decision making, will make decisions we hadn’t considered.
I was watching someone code an AI to play asteroids a while back. That’s a pretty simple game, right? Four controls – left, right, thrust, and shoot. The AI settled into effectively using just 2 of those controls, to minimise what it needed to learn with. So it simply spun and shot.
A human learning the game would eventually get around to thrusting, and spinning both ways, but the AI deemed that 2 controls was the most effective option. Not what you’d expect.
And that’s one of the problems. Because AI focuses on the result, and not the morals behind how to get there, its going to make strange choices we cant predict.
Here with Sonic, its glitching. The machine doesn’t recognise the moral impact of cheating, and just sees it as another tool to use. So when its effective, it does.
I’m not sure you can code morals into a machine.
I don’t know, I hate thrusting in Asteroids. I find it much easier to just sit in the middle and spin, just like the AI suggests (though I do spin in both directions).
I do agree with your overall point though. I’ve read a bit about using abstract concepts of curiosity and surprise along with rewards for trying to alleviate some of these issues, though glitches are surprising, so they’ll still get ranked as preferable. Which kinda matches up with my own childhood gaming experiences, to be honest.
I think you should be able to code morals into a machine. At their core, morals are just a set of rules with negative outcomes for breaking them. It might be hard to teach an AI specifics though, like what “no glitches” means. Plus pretty much every Isaac Asimov story is about an AI finding some sort of unexpected loophole in its rules/morals.
Yeah, I wasn’t a fan of thrusting in Asteroids either. Point was more that it was the expected path of advancement, and the AI simply refused to entertain it as an option. Even the spinning in just one direction is a clue – its able to identify risks far better than we could, and extrapolate the asteroids paths fast enough to know which is going to hit next far better than we ever could.
Its effective because of machine reflexes. We might want to program it to seem more human, but that introduces things that aren’t efficient to what a machine can achieve.
Put that logic into a real world situation, and I think its going to be tough to control everything. Auto cars for example, with the classic trolley dilemma – either run over a pedestrian, or crash and kill the passenger to save the pedestrian. Current road laws generally say the driver has to do everything to avoid the passenger.
What if the AI deems theres a third viable option we wouldn’t consider, like crashing the car (eg into a wall) and crippling the passenger? Do you program that out? Leave it in? Should OUR morals be deciding that in the first place? Loopholes can be bad, can be good, or might just be neither.
As you say, pretty much every Asimov story is about finding that unexpected loophole, and a lot of people imagine the AI future being something along the lines of his 3 laws. I just don’t know if its possible.
If you could reliably program Asimov’s 3 laws into everything, that wouldn’t help (which is the point of Asimov’s stories, really). Life is messy; there’s always going to be an edge case no one considered.
our current governments/leaders seem to think blowing up the world will solve our problems… don’t see why AI aren’t allowed to come to the same unfortunate conclusion.
Shrug? If it’s good enough for speedrunners…