The Artful Gamer · The Changing Nature Of Gaming Interfaces

The other day I was having coffee with friends who brought their 2 1/2 year old son over for a visit.

He was bored, looking for anything to do in our (boring) house — so I handed him an original Game Boy with Super Mario Land 2. I figured that a toddler would enjoy smashing the Game Boy's bulletproof buttons, making Mario run and jump, and hearing the ear-piercing four-channel music. He took the Game Boy from my hand with interest, and held onto it in the familiar way that all of us hold portables. He look at the cabbage-green screen and squealed, "MARRIOO!" I asked his mother if he had played games before, and she said, "Oh yeah. He loves playing kiddie games on our iPhone."

I turned back to her son, and he was frowning intently at the Game Boy. He reached out tentatively and pushed on the plastic screen. Nothing happened. He pushed again, in a different spot. Nothing. I reached over and pushed a button — Mario jumped. He looked at me with a puzzled expression, and turned back to the game. I eventually had to slide his fingers over to the D-Pad and buttons, pushed them down a few times to show him how it worked, and he started to "get it".

I realised in that moment that we are now living in a time when the standard D-PAD + Buttons layout can no longer be assumed the "standard" way of playing a game. A new generation of players are growing up with motion-based interfaces from Sony (the upcoming Playstation Move), Nintendo (Wii MotionPlus, Balance Board), Harmonix (Rock Band), as well as touch based devices from Apple (iPod Touch/iPhone). Where the 1980s and 1990s almost always guaranteed a familiar mediating interface — whether it be a keyboard, mouse, or D-Pad — I wonder at how the recent explosion of alternative interfaces has changed the way gamers understand what a game is?

For instance, can we really say that Myst or Monkey Island 2 SE for PC are the "same games" as their iPhone variants? On what basis could we distinguish between our experience of playing the two (temporarily setting aside differences in sound quality, resolution, etc)? Is the "touch" aspect really that different from a point-n-click interface using the mouse?

I'm going to waffle here, because I just don't know. And here's why:

When I play any game, using a standard NES/PS2/PS3/Xbox/GameCube controller layout — my fingers and thumbs find their places. If it's a NES, my right thumb handles the A+B buttons while my left thumb takes care of the D-Pad. There are no moments of confusion, I never have to ask myself, "which button is it again?". The same goes for the PS2 and PS3 games: my fingers know their business. As soon as I settle down to play the game, my fingers are no longer fingers to me. They are a part of the game — my fingers become something like my mouth when I am speaking — they spring into action when Mario needs to bound over a Chain Chomp or needs to go down a green pipe. My fingers never become a part of my foreground or focal experience — in other words, my fingers become repressed parts of my bodily experience. If I had to think about what I was going to do next before committing myself to the act, Super Mario 3 would become unplayable. In other words, games like Mario 3 require us to forget that we have fingers for a few moments in order to bring a natural flow into the game. Without getting too artsy or mixing metaphors, many games demand that the player become a pianist of a kind.

Mouse-based interfaces that we typically see in adventure games require a different kind of skilfulness. My hand has to learn to map the horizontal two-dimensional space of the mouse to an on-screen virtual space. I have to learn that forwards-is-up, and backwards-is-down, and that I have to move the cursor to the right position in order to make my character do something. In this kind of interface, I still "repress" my hand — at some point my hand disappears and the cursor becomes invisible to me. The cursor moves simultaneously with my hand. My hand knows where it needs to go on-screen in order to make Guybrush Threepwood pick up a wooden mallet. I don't think to myself: there is a mallet, and I need to click ‘pick up', then click on the mallet. Exploring the world of Monkey Island 2 becomes a natural gesture for me.

This review demonstrates how focal one's finger can become when playing Myst on a touch device.

But can the same be said for touch-based devices that require us to make physical contact with the display in order to play the game? For instance, while the Myst interface is more or less the same between the PC/Mac and iPhone versions, the fact that I have to occlude some of the screen with my fingertip in order to "do" something changes the game subtly. Every time I reach forward and click on the screen with my finger I feel the cool glass push back at me, and I leave a fingerprint. There is something very focal in interacting with touch-based devices, because my finger does not fall into the background as easily. Compare that to the PC version: my hand is always on the mouse, my fingers always in their familiar positions on the mouse buttons. They never leave that surface, and the mouse becomes an extension of my body. On the iPod, my finger is constantly leaving the surface, popping in and out of my visual field.

But it's the same game, right? Not for me. While the iPhone version of Myst is a wonderful port of the original game, I cannot quite dwell in the world simply because I cannot repress my awareness of my fingertips. I feel like I am playing a game. It is not quite bad enough to totally remove me from the world, but it is enough to remind me that yes — I am playing a game on my iPod Touch and this is a virtual/fictional world that I am interacting with. The PC version of Myst is nothing like that — when I click something I am reaching into the world and flipping a toggle switch.

Returning to my anecdote: does my friend's 2 1/2 year old son experience his favourite iPod Touch game as a ‘real' world? Or is his experience like mine - somewhat disembodied and self-conscious? Is this an inherent problem with touch-based interfaces, or do some of us already experience bodily repression that allows us to ignore our fingertips when we touch the display? How much have designers appreciated the qualitative change in gameplay experience as a result of the massive turn towards touch-based gaming, and have they done anything to respond to it? What are your experiences with touch-based (or even motion-based) interfaces; how do they change your experience of the game?

1. I am using a very special meaning of the word "repression" that Merleau-Ponty introduces in his phenomenology of the body. It is not the same as Freud's notion of repression. (For more details see Lawrence Hass's book Merleau-Ponty's Philosophy, pp. 89-90).

2. I am always struck that people who have never played side-scrollers like Mario 3 often become frustrated that they have to "think" before acting. The same experience is felt by those learning a second language.

Chris Lepine is a doctoral candidate in psychology. The Artful Gamer is his attempt at revealing the rich and poetic experiences we have as gamers. He lives in Western Canada with his fiancée, who together have created a game studio focusing on the art of visual storytelling.


    More to the point, how can that kid not know how to use a button?
    I know he had never used a device like that before but seriously there is some learning and initiative issues there.

    Anyway I played some gameboy roms on my iphone and found the controls somehow decidedly lacking. Without the tactile feel and response of physical buttons you just don't have the same level of unconscious control.
    You cant feel that you are pressing the right button or even that you have without looking at your finger or seeing the result. Fine for turn based kinda stuff but faster paced real time games just are not as playable on a touch screen.

    I think touch screens are more (hope I'm using the right terminology here) tactile than reflexive.
    Rather than a twitch or semi-conscious gesture with the mouse, players must interact with the same screen they're playing on.
    For me, this breaks the immersion I feel with the game.
    I believe this knowledge is what led nintendo to include buttons on the DS and often give people a choice of how to play.

    One of the many reasons why motion control is the death of us all, and further proof that gaming executives simply don't understand gamers. Stupid gimicks like the Wii/Move/Kinect break the immersion, not enhance it. When i'm playing with a controller, it's out of sight, out of mind. I don't need to look at it for even a second once i've started. Whereas motion control, it's going to get to the ridiculous point that you have to go out and train in order to have any decent skill with a sports game. It's taking all of the brain out of games, and replacing it with brawn.

      If by 'gamers' you mean 'people who play games'- then of course they understand them, and they pander to this newly-created casual majority.

      There is no LESS skill in throwing a perfect 30 on WiiSports Resort 3-point shootout, than finishing Demons Souls. Its just a different, simpler, purely physical kind. It also happens to be nearly impossible.

      If you think that you don't have to "train in order to have any decent skill" with ANY competitive game, you are sadly mistaken. The training may take a different form (a Starcraft player is unlikely to work on upper-body strength; a Kinect boxer may well require some), but it's still training.

Join the discussion!

Trending Stories Right Now