Ghosts In The Machine: The Future Of Video Games

Ghosts In The Machine: The Future Of Video Games

Games are a constantly evolving medium, but after recent evolutions in user interfaces with the Nintendo Wii and Kinect, what’s next for gaming? To find out we spoke to inventor, futurist and all-round smart guy Mark Pesce about the future of gaming and what we can expect over the next 10 to 20 years.

Predicting the future is tricky – just ask Bob Gale. Hoverboards, flying cars, hydrated pizza – Back to the Future is strewn with the litter of misguided clairvoyance, leaving an entire generation of 20-somethings depressed, empty husks of men, F5ing the homepage of their Facebook profile when they should be on the streets kicking ass in their auto-fit bodywarmers and Nikes that lace themselves.

So yes, predicting the future can be a fruitless exercise – but when it comes to videogames we just can’t help ourselves. New technology continues to drive this industry to new heights, so it makes sense for us to gaze into the crystal ball, aimlessly wondering, dreaming of what the beguiling world of video games will show us next.

Ghosts In The Machine: The Future Of Video GamesTHE STARTING POINT “The starting point is Kinect,” claims Mark Pesce – and he should know. He’s only the co-inventor of VRML – a source-free remote that pioneered what Nintendo attempted, 15 years before the Wii hit shelves. He’s also the author of The Playful World: How Technology Transformed Our Imagination and a judge on ABC’s hit show The New Inventors.

“The Kinect is without question the most important user-interface innovation since the mouse. Everything we do in regular gaming and computing is going to start reconfiguring around what it is and what it does. It turns the tables round – we’ve been watching computers for the last 30 years, but now for the next 30 years computers are going to be watching us.”

So, in a sense, we’re at the start of a new learning curve. We’ve spent the last 20 or 30 years interacting with devices that, in short, did what they were told – directly – because we told them to. User interfaces like Kinect will now define the way in which we interface with our technology by interacting right back at us.

But isn’t Kinect a laggy, frustrating piece of kit with its own myriad of issues? It struggles in small living rooms; it can’t recognise more than one person at a time. Our experience with Kinect has been punctuated with moments of awe, sandwiched between a host of problems and frustrations.

“I think you’re right,” begins Mark, “When Kinect was first announced we didn’t really get a feel for the actual limitations.

“But the limitations are half the device and half us – we don’t really understand all of the metaphors for working with the device yet. People did stupid things with mice for the first five years they were on the market before people started to discover what they were good for. It’s a co-evolution.”

Those limitations, and the continuing struggle of developers to invent ‘metaphors’ for Kinect, probably accounts for the fact that the software line-up for the device has been lacklustre to say the least. The Nintendo Wii has struggled in a similar way – even Nintendo’s first party games struggled to innovate within this volumetric space – hence the infamous “waggle”.

“The Wii is not a bad way to approach the same problem,” claims Mark “It’s a different approach – it opens different doors. And Nintendo has made everything so friendly, so easy. In some ways they’re the Apple of video gaming – actually it’s the other way around, Apple has been copying everything Nintendo does! You start to see that there are strengths that certain companies can run with – Nintendo are family friendly and have the ease of use thing.”

Ghosts In The Machine: The Future Of Video GamesAN IDENTITY CRISIS Speaking of Apple – they’ve made grand leaps in user interface design and, in some ways, have made the grandest leaps in terms of evolving the control metaphor. What’s the next step?

“It’s rumoured,” Mark hesitates, “that there will be facial recognition put into the next iPad. One way or another the devices are going to start looking out. And that changes the emphasis, because when you change the interface and the device you change the spectrum of what’s possible. We’ve only recently opened this door.”

But where do Sony fit into this?

“Sony’s having an identity crisis,” laughs Mark. “They don’t really know what it is they want to be doing. With the PS3 it’s like they sold everyone a supercomputer and asked everyone to use Blu-ray discs! It’s a bit of a miss and I don’t know whose fault that is.

“As to where they should go, well… Sony has strengths in consumer technology that both Nintendo and Microsoft would kill for. They also have a wealth of intellectual property. It seems to me that they’re just not really good at figuring out what they want to do. Are we doing the phone? Are doing the handheld? Well, OK. What are they doing? Sony, I guess, should be trying to sell a seamless experience across its entire platform.”

But technically that is what Sony are trying to do – sell a seamless experience across an entire platform. The PS3’s new slogan is, after all, ‘it only does everything’. The problem is, according to Mark, is in the execution and a fear of committing whenever they, as a company, find their own point of difference.

“We saw this with the Eye Toy,” begins Mark. “Honestly – how badly did Sony f**k that one up? I was in Sony’s lab when they were working on this, maybe in 2001? Then they released it and they just sort of dropped it! So here they are – they’ve now developed a next generation interface. They could’ve gotten close to Kinect before Kinect.

“When it comes to consumer tech it’s all about presentation,” he continues, “There are two things that are going to work here – one is going to be your marketing and the second is the word of mouth. Sony’s got OK word of mouth. But the word of mouth was that it was really expensive. You have to fight this by having your marketing be so freakin’ cool. You have to have these two things working together.

“Look at Kinect – Kinect is like 170 bucks? For all that technology? It’s shooting lasers around the room! It’s amazing and Microsoft must have known – ‘if we wanna get it out there it’s gotta be cheap’.”

Ghosts In The Machine: The Future Of Video GamesTHE ACCIDENTAL GAME COMPANY If any company understands how to market and sell new technology to a mass market it’s Apple – and they’re shifting units one innovation at a time, which is perfect for a mainstream audience that doesn’t want to be overwhelmed by innovations they don’t necessarily understand. But when it comes to the video gaming space Apple are – how can we put this… N00bs?

“Apple is the accidental video game company,” laughs Mark. “The thing with the iPad is that Apple did not know what people would use the iPad for. Now, you show any child older than 18 months an iPad and you cannot pry it from them until they fall asleep! I’m serious! This is the way they want it – it’s become their personal screen.

“So in a sense Apple has been handed the entire game market for the next generation of children. It’s up to the developers to work out what they can do, or what they want to do with these interfaces. When it comes to touch, the way we play and the things we can do are going to be radically different. There are a whole bunch of directions this can go in. You’re right – this is absolutely Apple’s game to lose. But Apple is not a game company! They don’t even have a game division.

“They have to feed the beast. They’re already the hold the largest share of the handheld market, but are they doing enough to feed that beast? If I were running the company while Steve was away, I would maybe start up a little initiative to work out how I can feed the beast…”

Ghosts In The Machine: The Future Of Video GamesTHE VOLUMETRIC REVOLUTION But what about 3D? In the near future, at least, the technology being force-fed down our gullets is 3D. Sony are betting the farm on this new tech – using it as a focus to sell their seamless multimedia vision. Nintendo will be first to market with a mainstream device that delivers 3D visuals without glasses – will this be the game changer everyone expects?

“There are a lot of risks involved with 3D and the reason we haven’t heard that much about it is because there really isn’t that much 3D content yet. Thank God! People are popping in the occasional 3D movie and that’s pretty much it.

“I am a little concerned – Nintendo has expressly forbidden anyone under six years old from playing the 3DS? Why would they do that? It’s clear they understand there’s a potential issue here. I’ve been very clear when I talk to anyone – don’t let any of your kids under seven watch 3D at all. After that, well… we’ll see. We’re all adults and our lives are already ruined anyway!”

Mark Pesce, famously, has spoken out vehemently against 3D but even he is apprehensive about labelling it a fad. The problem, he believes, is the sheer force and marketing weight behind it.

“I’m not going to make a call on whether it’s a fad or not,” he claims. “It’s early days and there’s so much energy behind it. They really want this to work. I can think that it’s a train wreck, but they really want to push it. That kind of thing makes it into market all the time – because there’s enough force behind it.”

Ghosts In The Machine: The Future Of Video GamesHOLOGRAPHIC TECHNOLOGY – IT’S OUR ONLY HOPE According to Mark the real 3D revolution is in Holographic technology – and it may be closer than you think.

“Believe it or not, we are starting to see projection holography,” he begins. “They’ve got it working at a single frame every two seconds! And that’s already being fast-tracked.

“It’ll be something that occupies space,” he continues. “The only problem, of course, is that the processing power required is going to be of a completely higher magnitude. Everyone’s wondering what they’re going to do now that they can stick a thousand CPUs on a chip – well I think we have an answer! You’re going to need it to drive a holographic display, because they’re volumetric – they use voxels instead of pixels – you have to render the entire volume, not just the surfaces.

“I think you’re going to see that sort of thing being fast tracked. When you hit even 15 frames per second, I think people will be happy with that – to begin with. I think we’ll see that by about 2020, maybe even before that.”

But won’t we need some sort of new user-interface to navigate this volumetric space? Apparently not. Some of us already have the necessary kit in our own living rooms.

“Actually, Kinect is ideal for this sort of thing,” claims Mark. “Kinect is volumetric – it is literally 3D. It actually gives you points in space – it’s doing that work. Kinect is setting the stage for holographic devices and I wouldn’t be surprised if we started to see games that play on that – even it’s only in 2D. It will play off the fact that Kinect is giving you a volumetric space to play in.

“In this case the interface is leading the display technology.”

What’s becoming increasingly clear, especially over the last few years, is that the way in which we play games in becoming increasingly more important than what we’re playing – the success of the Wii and Kinect can attest to that. In that regard Mark Pesce is correct – the user interface is leading the display tech and showing us signs of what is possible for interactivity in the future.

We can only hope to be around long enough to see it come to fruition.


  • Couldn’t help yourself with the BttF reference, hey Mark? 😛

    Some interesting points raised in that article. I’d like to see some more of these with other people in the industry.

  • “Apple is the accidental video game company,” laughs Mark. “The thing with the iPad is that Apple did not know what people would use the iPad for…”

    Tell me about it. My father one an iPad at an industry convention as part of a Q&A quiz prize a few weeks before its release here. Over the last six or so months, it has only seen use as a Solitaire simulator, digital TV guide and encyclopedia whenever someone wants to correct someone else. But hey – XBMC has just been ported to iOS, I’ve been trying to convince him to let me jailbreak it so he can watch his recorded motorsport in the garage.

    Got a little off-topic there, sorry.

  • Another great article Mark.
    I think that Sony comes closest to providing the seamless experience because their tech is intergrated and will only be more so when they make up their minds about PSP2.

    “So in a sense Apple has been handed the entire game market for the next generation of children”

    This is so true, but doesn’t it scare the absolute sh*t out of you!

    • Scares the hell out of me.

      I’ve always said Apple was a cult, with their design > everything mindset. I’m concerned that it’s going to raise generation of engineers who build things that don’t work… but look pretty.

      • What could possibly be more important than design? design doesn’t just mean industrial design, it means user experience design. A entire generation of people has been brought up in world of computers designed for nerds and thankfully thats beginning to change.

  • Some of his opinions are sure to stir controversy (fanboi lolololololol), but it was a good insight with some interesting speculation and ideas – keep up the awesome features, Mark!

  • I’m sorry, but as soon as I saw Pesce’s name attached, I lost most of my interest in the article. He has become a media hound that just enjoys getting his name out there, like his totally unresearched rant about the dangers of 3D that he spouted everywhere a while ago, and his version of events around the original Sega 3D glasses, which were, well, let’s say his opinion differed from everyone else that I worked with on that project a lot of years ago. His scaremongering of not letting ‘any kids under 7 watch any 3D at all’ is just ludicrous and smacks of Reefer Madness. I’m sure if a kid watches one 3D movie that it will destroy his vision and processing for life. They get more eyestrain playing and crossing their eyes for fun and by reading.

    There just isn’t much in the article that doesn’t come under the banner of completely blindingly obvious, and almost nothing in the way of predictions. In fact the only prediction appears to be that people want holographic technology, which has been true since at least the 60s.

  • Mark Pesce is the voice of reason and a living legend. I met him (briefly) at the Tech 23 conference, and he was knowledgable and friendly. Something you don’t usually see in professionals.

    I agree with almost everything said here, also! xD

    • I would be more impressed if there was a futurist who could actually say, “I said what is happening now would happen now, 10 years ago”.

      Futurists are up there with Astrologists.

    • It is his habit of quoting non-existent or highly questionable research that gets under my skin, and re-imagining of events that I and many others worked on.

      I think the mulit-touch interface will change gaming and computing way way more than kinect. Kinect type tech is mostly good for dancing, motion capture and sports/rehab. It will probably change physio, but not much in the way of interface and gaming. A lack of tactile response will keep it out of the UI for anything vaguely useful.

  • Mark, that picture of the TIE fighter… is that from the solid state projection demo MIT was showing last year?

  • I completely disagree that Kinect is the biggest interface revolution since the mouse. Not when multi-touch screens are around as well. Combined with a well-designed UI, a multi-touch screen is intuitive enough that toddlers can figure it out. Plus there’s the tactile element. There’s kind of a disconnect with the Kinect, where you’re manipulating things in an entirely virtual space, without any real feedback other than visual.

    Also, while it’s cool for a few things, as an interface itself, the Kinect is very limited, especially for games. You can’t even approach the necessary complexity to be able to do most modern games with it. I think on its own it’ll end up being a dead-end technology. Instead, we’ll see elements of it being integrated into successor technologies. What it enables is being able to track the orientation and position of a specific person in an environment. Essentially, it’s a computer vision device, marginally repurposed.

    3D is an absolute gimmick and the sooner it goes away, the better. Holographic tech might come around, but they’ve been saying it’s ‘only ten years away’ since the 70s.

    Personally, I think we’re far more likely to see things go in a different direction. What is really limiting us isn’t really our input devices, it’s our output. We’re tethered two fixed 2D panels. Screw 3D. Move this information directly into our eyes. Bring back the old VR goggles, except instead of bulky, heavy and expensive contraptions, you have essentially goggles or even glasses, which can overlay an image into what you’re seeing. Combine this with highly accurate GPS, CCTV systems with Kinect-style person-tracking and you have a setup which could allow for entire virtual worlds to be projected into the real world. Augmented Reality games would be an enormous boom area. We’re talking Star Trek Holodeck stuff here – giant room with Kinect-style cameras mounted at the appropriate places to track all the participants in all dimensions, lightweight head-mounted displays to project the augmented world the players are seeing into, and they have full range of motion in the area while they’re doing it. It’s probably a decade or so away too, but unlike Holographic projection, many of the necessary pieces of the puzzle are technologies that already exist and are nearly cheap enough to actually mass produce.

  • I think he was a little off the mark with apple. They haven’t been handed control of the games, they’re just selling the medium. It’s like saying television makers were handed gaming when the first console came out.

    No, what apple have done is provide a platform for indie developers. That’s who is gaining ground through the idevices and smart phones.

    Apple are simply benefitting from it.

    Also, Nintendo are warning against children under 6 using 3D because they are family-friendly and don’t want to hurt their key demographic.

  • Interesting article… but as always the Grammar Nazi in me can’t help but point out a typo. “It’s a bit of a miss and I don’t know who’s fault that is.”

    who’s -> whose

  • Kinect is interesting. It doesn’t have a place in traditional fantasy-gaming right now, though. We can’t use it as a “virtual reality” enhancement, because there is nothing haptic about it. You can’t put a head tracking set on a person and tell them to go explore in their virtual world, because you can’t simulate movement without the environment that provokes it. For the purposes of a traditional interface, it’s inferior to a traditional interface.

    Holographs have a possible value in the entertainment sector, but they have an issue: They cannot create the illusion of depth. Popular 3D technology works for games and entertainment, because they function as a window to another world. Holographs simulate objects in true 3D space. Their hugest flaw is just that; they occupy space.

    My cramped man-cave already is incompatible with kinect-style motion games. I’m happy with that, there’s nothing worth playing on the silly thing. Try and stick something holographic in here? Are you nuts?

    But then, we have the universal failure of holographic technology for interactive software: haptic feedback. Humans crave to touch. There’s no method to simulate a weighted object. Until there is, it’ll be total niche.

Show more comments

Log in to comment on this story!