HoloLens Hands-On: This Shit Is Legit Bananas

Last night, we were given our first personal glimpse of Microsoft's augmented-reality device for Xbox One; the HoloLens. This wireless transparent visor creates the illusion of tangible objects in the real world by adding graphical elements to the player's surroundings — imagine being inside Who Framed Roger Rabbit? and you've got a pretty good idea of how it all works. Over the course of an hour, we built Minecraft worlds on a table, shot at rampaging robots crawling through the walls and even stepped inside the Halo universe. So is this stuff truly game-changing? It just might be.

Our hands-on HoloLens session was conducted behind closed doors at Microsoft's E3 Xbox booth. It was all super-confidential with no photography permitted and only a handful of journalists allowed to enter. (There appeared to be a grand total of two Australians there, including me.)

The first demo we were shown was an interactive intro to Halo 5's Warzone multiplayer mode. Attendees were each fitted with a HoloLens and ushered into a physically built set that resembled a Spartan stronghold complete with gunmetal corridors and a command deck. The environment already looked pretty impressive but the HoleLens turned it into something truly spectacular.

At first, the only addition to the real-world environment was an interactive HUD display on the visor — this directed the wearer around the set and let them know when and where to stop walking. It was a subtle addition, but it definitely helped to make me feel like I was inside a Halo game.

Things then kicked up a notch — after being prompted to stand next to a wall, a blast hatch opened up to reveal a huge war base teeming with UNSC soldiers and weaponry. In reality, we were still just looking at a wall with the HoloLens "painting in" the graphical elements. This was much more than a simple 2D projection: by leaning forward, it was possible to peer around the sides, just like a physical window. Impressive stuff.

We were next moved to the command deck which was dominated by a holographic display that showed nifty projections of our mission — this included battleground maps, enemy models and myriad spinning 3D effects. Again, the hologram's base was the only physical element in the room with the HoloLens providing the rest. And with that, it was back to reality.

The Halo demo provided a great example of how this tech can be utilised in a commercial setting. Obviously most people aren't going to build a Halo set in their homes — but inside an arcade-style environment it could be phenomenal. If nothing else, it should make the wait while queuing for theme park rides far more tolerable. Plus, it will make laser tag balls-out awesome.

The next HoloLens demonstration was for an FPS action game dubbed "Project X-Ray". This started out life as a tech demo but is now being considered for official release; much like Wii Sports on the Nintendo Wii. It involves blasting various robotic beasties which burst through the walls of your house and using an X-ray power to find where they're hiding. Your weapons are controlled with a standard Xbox controller with the visor taking care of movement.

The enemies are spatially aware which allows them to navigate the real-world environment as if they are actually in the room. This mainly involved them ducking in and out of virtual holes in the wall and flying towards the player's face to launch attacks. We were particularly impressed by the way you could duck under enemy lasers — a true AR experience.

Project X-Ray showed off HoloLens' intriguing potential as an in-house gaming peripheral. While the actual game was pretty one-note, it's not hard to imagine deeper and more complex gameplay possibilities using this technology. Colour us excited.

The final demo happened to be the most impressive of the bunch: a dedicated version of Minecraft for HoloLens. By now, you've doubtlessly seen the onstage demonstration of this game from Microsoft's E3 press conference. But you really need to experience it first-hand to properly appreciate how special it is.

The game uses a combination of finger gestures and voice commands to navigate the world. Players can throw their map onto a wall and choose between various screen sizes; a handy way to get your Minecrack fix when your roommate is hogging the TV. The HoloLens essentially tricks the wearer into seeing a screen on their wall. In fact, it's more like a window — like the Halo demonstration, it was possible to peer around the sides to see more of the game world beyond. You can also leap inside the game world and control your character directly with an Xbox One controller.

Of course, this is just one way to play the game: the chief draw card is the ability to create fully interactive 3D maps on any flat surface. This was truly amazing. Pinching your finger allows the world to be dragged around in any direction while a downwards tapping motion selects menu items.

Get A Load Of Microsoft's Holographic Vision For Minecraft's Future

You can also use voice commands in conjunction with the HoloLens to trigger specific game actions. For example, focusing the cursor on a dynamite block and saying "lightning" achieves the desired effect. Similarly, saying "zoom in" brings you nice and close to the action.

I'm not sure what was powering the game's voice-recognition but it was ridiculously impressive: even with my broad Aussie accent to contend with, the game didn't mishear me once. That's bloody first-grade alchemy that is.

There are a range of different game modes on offer. We were particularly enamored with the "follow character" option which automatically tracks the selected avatar as they potter around the game world — it's like having a flying drone for eyes. Amazing stuff.

Once you get used to "pointing" the cursor with your eyes, it all works pretty seamlessly. There was certainly none of the extreme trial-and-error associated with most Kinect games (other than the setup, which we'll get to in a moment). It's become abundantly clear why Microsoft shelled out so much money for the Minecraft licence. This thing is going to make fans of the franchise lose their minds.

So was our HoloLens experience faultless? No. We encountered some significant issues during our hands-on session which we hope are ironed out before launch. First and foremost, the calibration process is currently a bit ridiculous: you need to have your eyes measured with a separate focusing gizmo. No, really.

Presumably the final HoloLens product won't require additional calibration hardware to run, but we're extremely leery about how Microsoft aims to solve this problem. Let's not forget that the setup process for the Kinect was an absolute nightmare.

The visor's functional area is also a little on the small side — your peripheral vision frequently disrupts the illusion with objects zipping in and out of existence. We also aren't completely sold on the design of the current visor: it seemed alarmingly fragile for a consumer device (we were told not to touch the glass or sensors under any circumstances) and the overall fit was a bit off. While it's possible to adjust the straps at the back, it's extremely easy to end up with an uncomfortably pinched nose. Some more time is needed at the design stage, methinks.

All in all, HoloLens for Xbox One is a mighty intriguing proposition that is guaranteed to deliver unique game experiences. Like the Kinect, we imagine its potential outside of gaming is going to be huge. As with any video game peripheral, it now comes down to crazy and brilliant developers to unleash the platform's full potential. We're waiting with bated breath.

Kotaku attended E3 in LA as a guest of Microsoft.


Comments

    Certainly ground breaking stuff for the home. This is definitely on my want list. Would love to get on the devwagon for this

    It's strange, as someone who isn't a fan of current VR technology and sees all the flaws that stem up between what's being shown here and what will actually end up in our hands I shouldn't be this into AR, but this is the first time in a long time I've been genuinely excited about something that may end up being game changer tech.

      The VR tech doesn't excite me as much for gaming purposes, for me it looks more comfortable and suitable for immersive videos and stuff like that where it can be in really high def but the hololens excites me more as a gaming device I think

        What sort of gaming experience are you seeing that has you excited?
        For me VR takes you to somewhere else, somethign like the CCP valkyrie game to me is amazing. sit back and really be taken to another world.
        All the stuff mentioned above as AR seems so gimicky, the actual gameplay seems limited and cool for a moment but aliens bursting through walls or whatever are very limiting.

        Not trying to quench your hype, but actually curious as to what you see this tech being used for in the gaming world?

          I just see it working better at what it does and being more practical than vr is at this point. The vr demos are awesome but the actual game play needs everything to go just right for it the not be broken. I guess it's partly that i can see hololens type ar being easier to fulfill the experience it is promising. I'll likely get both so i hope they both find their place

          Last edited 18/06/15 5:28 pm

            Interesting.
            I see it the other way around, that trying to work with whatever in in the environment is going to be a lot harder than a dedicated world where you only see what is being projected. I see lots of pop through issues and trouble with stuff moving around you.
            But for me the biggest issue is I don't see what sort of game is going to benefit from AR. Maybe new genre will be created and some interesting stuff will be done, but I can't see what that is right now.

          For me VR takes you to somewhere else

          For me VR simply shows you something else. Or at least the current crop of VR systems. VR is built around the idea of merging the player and the avatar, which it hasn't made any significant progress towards in, what? 30 years? Best case scenario is games that work on porthole systems like racing games and flight sims. Those are neat but there's not really much meat to them.
          Ultimately current VR systems just don't do what's advertised on the back of the box. They don't put you in the game and that's pretty much all it exists to do from a gaming perspective (unless you want to convert it to AR with captures). It just seems VR is at a dead end. It's been getting mountains of cash pumped into it, tons of really, really smart people are looking to make that breakthrough and win the jackpot, and yet nothing is materialising.

          AR may be advertised as breaking the fourth wall, 'bringing the game to life' and doing similarly impractical things that won't actually work that way in the real world, but it's applications are more varied and practical thanks to the fact it doesn't isolate you from reality. It has a lot of potential to change the way we interface with systems which could be huge for games. It's not trying to create an entire reality just augment the existing one.
          Imagine playing Rock Band with AI band members standing right there in your lounge room. Imagine playing Mass Effect with the conversation wheel outside of the TV. Imagine GTA with the mini-map/GPS on your coffee table instead of shrunk down in the corner. Imagine playing Theme Park with your bedroom as the park. Imagine keeping your Pokemon out even when you're not playing the game anymore. Imagine playing Mouse Trap except it actually works.
          It can use digital solutions to break the limitations of screen based displays. That's obviously going to come with some drawbacks and it's not as simple as just creating AR entities and dumping them like physical objects, but there's room to grow towards that and far, far less pressure to find a magic solution to those problems just to achieve basic success.
          If this is as far as AR goes there's still something there. If all they do is up the specs like they do with VR there's still something there. However if this is just the beginning we're going to see AR grow into something pretty huge.

          It's probably not going to go as far as I'd like. VR or AR a headset is still an optional device, so unless Microsoft, Sony or Nintendo release a console that ships with a headset or the tech goes Wii level viral games are going to struggle to get made and (like with the Kinect) developers are going to struggle to move beyond gimmicks because uncharted waters are scary. I just feel like AR has way, way more potential for gaming than headset centric VR.

            I think I would probably need to actually use one of the AR headset to get a feel for what is possible, I've used VR and can see the potential. Even as far as the arcade style things that Chris is talking about when he references Game of thrones. I just don't see that with AR, maybe because of the lack of quality content being shown.
            Everything I've seen and most of what you have mentioned just sounds gimmicky, but maybe it would work wonderfully in practice. Augmenting your existing gameplay experience, could work, but making you look away from the screen to see a map or something, I'm not sure. Seem more annoying than helpful. I guess having your table or something away from the screen as you inventory could work.
            I guess with all that you mention I see VR working better. Why would I want to have the band in my tiny little apartment when I could be on stage at one of the largest music festivals in the world, looking out over a massive crowd.

            tldr: I need to use AR before I can see the possibilities

              It sounds like you're expecting AR to try and be VR. It's not about putting things on the table so you can sort them by hand or pretend they're real, although simulating 3D objects is a cool part of the tech, it's about breaking open the GUI so it's not confined to a 42" screen. Your inventory doesn't have to be on the table, it can just be on the wall or floating. Your map can be the size of a can of coke (or not there at all) until you hit pause, at which point it becomes the galaxy map from Mass Effect. Did you ever play Dead Space? Think how the holograms work in that. You'll almost certainly use a gamepad and pause to interact with it, but that doesn't mean it has no merit when compared to a traditional pause screen.
              These may be gimmicks but I don't think that means they should be dismissed. Where something like Fruit Ninja Kinect has a gimmick core that makes it all about waving your arms, AR opens up optional gimmicky stuff that can be placed on almost any traditional game. It can be 100% gimmick in one developers hands but it can also be a series of minor improvements to core functionality in the hands of another developer.

              Why would I want to have the band in my tiny little apartment when I could be on stage at one of the largest music festivals in the world, looking out over a massive crowd.

              If you can do that that'd be great, but with the Rift at best you'll be able to look out through the eyes of a puppet. The AR version allows you to see your guitar, reach down and grab a drink, see your friends playing co-op. You can still look out and see the crowd, but your lounge room is on stage. That sounds counter productive if your goal is VR, VR is all about substituting one reality for another so you want to block out the real world as much as possible, but if your goal is to just have fun playing a game it's super cool.

              Like I said in my original comment I'm definitely too optimistic about AR. I just find there's potential here that doesn't require a magic wand to meet. There's concepts to be explored that don't hit the same dead ends every time you start to do something significant.
              When we talk about the flaws with AR it's all 'well, this doesn't sit properly, the field of vision is bad, latency causes problems'. Those might stay a problem forever, we may already be at the absolute limits of sensor technology, but the solutions are pretty easy to imagine. When we try to imagine the fixes to VRs problems we land in these over complicated half solutions. You can move 1:1 with this huge setup, but you can't simulate touching anything that's not actually there. When we just assume that technology is magic that can fill in the gaps the best we come up with is Pacific Rim style rigs that still don't quite work. After that it's just Matix style jack into the brain tech that completely ditches all the current VR technology.
              Show me something that lets me to do more than just look at a virtual environment and pilot a puppet in first person and the I'll get excited. Until then I hesitate to even call headset VR virtual reality.

              Sorry if this seems a bit ranty. I don't care if you like AR or VR. The concept doesn't need me to fight for it. I'm just trying to explain where I'm coming from because I find AR so much more interesting as a concept.

    I was thinking that we could see a resurgence of arcade visits if companies start making dedicated vr/hologram environments that you can't match at home. These home devices sound great but with a dedicated space you could make some insane set ups.

      My thoughts exactly: http://www.lifehacker.com.au/2014/07/could-game-of-thrones-on-the-oculus-rift-spark-an-arcade-parlour-resurgence/

    Yu-Gi-Oh Hololens when

      I will buy that, I would like a significantly larger FOV though.

      Needs a 'It's over 9000' app

    And here I was thinking Occulus and such were the future. *Looks at Dk2 and pushes it off the table.

      [catches it before it hits the ground and runs away, thanking you]

      Last edited 18/06/15 7:08 pm

    I am still cautious. Kinect was a massive letdown, so will be wait and see.

    Okay, I'm interested. Particularly in how the tech will work with other games. I'd like to see something like this combined with TrackIR to provide persistent & movable HUD elements for something like Elite Dangerous while you're moving the view

    It needs to cover the whole lens area and not just a middle area of it. Getting rid of the peripheral view area issue is important (without just blacking it out into tunnel vision). I will wait until that actually happens, until then no thanks.

    I also hear objects rendered jiggle into place and when you move around such as shown in all demo videos, objects will not be perfectly still when moving due to respond latency of the head tracking!

    MS have done a good job selling the idea, but what they show and what it is, are two separate things IMO...

    Last edited 18/06/15 2:31 pm

    Honestly, I see more commercial applications for this than gaming ones.

    Watching the segment during their press conference I was thinking that this would revolutionise sand tables for military deployments and excercises, or pre-surgery walkthroughs for surgeons, or large construction project design meetings... stuff like that.

      Anything that's good for military sand tables is good for real-time strategy games. Can't wait =)

    Starting to feel bad about getting old. Some pretty awesome stuff coming out.

    I just want a HUD in real life. I want to walk down the street, tell my phone I want to find an atm and have an objective marker appear in the distance.

    Give me that Microsoft. Give it. Now.

    How accurate is your finger gestures being relayed to the Hololens. Is there input lag?

    Am I the only person who is reminded of the AR cards that come with the 3DS with this? There is an inbuilt app called "AR Games" on the 3DS which stuff like virtual golf course style puzzles on table tops, including holes and hills that do mot exist in real life.

    I know it has mot really been used much on the 3DS since, but that is really old tech, it came with the 3DS at launch.

    Sure, the Hololens allows you a larger field of view and it may free up your hands for gestures and take voice input, but it just does not strike me as new.

    For that matter, considering how hardly anything was made of the 3DS AR tech, I just doubt much will become of the Hololens either. :-S

Join the discussion!

Trending Stories Right Now