Picking up where Pokemon GO leaves off, Microsoft's Hololens delivers a new way of looking at the world.
Virtual reality is stealing the headlines this year as the major VR headsets finally hit the shelves, but in the long term I suspect augmented reality will have a greater impact on our lives. VR whisks you away to a fantasy land, blocking out the real world, while AR seamlessly blends the digital and physical realms to place virtual objects into the world around you.
After an early limited release Hololens has just gone on sale to developers in the US and Canada for US$3000. Right now it's aimed at business users and Microsoft doesn't have any immediate plans to sell it to the general public, so you might consider it a work in progress compared to what might eventually be offered to consumers.
Melbourne software developer Two Bulls was one of the first in Australia to get its hands on Hololens headsets and prefers the term "mixed" reality to "augmented" reality because Hololens is capable of doing far more than just pasting graphics over your view of the world.
Pokemon GO offers an example of basic augmented reality, letting you hold up your phone to see the Pokemon standing before you in the real world. If the game was designed for Hololens then those Pokemon wouldn't just be pasted on the top, you'd be able to walk around them to view them from any angle.
Like a real world object, the Pokemon would appear larger as you got closer to it, with more detail, and you could reach out with your hands to interact with it. Even more impressive, if there was an object between you and the Pokemon, such as a tree, then the Pokemon would seem to disappear from view behind the tree as you moved around.
All this spatial awareness is thanks to several cameras built into the Hololens headset. It uses two cameras and an infrared sensor to build up a 3D model of the world around you – so it knows the dimensions of the room and location of objects. In this way it's similar to Google's Project Tango AR tablets which were on show at the Google I/O developer conference back in May.
Microsoft's Hololens headset is surprisingly comfortable considering its bulk, and it isn't as front-heavy as something like the Oculus Rift VR headset used by Melbourne's Zero Latency VR system.
The dual-band design helps distribute the pressure of the headset across your head, with a thumb wheel to tighten the main headband at the back. On one side you've got brightness controls above your ear, on the other side volume controls for the built-in surround sound speakers.
The computer-generated objects are projected onto the lenses in front of your eyes, offering a slightly different view to each eye for a true sense of depth. Those projected images line up perfectly with your view of the real world to create the augmented effect. Two more cameras in the headset record your view of the world, complete with AR effects, and can send it to a computer via Wi-Fi.
Unlike most VR headsets, the Hololens is completely wireless and self-sufficient as it has a tiny built-in computer running Windows 10. You can walk around the room without the need to be tethered to a computer or rely on external tracking cameras, offering a more engaging experience than VR headsets which require you to stand still.
The headset's cameras are accurate enough that they can track your hands, letting you reach out to interact with objects as well as use gesture-based controls – similar to Intel's RealSense 3D cameras which will be built into its upcoming Project Alloy headset.
I first donned the Hololens headset to discover a large cricket ground floating in the middle of the Two Bulls open plan office in Collingwood. Roughly five feet across, the cricket ground remains perfectly aligned with the floor and doesn't waver as you walk around to view it from every angle.
Looking down at the cricket ground, I could watch the bowler run up to the crease and then flick my finger to swing the bat and hit the ball. After a few tidy singles I managed to smash the ball out of the ground, watching it sail through the air and then bounce on the office floor and keep rolling. Wherever I stood, the digital and physical realms lined up perfectly – allowing the pretend ball to roll across the real floor.
You can adjust the brightness of the augmented content but in this sunlit office the stadium looked slightly transparent (more so than in the screen shots), yet still very believable. In a darker room objects appear more solid. Unlike VR, you can still see the real world and interact with people around you, so there's no sense of isolation or disorientation.
Looking around the office I could see the Earth and Moon floating before me, along with animals sitting on the floor. I could reach out with my hands and touch these objects, pinching their corners to manipulate their size and orientation.
Each object has small menus on the side, so you can look at a menu option – assisted by the headset's built-in pupil tracking – and tap your finger on your thumb to select it just like a mouse click.
As an alternative to tapping your thumb, the headset also supports a one-button wireless remote control which acts like a mouse click. Just look at what you want to select and then press the button. This might seem like cheating but at times it feels more practical, especially as it means you don't need to hold your hand up so the headset's cameras can see your gestures.
The "bloom" gesture – opening your hand upwards like a flower opening its petals – calls up the Windows 10 start menu. From here you can take screen shots, launch apps and even dive into the Windows store to download AR content. The headset also supports voice commands.
One limitation is that the Hololens has a relatively narrow field of view, you can see augmented objects straight ahead but they don't extend to your peripheral vision - it's roughly a 30 to 40 degree horizontal field of view.
More frustrating at first is that it has limited vertical field of view and if you don't get the headset to sit correctly on your head it crops off the very top of the augmented display.
When you look closer at the Hololen's design the reasons for these limitations become clear. The headset isn't projecting augmented content onto the entire visor, only on to small rectangles – similar to bifocal glasses but located in the centre of your vision (you can see this in the photo above).
You need to be looking through these small areas to see the augmented content. If the headset sits too low on your face you end up looking over the top of these areas, as if your reading glasses had slipped down your nose.
This is frustrating at first but not a deal-breaker and you soon forget about it once you get the headset to sit in the right spot. The field of view issue only becomes noticeable again when you're dealing with fast-moving action such as games which require you to look quickly around the room.
Stepping into a small office we fired up Microsoft's Robo Raid first person AR shooter with flying robots bursting through the walls. With target cross-hairs overlayed on my view of the room I could line up the robots by looking at them and then blast them out of the air, using the remote in my hand to fire.
At this point the limited field of view made itself felt again, as I could hear the robots breaking through the different walls but had to keep turning my head because I couldn't glance at them in my peripheral vision.
If Microsoft was actively pushing Hololens as a gaming platform I'd be more critical, but it's easier to forgive the field of view limitations knowing this isn't intended to be a consumer-grade gaming device.
Turns out I'm dynamite with a laser beam, testament to the headset's accurate tracking which ensured the real and virtual content remained in alignment. I was also assisted by the headset's built-in accelerometer and gyroscope which let me duck and weave as my flying foes returned fire.
Miss your target and you blow a virtual hole in the real wall, exposing the beams beneath. These holes are more than flat projections, you can walk across the room and peer into the hole from any angle. Used in any empty room with white walls, the Hololens could certainly offer a holodeck-style VR experience and you can create AR content using the same Unity programming environment popular with VR developers.
When will we see it?
Two Bulls is developing commercial Hololens applications for several Australian clients, including businesses in the construction and retail sectors, says Chief Operating Officer, Evan Davey. He doesn't expect this first-generation Hololens will ever be targeted at consumers, but Microsoft may broaden its horizons down the track as it hones the design.
"The first time your average person experiences Hololens it will probably be in a retail store - where they're simply trying to sell you something else – perhaps road-testing a kitchen renovation or seeing how a new couch will look in your lounge room," Davey says.
"Right now the hype is around virtual reality and a new generation of games but in the long-term augmented reality's practical real-world applications might make it the real game-changer. Hololens is just a taste of what's to come."
This story originally appeared on the Sydney Morning Herald