One of the most interesting parts of the Nintendo Switch has gone nearly unused by game developers, including Nintendo, for the first year of the system’s existence. A mind-blowing demonstration last week, however, showed that the little sensor on the bottom of the right Joy-Con is more capable than we ever imagined.
Specifically, it can be harnessed to transmit simple video feeds to the Switch screen or even detect heat sources. The sensor’s capabilities have been hiding in plain sight for a year, and are about to be used to power an impressive array of gaming experiences.
Nintendo shows off the IR camera sensor at a Switch showcase event in Tokyo in January 2017.
Nintendo first showed off the sensor at a pre-launch Switch event in January 2017. There, the system’s producer, Yoshiaki Koizumi, pointed out that the right Joy-Con’s “motion IR camera” could be used to detect the shape and distance of something held in front of it. He showed that the sensor could distinguish the shapes of a hand making gestures in the game Scissors, Paper, Rock.
The motion IR camera was used in the March launch game 1,2, Switch for an eating contest challenge. With the sensor pointed at the player’s face, the system could track the opening and closing of a player’s mouth.
1,2 Switch is the rare Switch game that already uses the right Joy-Con’s IR motionsensor.
The sensor wasn’t used for many other 2017 games. It’s understandable, given that it is positioned to point at the the floor or at a player’s torso when they’re holding the Joy-Con. A Resident Evil re-release seemed to use it to track a gesture for reloading a gun, but otherwise, it hasn’t seemed to be good for much.
But the Joy-Con (R)’s humble sensor is getting its day in the sun with the Nintendo Labo project, which combines the various parts of the Switch console with folded cardboard.
Many of the Labo builds involve holstering the right Joy-Con so that it points inside a cardboard piano or backpack or other enclosed space, where it can then detect the movements of specially marked stickers, sending signals back to the Switch to, say, emit different piano sounds or register different motions in a robot punching game.
Image via Nintendo’s trailer for its Labo project.
What’s more extraordinary, and what Nintendo has barely hinted at up until now, is that the “motion IR camera” can indeed function as a camera. A Labo game that positions the Joy-Cons on the side of a skittering cardboard car is set up so that the sensor takes transmits a live, infrared feed of what it is pointed at to the Switch’s screen.
In that same car mode, the IR sensor can be switched into a heat-sensing view, which it allows it to display which sources of heat it can see.
And when the car is set to auto-drive, that heat-sensing mode, combined with vibrations of the Joy-Con controllers, can be harnessed to make the car move toward any heat source. That could be your hand, or your dog, or your fireplace, so watch out next Christmas morning.
Suddenly, the mind boggles at what Nintendo can do with a camera that’s been built into every Nintendo Switch, but it’s clear that Nintendo has already been thinking of some fun stuff. After all, the initial trailer for Labo showed the right Joy-Con being used as the lens of a big cardboard camera.
That may seem structurally redundant, and it’s not like the IR motion camera is going to be shooting high-res shots, but there’s still some pretty cool potential here:
Image via Nintendo’s trailer for its Labo project.
It’s enough to make you wonder what other secrets are buried in the Switch.
Comments
5 responses to “The Switch Joy-Con’s Infrared Sensor Is Cooler Than We Thought”
Heat sensing view? Internet I expect Predator parody immediately!
Or as cool as anyone who’s played with a Leap Motion thought 😛
Don’t know why they didn’t put one in both joycons though, could have had far more possibilities that way.
Cost, most likely. Remember how much everyone whined about how expensive joycon are?
Yeah…
Funny though. They mask the price difference even though the right joycon clearly must be far more expensive to make.
It has this IR camera and the NFC reader in it, while the left joycon has nothing. (that the right one doesn’t)
I wonder why the distribution of tech between the two is so unbalanced?
Yeah, I always thought the left one had the NFC reader in it but it was the right. And yet the left one was the one that had the connection issues, seemed so strange.
So that’s how that Labo piano works! I was wondering how they sensed the key press, and how they made it polyphonic. Pretty neat solution, really.