1.1 Distinguishing VR, AR, and MR Systems

At NASA, we're excited to apply mixed-reality technologies to the challenges we're facing in space exploration. Through a collaboration with Microsoft, we're building applications to support engineers responsible for the design and assembly of spacecraft. Astronauts working on the International Space Station and scientists are now using our Mars tool, OnSight, in mission operations. OnSight is a powerful tool for our scientists and engineers to explore Mars, but because we always felt it shouldn't remain only within NASA, we've taken the core of OnSight and made an amazing experience that allows the public to explore the red planet. We call this new experience Destination: Mars.

Mars can be a lonely place, so we've added photo-real holographic captures of an astronaut and a member of the Curiosity rover team to be our guides on this journey. This gave us the opportunity to immortalize a hero.

Hi there, I'm Buzz Aldrin.

To help Buzz explain how we're doing science on Mars today is Curiosity rover driver Erisa Hines. Welcome to my office. We can put the public, the rover and Erisa together at the exact place where Curiosity made some it's most amazing discoveries. We're looking forward to opening the Destination: Mars exhibit at the Kennedy Space Center Visitor Complex in summer 2016. We can't wait to share this journey with the world.

Let's go to Mars!

Today I get to show you something very special. It's something we've been looking forward to for many years. When we set out to build HoloLens 2, we wanted to create some things that you didn't need to learn how to use. You should just know it. And we call this, instinctual interaction. Let me show you. Now, as I mentioned the HoloLens is very comfortable, it fits just like a hat. And the only thing that's even more effortless is how I'm automatically signed in. With Windows Hello and RS syndication, the Hololens 2 is actually signing me in as I put on my device. Now, not only does the Hololens 2 recognize me, it also recognizes my hands. Look at this. Fully articulated hand tracking. And as I move my hands around, the Hololens is actually calibrating to my unique hand size. And of course, not only does the Hololens 2 recognize me and my hands, it also recognizes the world.

Welcome to my mixed reality home. This is the place where I have all the apps and content that I use every day. Let's check some of them out. Now, I've seen many people put on HoloLens for the first time and the first thing people do when they put on the device is they reach out their hands and try and touch the holograms. And now you can. Look at the way that the hologram is responding to my hand, almost inviting me to touch it. And, in fact, I can just grab this corner to resize it. Or I can rotate it or move it. That's right, we are touching holograms. This is an app I've got called spatial. Let me just put it right there. I've got another app here called vurfoiral view. It's a little big, so let me just use two hands here to make it smaller and then rotate it so you can see. There we go. And then let me put it down here near spatial. Maybe make it bit smaller. Yeah, that's nice.

Alright, now let's switch gears and talk about a different kind of application. I've got a browser over there but it's kind of far away and I don't really want to walk over there. So let me just call it over with my voice, "follow me". This is a browser that's running Microsoft Teams, which is a tool that we use back home to collaborate. Let me see what the team's been working on. Ok, it looks like they've got a surprise for me in the playground app. I just have to say the words, "show surprise". Alright, so let me just open up that Start menu here, and then place the app and then launch it.

So now we're actually exiting my mixed reality home and going into an immersive experience. But notice that that browser that I had, actually followed me in. Now, this can actually be really useful when you have things like emails or PDFs that you need to reference while you're doing your work. I don't really want it following me around though, while I'm showing you all this cool stuff. So let me just put it over here and then we'll get back to it later.

Welcome to the playground. We spent years exploring and refining interactions for HoloLens2 and the playground's just a tiny sampling of the many prototypes that we built, tested and learned from. Our approach was basically to try out as many things as we could and look for the things that stood out. So, for example, here I've got three sliders. Each of them is controlling this wind farm simulation, but each in a different way, using a different interaction technique. One of the things we tried is this touch slider here. So here I can just stick my finger in the slider and have it go to a particular value to control that wind speed there. It felt okay. We also tried this push slider. So this guy you kind of nudge from side to side, kind of like an abacus, which was interesting. Now the interaction that really took home the cake though, was this pinch slider. The way that works is you just pinch it and move it wherever you want. And what we found was that people really like that tactile sensation of their fingers touching as they grab and then release. And across the board, for all interactions, the audio and visual feedback as a grab, move, and then release, were all really critical for making this experience feel connected. Oh, this is just so satisfying. I can't wait for you all to try this out.

Alight, now let's move on to a different kind of control, buttons. How do you press buttons on HoloLens 2? Well, you just reach out and press them. Now, one interesting thing that we found about buttons was that the size of the buttons actually impacted the way that people interacted with them. So, for example, for the smaller ones, most people would use one or maybe two fingers. But for the larger ones, pretty much everyone used their entire hand. And this is kind of interesting when you think about it because these objects don't really weigh anything. They're just digital things. But despite that, people would treat them like real things, almost as if the bigger one had more weight. I just love the way these move and the sounds they play when I press them, it's great. Alright, how about something that uses all ten fingers. Well, to test that out, we built a piano. So here I can just play a chord or I can play the keys one at a time. [Music]

Alright, so where's that surprise that the team had for me? Oh, that's right, I had to say those words. Show surprise. Ooooh, look at that hummingbird over there, it's gorgeous. I wonder if it will fly to my hand. Yeah. Oh wow. This is beautiful. I just love the way that it's following my hand around. I've got to tell the team they've done a great job. And in fact, I don't even need to use my hands to do this because I can use my eyes and my voice. That's right HoloLens 2 has eye-tracking. So I could just look over to this browser here and look at the bottom of the screen to scroll it, and then send my message. Start dictation. The hummingbird looks great, exclamation mark. Send.

So this is what we mean by instinctual interaction. By combining our hands, our eyes, and our voice, HoloLens two works in a more human way.

Anand Agarawala: Today companies tackling the world's biggest problems are increasingly spread across the world. And the hololens lets us work together as if we were standing next to each other face-to-face. To show you how that works, let me just flip into spatial here with my HoloLens 2 and materialize the room. Hi Jinha!

Jinha Lee: Hi everyone, it's great to be here on stage holographically.

Anand: It's great to have you here Jinha. Can you tell everybody a little bit about what you're seeing?

Jinha: Sure. I can see your lifelike avatar which we generate in seconds from a 2D photo. And we can walk around the stage with spatial audio and use this whole space around us to collaborate as if we're all here together in the same room.

Anand: Cool. Now to show you how companies are using Spatial to transform the way they work, let's invite Sven Gerjets, CTO of the iconic toy brand Mattel onto the stage.

Sven: Hey guys, how awesome is this. So at Mattel, we're undergoing a massive digital transformation. It's touching all aspects of our business. This includes the way that we're using technology to design and to develop our products. Our classic brands like Barbie and Hot Wheels and Fisher-Price, they have diverse teams of designers, engineers, marketers, and manufacturers that are spread all over the world. They can now come together in a spacial project room, reducing the need to travel as much, to get everybody on the same page. When we're developing toys, like this Rescue Heroes Fire Truck here, we're able to pin up dynamic content on virtual walls, like this powerpoint for instance, or videos, or even concept art. Teams can now rapidly exchange ideas and create a shared understand you to find potential issues much earlier in the cycle. For instance, guys, take a look at this wheel clearance here.

Anand: Oh yeah, I can see that's gonna be a problem. So I can pull out my spatial phone app and write a quick note, and then just hit Send, and it becomes a digital sticky note, which I can just grab and stick right on to the fire truck so that we can have engineering revise this later.

Sven: We can also use spatial much earlier in our design process to ideate and generate inspiration. Okay, guys, let's come up with some ideas for a line of aquatic toys.

Anand: Yeah, how about sea turtles.

Sven: Oh, that's really cool. Let's try sharks.

Jinha: That's cool how about jellyfish?

Anand: So all we have to do is say the words and they're instantly visualized right before our eyes. You can even click into one of these bundles and they expand it to a full-blown internet search, complete with 3d models, images, and webpages. Jinha, why don't we click into just the images here so we can get some good inspiration for this new aquatic line.

Jinha: Mm-hmm, sure. Every object you see in spatial is physical and tactile. So you can scroll through or pick up images you like and toss them up on the wall with physics.

Anand: And we don't have to just stick with digital content, I can actually pull up these sketches I did on my phone last night using that same spatial app. I just pull up those photos and hit Send and they're instantly transformed into this digital environment.

Jinha: Nice drawings. It's so easy to fill up your room with ideas, so we built this tool to help you quickly organize all these ideas. So let me select all of these and let's make a grid. This goes to the wall.

Anand: Now this entire room we've created is a persistent digital object that anyone can access or contribute to at any time, whether they're using an AR or VR headset or even a PC or mobile phone.

Sven: So that's right, now we can have virtually rich visual rooms that we can keep up for the life of the product. That means no more tearing down war rooms all the time. So Spatial and HoloLens are helping us drive improvements in our digital design and development process changing the way we create new toys. By bringing people together from all over the world to collaborate in the same virtual room, we're overcoming a natural barrier to our collective success, that's people's desire for direct face-to-face interaction when building commitment and trust. We're so excited to see faster time-to-market and reduced need to travel, as well as the many other benefits that we're gonna unleash at Mattel, as we collaborate in mixed reality.

Sound of a box opening
Background Conversations

Valve Headquarters, WA

That's very green.

Come on in. So we're going to show you some virtual reality today. You know it's really hard to show people what it's like to be in virtual reality without having them try it for themselves. Filming you in the green screen studio is just the best way we found to help everyone else understand what it's like to be in VR. Any questions?

Can I go first?

Laughing.

All right, go crazy!

Hi! Awe, you're so cute! Oh. Laughing. Oh, he's doing a little trick. Oh. Hey, I have a stick now. Come here. You want the stick? Go get it! Laughing. He actually gets it. Come on. Oh, you're so cute.

It makes you feel like you're pulling a string back.

Left, check left.

Ooooh. Oh yeah. Nice shot.

No way! Laughing.

Hh my goodness, these guys are so cool.

Look behind you Kyra.

Screaming. Laughing.

Order up.

Ohhh. Bacon.

Oh no.

Computer voice: Is that how you cook for your family. How human.

It's burned.

Eat those doughnuts, get 'em. Get those donuts.

I don't want to stand in the way of his ball.

A little bit more toward us. There you go.

Laughing. Yay!

No, the other way, the other way. The wheels are a little off-kilter.

Hold on, hold on, I'm gonna get really hunkered down here. A little bit of a torque issue.

Yeah. There you go.

Right into the black hole. Yay! Laughing. Clapping.

What?

Kelly, go into space?

No, I can't, the grids here.

The grid wall here has to stop you from walking into the couch, walking into the walls.

Oh, this is so cool. I can move all this stuff.

Oh, oh, oh.

Look at this thing.

Laughing.

You've gotta keep with the beat.

Wha. Wha.

Ohhh. That was good.

Can you see me?

Yes. Hold up some fingers, let me see if...five!

Oh no! Laughing.

Oh my God.

Use both of your guns Kelly.

Oh, we have a little fire. It even feels hot, I think. I'm pretty sure I feel it.