GEOG 497
3D Modeling and Virtual Reality

Interaction in VR


Interaction in VR

Interactions with GameObjects in non-VR environments are limited to conventional modalities and their affordances. In desktop experiences, for instance, interactions are limited to pointing at objects and graphical user interface element. In certain gaming consoles, however, such as Nintendo Wii and Xbox Kinect, users can perform natural gestures to interact with objects in a game or to perform actions.

Interaction with GameObjects in VR can be considerably more natural compared to desktop experiences. The immersive nature of VR HDMs combined with the possibility of locomotion in a natural point of view affords a much closer interaction experience to real-life compared to any other gaming console or desktop. Thanks to the power of sensors and controller data in VR headsets, we can constantly track the position, orientation, and intensity of hand movements in VR. As such, users can use natural gestures for interaction with different types of objects while perceiving them from a natural viewpoint.

Users can interact with objects by reaching out and grabbing them when they are in their proximity, or they can grab them from distance using a pointer. Once an object is grabbed, users can use the physics properties to place them somewhere in the virtual environment, throw them, and even change their scale and rotation.

Video: Natural Virtual Hand Interaction with Puzzle Cube in VR (1:13)  This video is not narrated.

An example of grabbing an object from distance using pointers:

Point Grab GIF
Point Grab GIF

More natural forms of interaction in VR are via gestures. For instance, users can spin a wheel using a circular gesture or pull down a lever using a pull gesture. The popular VR archery game is a prime example of using natural gestures to interact with game objects.

Archery GIF
Credit: Node via Giphy

Similar to the bow and arrow, some objects such as a fire extinguisher are used by two hands in real-life. The same principle could be applied in VR, where the user grabs the capsule with one hand and the hose with another. An overview of some of these interaction mechanics is demonstrated in the videos below from one of the most popular VR plugins for Unity called VRTK.

Video: [Basics] Grab attach mechanics (5:57)

Click here for a transcript of the Grab attach mechanics video.


Hello and welcome to another VRTK tutorial. In today's tutorial, we're going to be covering the different grab attach mechanics. So to do that, I'm just going to jump into scene 21, which is 0:21 control out grabbing object with joints. And this scene already shows us the different grab attach mechanics that are on offer. And the grab attach mechanics can be seen in script, interactions, grab attach mechanics. And these are the different types. And if we click on one of these, you'll see, for instance, on here we've got the spring joint grab attach. We'll see that in action in a moment. On the red cube, we've got the fixed joint grab attach. And on the green cube, we've got the track object grab attach. If we move over to this wheel, if we look at the wheel on here, we've got the rotator track grab attach. And if we look at the gun down here, this one has got the child of controller grab attach. So the main difference between the grab attaches is they all slightly attach the object to the controller in different ways or by doing different things. So the fixed joint and the spring joint both use a joint to fix it to the controller. And this can be very useful, for instance, if you pick up a fixed joint object it holds very close to the controller, but if you bang it against something, the joint will break and it will automatically drop out your hand, which can be quite useful if you're going for that level of realism that if you bang your hand against the wall, you may drop the gun or something like that. Obviously a spring joint can give you the feeling that something is not quite being tracked directly with you, so it's always kind of springing away. And you can change all the springs settings within Unity to set that up however you want. And then the track object, it actually uses the velocities of the controller. So to move that object to follow the controller around and this can be very useful as well that it can then also interact with other rigid bodies in the scene, rather than just disconnect like the fixed joints or the spring joint does. We've also got the rotator track and this works by checking out the rotations of the controller and then rotating things accordingly. So it works really well for wheels and doors and the hinges of that sort. Then we've got the, on the gun, if I just select that, we've got the child of controller, which literally takes the object and sets it as a child of the controller that you're holding. And that gives you really, really good accuracy in tracking. It does have some downsides, which it doesn't interact brilliantly with other game objects. You can also pass it through other colliders. But that can be very useful as well that if you don't want the physics to go crazy, if you accidentally stick your gun into a door, into a wall, using the child of controller will solve all that for you. We can also add multiple grab attach mechanical objects as well. So if we look at this fire extinguisher here, we've got a track object on the actual body, so when we pick this track its object there. And then if we look at the hose on the sprayer, this is also got a tracked object so we can grab this with one controller and then we can grab this with the other controller. And these are just connected with a simple spring joint to give it like a chain look. And so we'll jump into the see now and we'll see how each of these different tracked objects looks as you're grabbing something.

So we're in the scene and as I said the blue cube uses the spring joints. So if we pick that up you can see it kind of dangles underneath and it's got a spring affect to it from the controller. And we can swap it between the two and it has kind of this pendulum style effect. We can drop that down. The red cube uses a fixed joint so that's very good tracking to the controller, it holds it quite nicely. And also if you bang it into things, you don't have to release the grab, it will automatically release. So if you was to walk into there and bang it, it just drops onto the floor. The track object uses the velocities to make sure the rigid bodies track accordingly. It's a little bit more wobbly than a fixed joint, but it's very good for things like this. If I drag it into here, it won't actually disconnect from me and we can kind of get it to move around objects with us. This is very useful if you're dealing with rigid bodies but it can also give this untoward effect of things disconnecting from the controller. We've also got the rotator track which is extremely useful for hinge joints like on this wheel. So as we grab this we can move this and it kind of tracks where the controller is to make it look like we're spinning a wheel or moving a vowel around. And then on the gun over here we've got the child of controller. So if we pick this up you can see it doesn't actually interfere with these colliders because I don't have a rigid body on these ones. With a rigid body will still move them around, but we could put this into a wall or a floor and it wouldn't actually affect anything. And also on here we've got other nested interactions, so these have got grabs on the side, so we should be able to put our controller down there and pull this back and have a grab on there. And, as I said, the same for this fire extinguisher. If we pick this up, we can also reach out and grab this other partner and then when we do something with this controller, this other controller is doing the action. So that's the simple explanation into the different grab types. There's not one that fits all scenarios, so it's always good to pick and choose which one that you feel works best for you in your scenario. I hope that's been helpful. If it has, please leave some comments, leave some likes and think about subscribing to the channel. Thanks for watching and bye for now.


Video: Unity VRTK v3 ( VR Development ) Testing Interactables Demo Scene (10:34)  This video is not narrated.

Credit: null ref

In the videos posted above, you can see that users can naturally interact with different types of objects, by sitting down and pulling out a drawer, flipping switches, etc. The same principle can be applied to graphical user interface (GUI) objects. Users can use natural gestures to collide with different GUI objects such as buttons to interact with them, or they can use pointers to point at a specific GUI element. Users can grab a selector on a slider and move their hand to the left or right to decrease/increase a value, or they can move a text element in the scene.

As the inspiration of different interaction mechanics for your assignment, you can watch the following video:

Video: Top 10 PC VR Games

Credit: Ben Plays VR
Click here for a transcript of the Top 10 PC VR Games video.

Believe it or not, we're halfway through 2019. So here's a long-overdue update to my top ten PC VR games, the best as of June 2019.

VTOL is a detailed and complex flight simulator and not a casual arcadie flight game. For example, to take off from the ground, it takes all of these steps. So it feels very realistic in that regard and you'll definitely need to take your time and the training missions to get comfortable with everything. There's runway and vertical takeoff and landing. Some missions only allow a specific takeoff or landing. For example, mission 2 requires a vertical landing on a rooftop. And speaking of missions, the current state of the game has a handful of story campaign missions for the VTOL aircraft, as well as a free-flying mode. If it's a combat mission, then you need to choose your loadout for the aircraft, all within your allotted budget for that mission. [Music] Everything is done with the motion controllers, which I think is awesome. As such, you can customize the height of your chair and the main controls on either side of you so you can comfortably reach everything. On top of that, you can choose the optional fighter jet aircraft which gives you a whole new way to play and another set of missions. And thanks to the level editor, there's a bunch of community-made content like additional campaigns and custom missions to play. This game is constantly being updated. It's an early access game that always stays fresh with new updates. So even though it's already a great flight simulator, it's exciting to know that even more is on the way. In the gameplay footage here you can see my dad playing because he loves realistic flight simulators. After he got comfortable with all the training, he was in heaven and completely fell in love with this game. So if you enjoy realistic flight simulators as well then I highly recommend it. The price is $30.

Pixel Ripped 1989 begins with you living inside a Game-Boy-like world, playing a game on a TV. But after disaster strikes you'll then switch characters to become a young student playing on their game system, controlling the character that you previously were. And it's in this state that you'll spend the vast majority of the game. You need to play your portable game system while juggling responsibilities in the real world, like keeping your teacher distracted so you can keep playing the game without getting in trouble. The retro game that you're playing is a simple platformer, but needing to keep the teacher distracted, means you have to keep switching your attention constantly between the retro game and the classroom. [Music] Later things become more mixed and intense as the video game world invades the real world, like here, where you need to stop the bad guys from abducting your friends. Or here, where you play a vertical level attempting to hit TNT to attack the school headmaster. The default control scheme is with motion controllers, so you feel like you're actually holding a game system. But you can disable the tracking for motion controllers or ditch them entirely and play with a gamepad. It took me two and a half hours to beat it and I had a great time. I bought this during the Steam winter sale and I'm glad I did. Even though it's a little short, it's incredibly unique and it satisfied all kinds of retro gaming nostalgia for me. The regular price is $25 [Music]

Star Shelter is a solo space survival simulator. Your spaceship has undergone massive damage and you need to repair the ship, salvage supplies, and avoid a multitude of dangers to stay alive. And since you're in space everything is in zero gravity. You propel yourself by grabbing onto surfaces and pushing or with your oxygen thrusters. Oxygen and energy will be our two primary concerns, although there are many more things to stay on top of. You refill oxygen with blue canisters and energy with orange canisters. Inject those into your suit for supply on the go or return to your main ship for transfer to your suit. You repair your ship and salvage materials all with your magic finger. After you salvage things, you'll build up your inventory of raw materials. You'll need those raw materials to engineer new items that you'll need or even expand your ship. One of the things I enjoyed most was boarding nearby abandoned ships to see what I could salvage. Sometimes you'll need to hack a keypad to gain entry and you'll often find new dangers inside, like gun turrets or radiation. For long-term survival, you'll need to grow plants to generate oxygen and food. All in all, I think it's an impressive and addicting space survival game. And if things get too intense, you can play in creative mode, which has no dangers, and you can build anything you want. You'll get at least five hours of play to beat the game and the environments are randomly generated whenever you start a new game, so there's lots of replayability. For the regular price of $15, you get a lot of bang for your buck.

A voice from the game: A space ship has been damaged, oxygen is leaking.

I've very recently reviewed Final Assault, so a lot of this footage will probably be familiar to you. But be that as it may, it still deserves to be in the top ten because it's my new favorite strategy game in VR. It's a 1 vs 1 realtime strategy game. There's no building of structures. Each player is granted one command center that makes all of the units. And the first player to have their defense towers breached and command center destroyed loses the game. But even though there's no structures to be built, there's still lots of managing and strategy to be found in the combat itself. One of the first things I noticed about the game, is that you actually do command individual units, which I found incredibly rare in VR strategy games. Your command center steadily deploys a bare minimum of troops that march toward the enemy base, but you obviously need more than that. So, on one hand, you have a menu of units at your disposal with the simple units on the bottom and the advanced two units on the top. And to get access to the advanced units, you need to unlock those tiers with some cash. Grab a unit and then decide where in the battlefield they should start out. You can then order the unit to move around or attack a specific enemy. And the combat mechanics feature the usual rock-paper-scissors dynamic. Planes are best against infantry, rockets are best against tanks, etc. One of the most fun elements are the airplanes. It reminded me of the old Final Approach game where you can trace out exactly where they should fly. And if you trace a path on the ground that's where the planes will strafe attack. There's multiple generals to choose from, each with unique units and abilities. There is both single-player campaigns and a multiplayer. Like I said before, it's my new favorite strategy game. It's incredibly well put together with a fun light-hearted atmosphere and engaging strategy. The price is $30 and if you're an RTS fan then it's totally worth it [Music]

Transpose is a mind-bending game which records your actions so you can use loop two duplicates of yourself to solve puzzles. The goal is to get special cubes into their bases. But of course, the execution is much trickier than it sounds. Whenever you begin a level, all of your actions are recorded. After you're done with your actions, you then choose to discard or keep everything you've done. After choosing that, you then begin the level anew and your actions are recorded again. If you choose to keep the previous recording, you'll see it played out and all the actions you did will still occur. The previous recordings are called echoes. Early on, you're limited to just a few echos, but eventually, you'll be using up to eight echoes at once. Later in the game, gravity will become relative to each echo, and the physics of transferring cubes between different gravity directions adds a new challenge. I found the overall design of the game creative and refreshing, but especially with interfaces on your arms. On your left arm, you use a slider to fast-forward time, which makes waiting for echoes less tedious. On your right arm, each echo is represented by a ring. Highlighting a ring will show where that echo is and pulling a ring will destroy that echo. There's sliding movement and teleporting. Teleporting is required to cross gaps in almost every level. Based on my progress, I estimate eight hours of playtime. Personally, I had a blast playing this. And even though Transpose is number six in my overall top ten, it's my number one favorite puzzle game. The regular price is $20 [Music]

Multiplayer-only military shooters really aren't my cup of tea, but Zero Caliber really shines because it features a single-player story campaign. And, for me, that makes it stand out among the sea of VR shooters out there. The campaign is a series of progressively unlocked levels that tell the story of your fight against a worldwide terrorist organization. Each level has multiple goals to accomplish and sometimes your goals will change depending on how the story unfolds. Like here, when your team comes across a group of civilians, and you need to defend the civilians against an attack before you can continue on your original mission. The gun assembly, handling, and reloading are realistic. But the combat itself is arcadie action. For example, you have unlimited ammo. Whichever gun you're holding, your vest ammo will supply endless ammo for that gun. Switch to a different gun and your vest then supplies that ammo. If you get hit, you'll quickly heal up if you can avoid getting hit again. [Music] There's also some obstacle course elements, like needing to climb a tower to activate a radio signal. In the campaign, if you get killed, you can respawn from the previous checkpoint instead of starting the level all over again. At the beginning of every mission, there's a handful of weapons to grab, but after completing missions you'll earn cash which you can spend in the armory to choose something specific you'd like to bring with you to the mission. As an added bonus, you can also play the campaign levels in co-op multiplayer. When I originally played this I beat the campaign in three hours, but more levels have been added since then. The regular price is $25.

In Subnautica, you're super cool spaceship fails and crash lands on a water planet. But luckily for you, it's a very beautiful and lush water planet, so you need to get diving to find resources and do whatever you can to survive. There are different survival modes to choose from, so if you want the game to be a little easier without worrying about going hungry or thirsty, you can choose that. One of the most important things in the game is the fabricator. That's where all of the scavenging and harvesting is paid off by turning raw materials into resources, and resources into equipment, tools, and gear. Just about anything you find in the water can be used in some way, so just explore and grab whatever you can. There's virtually no tutorial at the beginning. This is a game that doesn't hold your hand and rewards you for having initiative. At first, it's a little frustrating how little air you have, you have to constantly go back up to the surface for air. But after creating air tanks and equipping fins for speed, it's a lot more enjoyable. One of the most important things to make is the scanner, which will allow you to collect data and quickly get more blueprints to create new gear after scanning wreckage. The graphics are amazing. It's beautiful and truly immersive. I really got hooked on the gameplay. Wanting to create just one more tool or piece of gear is addicting. I found it hard to stop playing. Since this is a VR port of a flat-screen game, that comes with the advantage of being a huge game with tons to do and see. I've barely scratched the surface and you can eventually build underwater craft and habitats. The downside is that there's no motion controller support, which is a bummer because that would make this game so much more immersive if you could use your hands. If you don't mind the lack of motion controller support, then this is an engaging survival experience with a huge amount of gameplay. The price is $25.

Blade and Sorcery is an interesting one because ironically, it isn't one of my personal favorites. I'll explain why in a minute. But this game has single-handedly changed the landscape of VR melee combat forever. This game is pure melee violence. It's all about arena combat, which normally does sound shallow and a little boring, but the sheer realism of the melee mechanics make it exceptional. All of the movement and contact with the enemy is legit. The blocking, swinging and striking feels true in a way I haven't really felt before. During combat, you can activate slow motion, which is really nice when you get overwhelmed with multiple enemies. It's tempting to compare this to Gorn, but I think there's two key differences, one being that you can cast magic in this game. The other key difference is the more realistic design of the enemy. In Gorn, it feels like you're killing stuffed cartoons, but here the enemy feels more realistic. Now I normally don't mind violence in video games, but this felt so realistic that I have to admit, it gave me some pause. Like wondering at what point does this crossover from a game to a simulator. I do enjoy fantasy violence, but I don't enjoy the feeling of actually killing another human being. This feels too real for me. The irony that I am including it in my top 10 isn't lost on me though. I'm including it because the combat mechanics are so good that it sets a whole new bar for VR combat moving forward. For that reason, I'm especially excited for the upcoming Boneworks game because that will also have advanced melee mechanics, but in a more pretend fantasy way. [Music]

In my opinion, Tin Hearts is the single most underrated VR game out there. It's one of the most beautiful VR games I've ever played, but it has so few reviews on Steam and I never hear anyone else covering it in the VR landscape. It's fundamentally a lemmings game, but I also find it to be a relaxing and meditative experience at the same time. Tiny toy soldiers marched forward and it's up to you to guide their path to the goal. You don't control the toy soldiers directly, you need to change other objects to steer them, starting with just simple blocks to make them turn, but eventually, you'll be manipulating other toys to steer them. And there's some creative contraptions that get involved as well. The levels start small but eventually you'll need to guide the soldiers through large toy shops. Sprinkled throughout you'll get lots of back story via reenactments and spoken letters. You can play seated or standing and for both modes, there's additional height adjustment so you can set it however you like. This game bleeds production quality. The visuals are outstanding and the animation is top-tier, even down to the way they crash on the floor and die. Which brings me to the time control. You can pause, fast-forward and rewind time, even after you've changed their paths. The ambiance is absolutely beautiful. It's a delightful place to dwell in. It's easy to take your time in this game because simply living in these toy shops is half the fun. This is the best lemmings game I've ever played. This early access version of the game I beat in three hours. The price is 20 bucks. [Music]

Windlands 1 was a Zen exploration and item hunting game, with a unique grappling hook traveling mechanic. Windlands 2 keeps the same unique grappling hook movement, but it's now a full-blown story campaign with exciting combat. In a beautiful abstract world, robots have run amok, so it's up to you to destroy them. The various tasks and missions begin by talking to an NPC. After hearing their story, you're given a quest marker so you know where to go. You'll be engaged in both combat missions and fetch quests, but even between combat missions when you're going from point A to point B, you'll have a good time, because simply traveling with the grappling hooks is enjoyable. The combat takes a while to get used to because you have to mentally juggle the grappling hook swinging and archery at the same time. But once you get used to it, it becomes second nature. The controls are completely customizable, so if you don't like the default controls you can make any button do whatever you want. There's both regular enemies and boss battles. The boss battles I found especially fun, swinging around to avoid fire while trying to aim arrows on the boss's weak spots, is super engaging. The later boss fights become massive engagements, giant ships to take down inside vast arenas. It took me about four hours to beat the whole campaign and I really enjoyed this because, Like I said, simply traveling with the grappling hooks is enjoyable and fun, especially with the sound design and music. They really nailed the audio mood of fantasy exploration and travel. [Music] Oh and I forgot to mention, you can play the campaign in single-player or co-op multiplayer. And after you beat the campaign, there are racing modes and hidden collectibles to find as well. Windlands 1 was fun, but it felt like half of a full game. Windlands 2 has nailed it. It's a must-own in your VR library. The price is $30. And if you only buy one game on this list make it this one.

Well, that's it for now. Thanks so much for watching. If you like what you see, please subscribe. See ya. [Music] you [Music]