Moving on to interacting with your apps on the Hololens: The main interactions you will be needing a lot are Gaze and Tap. This will make your Holo world a lot more fun!
Gazing is when you have a cursor that follows user movement while user is wearing the Hololens. More technical explanation here: https://developer.microsoft.com/en-us/windows/mixed-reality/gaze_targeting
Tapping is the Hololens version of selecting your object in the field of view.
Make sure you do the tapping, inside the field of view of the Hololens. It cannot detect taps outside the field of view.
This week, I received an article request to play different movies on the same MovieScreen (go to post here) based on different buttons selected. So I will show in this example how to gaze at different buttons and tap on them to play different videos. This way we also apply all that we have learnt until now in my series of blog posts.
(if you want to learn the video part, go to my previous post here)
First, do the basics:
1. Create a new Unity project
2. Import the MR Toolkit (We need it extensively for this example)
3. Save your scene
Delete the Main Camera. We don’t need it. Instead, we will use the MixedRealityCameraParent prefab. Drag and drop this from your HoloToolkit folder into the Hierarchy
Change the Camera settings to Hololens preferred:
1. Skybox to Solid Color
2. Background to 0,0,0,0
Now for the Gaze part:
Let’s add 2 buttons to the scene. Right click on the Hierarchy and add UI -> Button (You will notice that this will add a Canvas and an EventSystem). Make sure that both buttons are child objects of the same Canvas.
I called the buttons Video1 and Video2.
You need to change the Text value on the Buttons to Video1 and Video2 for it to be displayed accordingly on the screen. Just expand the Button to see the Text and change it there
The camera needs to see the buttons in the world space. So we will change the canvas settings:
Render Mode to World Space
Drag and drop the MixedRealityCamera to the Event Camera
Now we arrange the buttons on the Canvas in such a way that both of them can be seen by the camera. We need to make it small enough that even the MovieScreen can be viewed together
Setup the MovieScreen with a new Plane GameObject. Rename it MovieScreen. Drag and drop the MovieScreen as a child of the already existing Canvas object (I won’t cover the details of the MovieScreen in this tutorial. Please refer to my last post here)
You can match the values of the Canvas elements below for best view on the Hololens. Of course the best way for you to learn would be to experiment by tweaking the elements yourself in Unity along with the MixedRealityCameraPrefab to see what’s the best possible fit.
Values for the Canvas
Values for the Button Video1
Values for the Button Video2
Values for the MovieScreen
Values for the MixedRealityCameraParent
Now we need to add a cursor so that it can capture our gaze, or where we are looking at
Search for the DefaultCursor in the HoloToolkit. Drag and drop into the Hierarchy
HoloToolkit has another powerful prefab called the InputManager which also is responsible for many other input interactions along with the Gaze. So our next step is to add this onto the Hierarchy
Search for the InputManager in the HoloToolkit and drag and drop onto the Hierarchy
When you scroll down on the InputManager, you will see the Simple Single Pointer Selector script which requires an input for the Cursor. This needs to take in our DefaultCursor which we added previously. So drag and drop this onto the Cursor field
UPDATE: The FocusManager script on the InputManager asks for a UI Raycast camera. Make sure this is also set to the MRcamera used in the scene. Else gazing won’t work since it would be using the wrong camera.
This script and the Gaze Manager take care of the Gaze logic, so that we don’t have to worry about it
Now we are at a stage where we can check the status of how we have done so far. Since we have done just UI changes, we can quickly check this via Holographic Emulation – a feature that Unity provides already in it’s interface. This eliminates the need for us to build and deploy the app (which annoyingly takes time)
Please note: Holographic Emulation is only applicable when you want to test UI changes and NOT when you have changed something in your scripts. More information here: https://docs.unity3d.com/Manual/windowsholographic-emulation.html
Of course the prerequisite is that both your PC and the Hololens are connected to the same WiFi network
To start Holographic Emulation, go to your project in Unity, click on Window in the top menu, Holographic Emulation in the list and then a small popup opens up
Change the Emulation Mode to Remote to Device
Remote Machine IP Address to your Hololens IP Address – For this, on your Hololens, run the Hololens Remoting app. This will display it’s IP Address. Enter that into the Remote Machine field on the popup in Unity.
Wear your Hololens (no USB connection needed to your PC)
Then click Connect and press Play on your scene in Unity. You can see the scene simulate on the Hololens.
At this stage, we see that the cursor moves around but gets blended with the buttons. Let’s solve this problem by changing the Highlighted color of the Button to another color. I chose red.
For this click on the button Video1 and change the Highlighted Color to red.
Do the same for Video2
Now try on the Holographic Emulation again. You should see that whenever the cursor interacts with the buttons, the color of the buttons change to red!
We have achieved the gaze part of this tutorial. 🙂
Now for the Tap part:
So the HoloToolKit also comes with HoloToolKitExamples which I’d strongly suggest you also download so that you can look at the Example scenes for different features. One such example was the InputTapTest scene which shows how the Gaze and Tap should work.
I have used their TapResponder script and modified it to enable Taps in this example
Let’s have a look at the objective again:
1. We want to gaze at the buttons
2. We want to tap at the buttons
3. The corresponding tap on each button should play different videos on the same MovieScreen
Since we want to tap on the buttons, click on the Video1 button to Add Component, and add a new script and name it TapToPlay or something similar
Before we edit this, below is the algorithm of how I have approached it, you’ll understand the code better this way.
1. When Tap is recognized, check which button has been tapped on
2. Based on the button tapped on, call a Loader script, which loads the corresponding Movie Texture to the MovieScreen (For example: clicking on Video1, should load Video1, clicking on Video2, should load Video2 and so on)
3. After we have loaded the right MovieTexture, play the corresponding movie on the MovieScreen
Click on the MovieScreen to Add Component and add a new script to play movie. Call this PlayMovie or something similar. This would be similar to what we did in the Play movies tutorial
Click on the MovieScreen to Add Component and add a new script. Call it VideoLoader or something similar
Copy paste the above code in your VideoLoader script
Now add this code into the PlayMovie script
And finally the above code in the TapToPlay script
(You can add your own logic for what needs to happen after tap occurs under OnInputClicked)
Now get two movies and store them in your Assets folder.
I took the same two that I used for the Play Movie tutorial. Here are the links:
Convert them into MovieTextures format. Click on them and on the Inspector, choose MovieTexture. Then click on Apply. The conversion will take a few seconds
Do the same for the other video
Now for the part where we tell the MovieScreen that more than one video is attached to it. On the MovieScreen’s Inspector, for the Video Loader script:
1. Change Size of Video Textures to 2 since we have 2 videos. This corresponds to the videoTextures array field in the VideoLoader script
2. Drag and drop the two MovieTextures that you generated in the last step onto Element 0, Element1
3. Drag and drop the MovieScreen GameObject onto the Movie Player field. This is to tell the movie should play on this object. This corresponds to the moviePlayer field on the VideoLoader script
4. For the Play Movie script, drag and drop any MovieTexture onto the MovieToPlay field. This corresponds to the movieToPlay in the PlayMovie script
The reason why we do this is there would be null pointer exceptions otherwise since the object is not associated with any texture to play. (The code is written this way. If you find a much optimized way, let me know too 🙂)
Do not forget to attach an Audio Source to the MovieScreen at this point. Otherwise it will throw a null pointer exception. For this, click on MovieScreen -> Add Component -> search for Audio Source and add it
Now we can test this out. Just click Play on the Unity scene and confirm that the video indeed plays.
Now we need to add the Tap logic, the final step!
Uncheck the two scripts from the MovieScreen GameObject: Since we want to initiate the playing from the TapToPlay script. Do not remove the scripts from the MovieScreen. The code is written in such a way that TapToPlay needs to identify which objects need to be used to play and where to find the MovieTextures from. Therefore, we attach it to the MovieScreen, but we uncheck it so that it doesn’t start automatically when the app is deployed.
Make sure the script TapToPlay is attached to both the buttons Video1 and Video2
The inspector on the buttons will show two values to be filled for the TapToPlay script. Drag and drop the MovieScreen object to both fields since it is looking for the VideoLoader and PlayMovie objects which are attached already to the MovieScreen (remember we have attached it but unchecked it- it is for this purpose)
We need to include the logic to reload the texture so that we can restart the movie from beginning whenever the buttons are clicked. This has been included in the PlayMovie script
I have also included in the TapToPlay script the logic for the buttons to change colors when they are tapped
That’s it! Build and try it out on the Emulator. You should see this:
On the Emulator it is a little unpleasant to try out Gaze and Tap, but it serves the purpose. I promise you, on the device it’s much nicer
So I have added 2 buttons, but you can expand this with your own logic..
UPDATE: if you are using an object which doesn’t respond to tapping, then check if you have a collider on it. Else tapping won’t work.
P.S: Thanks for the article request Marc. Hope this helps you.
Errors and solutions:
Error: Null Pointer Exception at PlayMovie.LoadAndPlay() .. Exception thrown whilst getting audio clip Object reference not set to an instance of an object.
Solution: There is a missing Audio Source component to the MovieScreen. Add it.
Error: Gazing doesn’t work
Solution: The FocusManager script on the InputManager asks for a UI Raycast camera. Make sure this is also set to the MRcamera used in the scene. Else gazing won’t work since it would be using the wrong camera.
Error: Tapping doesn’t work
Solution: Check if collider is present for tapping to work