Using Gaze and Tap to select the objects of your choice

   January 26, 2018  48 Comments on Using Gaze and Tap to select the objects of your choice  Posted in Gesture Input, Video

Moving on to interacting with your apps on the Hololens: The main interactions you will be needing a lot are Gaze and Tap. This will make your Holo world a lot more fun!

Gazing is when you have a cursor that follows user movement while user is wearing the Hololens. More technical explanation here: https://developer.microsoft.com/en-us/windows/mixed-reality/gaze_targeting

Tapping is the Hololens version of selecting your object in the field of view.

Make sure you do the tapping, inside the field of view of the Hololens. It cannot detect taps outside the field of view.

This week, I received an article request  to play different movies on the same MovieScreen (go to post here)  based on different buttons selected. So I will show in this example how to gaze at different buttons and tap on them to play different videos. This way we also apply all that we have learnt until now in my series of blog posts.

(if you want to learn the video part, go to my previous post here)

First, do the basics:

1. Create a new Unity project

2. Import the MR Toolkit (We need it extensively for this example)

3. Save your scene

Delete the Main Camera. We don’t need it. Instead, we will use the MixedRealityCameraParent prefab. Drag and drop this from your HoloToolkit folder into the Hierarchy

Change the Camera settings to Hololens preferred:

1. Skybox to Solid Color

2. Background to 0,0,0,0

Now for the Gaze part:

Let’s add 2 buttons to the scene. Right click on the Hierarchy and add UI -> Button (You will notice that this will add a Canvas and an EventSystem). Make sure that both buttons are child objects of the same Canvas.

I called the buttons Video1 and Video2.

You need to change the Text value on the Buttons to Video1 and Video2 for it to be displayed accordingly on the screen. Just expand the Button to see the Text and change it there

The camera needs to see the buttons in the world space. So we will change the canvas settings:

Render Mode to World Space

Drag and drop the MixedRealityCamera to the Event Camera

Now we arrange the buttons on the Canvas in such a way that both of them can be seen by the camera. We need to make it small enough that even the MovieScreen can be viewed together

Setup the MovieScreen with a new Plane GameObject. Rename it MovieScreen. Drag and drop the MovieScreen  as a child of the already existing Canvas object (I won’t cover the details of the MovieScreen in this tutorial. Please refer to my last post here)

You can match the values of the Canvas elements below for best view on the Hololens. Of course the best way for you to learn would be to experiment by tweaking the elements yourself in Unity along with the MixedRealityCameraPrefab to see what’s the best possible fit.

Values for the Canvas

Values for the Button Video1

Values for the Button Video2

Values for the MovieScreen

Values for the MixedRealityCameraParent

Now we need to add a cursor so that it can capture our gaze, or where we are looking at

Search for the DefaultCursor in the HoloToolkit. Drag and drop into the Hierarchy

HoloToolkit has another powerful prefab called the InputManager which also is responsible for many other input interactions along with the Gaze. So our next step is to add this onto the Hierarchy

Search for the InputManager in the HoloToolkit and drag and drop onto the Hierarchy

When you scroll down on the InputManager, you will see the Simple Single Pointer Selector script which requires an input for the Cursor. This needs to take in our DefaultCursor which we added previously. So drag and drop this onto the Cursor field

UPDATE: The FocusManager script on the InputManager asks for a UI Raycast camera. Make sure this is also set to the MRcamera used in the scene. Else gazing won’t work since it would be using the wrong camera.

This script and the Gaze Manager take care of the Gaze logic, so that we don’t have to worry about it

Now we are at a stage where we can check the status of how we have done so far. Since we have done just UI changes, we can quickly check this via Holographic Emulation – a feature that Unity provides already in it’s interface. This eliminates the need for us to build and deploy the app (which annoyingly takes time)

Please note: Holographic Emulation is only applicable when you want to test UI changes and NOT when you have changed something in your scripts. More information here: https://docs.unity3d.com/Manual/windowsholographic-emulation.html

Of course the prerequisite is that both your PC and the Hololens are connected to the same WiFi network

To start Holographic Emulation, go to your project in Unity, click on Window in the top menu, Holographic Emulation in the list and then a small popup opens up

Change the Emulation Mode to Remote to Device

Remote Machine IP Address to your Hololens IP Address – For this, on your Hololens, run the Hololens Remoting app. This will display it’s IP Address. Enter that into the Remote Machine field on the popup in Unity.

Wear your Hololens (no USB connection needed to your PC)

Then click Connect and press Play on your scene in Unity. You can see the scene simulate on the Hololens.

At this stage, we see that the cursor moves around but gets blended with the buttons. Let’s solve this problem by changing the Highlighted  color of the Button to another color. I chose red.

For this click on the button Video1 and change the Highlighted Color to red.

Do the same for Video2

Now try on the Holographic Emulation again. You should see that whenever the cursor interacts with the buttons, the color of the buttons change to red!

We have achieved the gaze part of this tutorial. 🙂

Now for the Tap part:

So the HoloToolKit also comes with HoloToolKitExamples which I’d strongly suggest you also download so that you can look at the Example scenes for different features. One such example was the InputTapTest scene which shows how the Gaze and Tap should work.

I have used their TapResponder script and modified it to enable Taps in this example

Let’s have a look at the objective again:

1. We want to gaze at the buttons

2. We want to tap at the buttons

3. The corresponding tap on each button should play different videos on the same MovieScreen

Since we want to tap on the buttons, click on the Video1 button to Add Component, and add a new script and name it TapToPlay or something similar

Before we edit this, below is the algorithm of how I have approached it, you’ll understand the code better this way.

1. When Tap is recognized, check which button has been tapped on

2. Based on the button tapped on, call a Loader script, which loads the corresponding Movie Texture to the MovieScreen (For example: clicking on Video1, should load Video1, clicking on Video2, should load Video2 and so on)

3. After we have loaded the right MovieTexture, play the corresponding movie on the MovieScreen

Click on the MovieScreen to Add Component and add a new script to play movie. Call this PlayMovie or something similar. This would be similar to what we did in the Play movies tutorial

Click on the MovieScreen to Add Component and add a new script. Call it VideoLoader or something similar

Copy paste the above code in your VideoLoader script

Now add this code into the PlayMovie script

And finally the above code in the TapToPlay script

(You can add your own logic for what needs to happen after tap occurs under OnInputClicked)

Now get two movies and store them in your Assets folder.

I took the same two that I used for the Play Movie tutorial. Here are the links:

http://www.sample-videos.com/index.php#sample-mp4-video

http://techslides.com/demos/sample-videos/small.mp4

Convert them into MovieTextures format. Click on them and on the Inspector, choose MovieTexture. Then click on Apply. The conversion will take a few seconds

Do the same for the other video

Now for the part where we tell the MovieScreen that more than one video is attached to it. On the MovieScreen’s Inspector, for the Video Loader script:

1. Change Size of Video Textures to 2 since we have 2 videos. This corresponds to the videoTextures array field in the VideoLoader script

2. Drag and drop the two MovieTextures that you generated in the last step onto Element 0, Element1

3. Drag and drop the MovieScreen GameObject onto the Movie Player field. This is to tell the movie should play on this object. This corresponds to the moviePlayer field on the VideoLoader script

4. For the Play Movie script, drag and drop any MovieTexture onto the MovieToPlay field. This corresponds to the movieToPlay in the PlayMovie script

The reason why we do this is there would be null pointer exceptions otherwise since the object is not associated with any texture to play. (The code is written this way. If you find a much optimized way, let me know too 🙂)

Do not forget to attach an Audio Source to the MovieScreen at this point. Otherwise it will throw a null pointer exception. For this, click on MovieScreen -> Add Component -> search for Audio Source and add it

Now we can test this out. Just click Play on the Unity scene and confirm that the video indeed plays.

Now we need to add the Tap logic, the final step!

Uncheck the two scripts from the MovieScreen GameObject: Since we want to initiate the playing from the TapToPlay script. Do not remove the scripts from the MovieScreen. The code is written in such a way that TapToPlay needs to identify which objects need to be used to play and where to find the MovieTextures from. Therefore, we attach it to the MovieScreen, but we uncheck it so that it doesn’t start automatically when the app is deployed.

Make sure the script TapToPlay is attached to both the buttons Video1 and Video2

The inspector on the buttons will show two values to be filled for the TapToPlay script. Drag and drop the MovieScreen object to both fields since it is looking for the VideoLoader and PlayMovie objects which are attached already to the MovieScreen (remember we have attached it but unchecked it- it is for this purpose)

We need to include the logic to reload the texture so that we can restart the movie from beginning whenever the buttons are clicked. This has been included in the PlayMovie script

I have also included in the TapToPlay script the logic for the buttons to change colors when they are tapped

That’s it! Build and try it out on the Emulator. You should see this:

Expected Gaze and Tap Output Video

On the Emulator it is a little unpleasant to try out Gaze and Tap, but it serves the purpose. I promise you, on the device it’s much nicer

So I have added 2 buttons, but you can expand this with your own logic..

UPDATE: if you are using an object which doesn’t respond to tapping, then check if you have a collider on it. Else tapping won’t work.

P.S: Thanks for the article request Marc. Hope this helps you.

Errors and solutions:

Error: Null Pointer Exception at PlayMovie.LoadAndPlay() .. Exception thrown whilst getting audio clip Object reference not set to an instance of an object.

Solution: There is a missing Audio Source component to the MovieScreen. Add it.

Error: Gazing doesn’t work

Solution: The FocusManager script on the InputManager asks for a UI Raycast camera. Make sure this is also set to the MRcamera used in the scene. Else gazing won’t work since it would be using the wrong camera.

Error: Tapping doesn’t work

Solution: Check if collider is present for tapping to work

48 thoughts on “Using Gaze and Tap to select the objects of your choice”

  1. Great article for the understanding and usage of the Gaze and Airtap in combination with UI-Elements!
    It worked really good and helped a lot!
    I’m looking forward to the upcoming articles!

    All the best!

    Marc

    Reply

  3. Hello admin, i must say you have very interesting
    posts here. Your website should go viral. You need initial traffic only.
    How to get it? Search for; make your content go viral Wrastain’s tools

    Reply

  4. Great tutorials, but Unity’s console is giving me a warning: “You are trying to create a MonoBehaviour using the ‘new’ keyword. This is not allowed. MonoBehaviours can only be added using AddComponent(). Alternatively, your script can inherit from ScriptableObject or no base class at all”

    Reply

    1. Hey Rupe, Gerard, Here are some possible solutions that you could try:
      1. Since this is just a Unity warning, did you try compiling and building the code with the warning? The app should deploy. I usually ignore this warning
      2. Alternatively, I also remove the inheritance (i.e the :MonoBehaviour) and try compiling. This also works. So the class TapToPlay or any others that I have used could also be standalone
      3. A third solution that I have observed (it works only sometimes, though) is : create the scripts brand new in the Assets folder of the Unity project and type in the code which I have pasted into it. You can type in the first few lines until MonoBehaviour and after that copy-paste the code into it. This is very strange, I know, but it works, sometimes

      Reply

  5. Did everything as explained, it does not work… Unity is giving me errors in the console. In the emulator, clicking the buttons does nothing.

    Reply

      1. Hey, Please try the solution in my reply to Rupe above. Still doesn’t work? then reply with a screenshot of your Unity error. In my opinion, this should be a Unity warning and not an error- so build must go through if it’s just a warning.

        Reply

  6. Hi! Thank you for your tutorial!
    Now I have a problem: My cursor didn’t directly sense the button, but sense buttons a distance from a distance. I don’t know why?
    I hope you will answer me! 🙂 Thanks!

    Reply

    1. Hi Jiawei, Could you send me a video or explain in detail what you mean by distance from a distance? The cursor should ideally hover the Buttons upon contact and should change colors. Maybe it is interacting with the Canvas over the buttons… ? The canvas and buttons need to be resized according to the specs I have provided in this tutorial. If you give me more details (the more the better), then I can be of more help 🙂

      Reply

  7. Pretty section of content. I just stumbled upon your website and in accession capital to assert that I get in fact enjoyed
    account your blog posts. Anyway I will be
    subscribing to your augment and even I achievement you access consistently
    fast.

    Reply

  9. Great site you’ve got here.. It’s difficult to find high-quality writing like yours these days.

    I seriously appreciate individuals like you! Take care!!

    Reply

  11. Hello, Nischita.
    Really good tutorials in your website.

    I’m trying to do this one, for other purpose, but the button don’t respond to the tap, i don’t know what i’m doing wrong.

    Keep the good work in this website, it’s really helpful.

    Best regards, Rafael.

    Reply

    1. Hi Rafael, Thank you for the feedback! 🙂 Happy that the tutorials help! If the button is not responding to tap, I’m sure something has been missed…
      – is the DefaultCursor in place? Can you see the cursor move at all?
      – If you bring your hand in the frame, then you should see the cursor turn into a larger rounder circle (even before the tap), is this happening?
      – is the InputManger in place? and is the SimpleSinglePointerSelector having the Cursor referenced?
      – if not, there may be something wrong in the TapLogic script on your side?
      My suggestion is to set up a debugger while deploying and see what’s happening there.. you can always write back with more details, and I will certainly help you out.
      Cheers, Nischita

      Reply

  12. Hello, Nischita.
    I found that i had something in front of the cursor and then it work fine.
    Thanks for the help.
    Best regards, Rafael

    Reply

  14. It’s difficult to find knowledgeable people on this subject, however, you sound like you know
    what you’re talking about! Thanks

    Reply

  16. Hi Shilpa, There are many reasons it could fail.
    1. Do you have a collider on the buttons? The tap needs the cursor to interact with the colliders
    2. Do you have a Cursor in the scene and do you have an InputManager which references the Cursor?
    3. Is the button configured correctly? Share a screenshot and I can help you more 🙂

    Reply

  20. Hi Nischita, thanks for the tutorial.
    My app works fine in Unity when I hit the play button. All the buttons and actions work.
    Once I start the app on the HoloLens, it recognises my gaze and when I air tap, the gaze cursor also acts but I am not able to select the buttons.
    I would really appreciate your help.
    Thanks 🙂

    Reply

  21. Hi Nischita, thanks for the tutorial.
    But I can not drop the two MovieTextures that I have generated in the last step onto Element 0, Element1

    Reply

  22. Hi Nischita, thanks for the tutorial.
    But I can not drop the two MovieTextures that I have generated in the last step onto Element 0, Element1

    Reply

  23. Hi,Nischita.The last problem I have solved is that I did not convert the video format from mp4 to ogv, but now I have a new problem need your help.
    No matter when I click the video1 button or video2 button, It play the default video file and I can’t switch another video. What should I do?

    Reply

  24. Hi ZXXh, Welcome! For the always playing default video file, check if the PlayMovie script is scripted fine. Sometimes, you may be checking for the same video file. Check for the names of the buttons tapped on and make sure you disable playing the other video in context.

    Reply

  25. Hi Nischita!
    Thank you for this wonderful tutorial. I am also stuck in the first level, like many others. I have added a box colider component to my buttons. However, they do not re act to being clicked on. My cursor turns into a ring when it detects my hand gesture. But clicking or hovering does not change the buttons.
    Do you have any suggestions?

    Reply

  26. Hi Sara, thanks for sending the project and because of that I came to know of two things which are updated in the article:
    – The gazing won’t work if the UI camera on the FocusManager Inspector field attached to the InputManager is pointing to the wrong camera as used by the Canvas. In our example we use MRCamera for Canvas. So we need to set the Inspector value of the FocusManager script on the InputManager to MRCamera as well. Thanks Sara!

    Reply

  28. Is there such a thing as GazeManager.Instance.HitInfo.collider for Windows Mixed Reality Opaque headsets when you use a controller instead of gaze? How do I check if my controller is pointing/raycasting an object instead of my gaze (which is also present in Windows Mixed Reality too but useless if you use controllers to interact instead)? Thanks

    Reply

  31. Hi Nischita, sorry for asking something again. But for some reason i cant put the movie textures into element 0 and element 1.

    Reply

  34. Hi Nischita! I got the problem fixed but for some reason when i deploy the app to my hololens i just see nothing … Why?

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *