Tag Along, Billboarding and other menu elements

In this article before the year ends, I bring to you another useful set of elements which you’ll need almost in every HoloLens application.

To be specific, I’m talking about a menu which can help you perform the following actions:

1. Menu appears as a tag along to the user’s gaze (Tag along)

2. Menu always present with it’s front facing the user (Billboarding)

3. Menu which can be pinned (remove tag along) and unpinned when it starts disturbing a user

With the MRTK,  it has become exceptionally easy to achieve objectives 1 and 2. For 3 there will be simple scripts which would be available in the article

Let’s start off with the usual things:

1. Create a new project. I have Unity version 2018.2.13f1

2. I’m using the latest stable release of the HoloToolkit 2017.4.3.0 Import it into your project

3. Do the usual tweaks:

a. Delete the main Camera.  Add the MixedRealityCameraParent.prefab in the Hierarchy. This will serve as the Main Camera

b. Change the Camera Settings to Hololens preferred:

Skybox to Solid Color

Background to 0,0,0,0

In the MixedRealityCameraManager script which is attached to the MixedRealityCamera object, change the Clear Flags value to Solid Color

Ensure that the Background Color is black

c. Go to Edit ->Project Settings -> Player -> Other Settings -> and change the Scripting Runtime version to .NET 4.x Equivalent and Scripting Backend to .Net

Make sure also that in XR Settings, Virtual Reality Supported is checked

We need both the InputManager and the DefaultCursor in the scene

4. InputManager

Drag and drop the InputManager.prefab into your scene

5. DefaultCursor

Drag and drop the DefaultCursor.prefab into your scene. In the InputManager’s Inspector Settings, drag and drop the DefaultCursor into the Cursor field of the SimpleSinglePointerSelector script

d. Save the scene

Now let’s create our menu:

I used a simple Cube and set the following scale ( 1,1,0.1) to make it look like a menu. Rename it to Menu or something similar.

Make sure the Menu is in front of the camera

For the material, we could use something which looks dark and almost transparent to achieve a good UI. I borrowed a material called UIDarkGray from the MixedRealityToolkit-Examples (btw, you can find a lot of other UX based prefabs and materials in there- it’s loaded). Just copy this to the Assets folder of your project. And then click on the circle next to the Element 0 on the Materials and choose this UIDarkGray

 Now let's create our menu: I used a simple Cube and set the following scale ( include the scale here) to make it look like a menu. Rename it to Menu or something similar. Make sure the Menu is in front of the camera For the material, we could use something which looks dark and almost transparent to achieve a good UI. I borrowed a material called UIDarkGray from the MixedRealityToolkit-Examples (btw, you can find a lot of other UX based prefabs and materials in there- it's loaded). Just copy this to the Assets folder of your project. And then click on the circle next to the Element 0 on the Materials and choose this UIDarkGray

Let’s add a heading for it:

Create a 3D Text object. Call it Title or something sensible. Make sure you move it to sit on top of the Menu like in the screenshot shown. Adjust it’s Scale, Text and other attributes according to what suits the app the best.

After you’ve adjusted, make sure you make it a child object of the Menu, such that whenever you move the Menu, it’s children move with it.

 Let's add a heading for it: Create a 3D Text object. Call it Title or something sensible. Make sure you move it to sit on top of the Menu like in the screenshot shown. Adjust it's Scale, Text and other attributes according to what suits the app the best. After you've adjusted, make sure you make it a child object of the Menu, such that whenever you move the Menu, it's children move with it.

Let’s add some Menu buttons. Typically, you will have many scenes in your HoloLens application and may want to switch between them. So usual buttons would be a Home, About, and some other scenes like Level1, Level2 etc. Let’s for the sake of our example add some dummy buttons.

Create a Cube. Rename it HomeButton or something similar. Change it’s Material to HolographicButtonPlate. Add a 3D Text element which serves as it’s Label. Rename it to Label or something sensible. Adjust it’s scale and other attributes to suit the button best. Move it as a child object of Menu.

The screenshot of my Label as a reference is below.

 Let's add some Menu buttons. Typically, you will have many scenes in your HoloLens application and may want to switch between them. So usual buttons would be a Home, About, and some other scenes like Level1, Level2 etc. Let's for the sake of our example add some dummy buttons. Create a Cube. Rename it HomeButton or something similar. Change it's Material to HolographicButtonPlate. Add a 3D Text element which serves as it's Label. Rename it to Label or something sensible. Adjust it's scale and other attributes to suit the button best. The screenshot of my Label as a reference is here. Move it as a child object of Menu.

Now we will add a TapScript to the button such that when the user taps on the button, it does something.

In this example, it will instantiate a Cube.

Click on AddComponent of the HomeButton and create and add a new script. Rename it TapScript or something similar.

Add the OnInputClicked method as indicated in the screenshot of the code. Do not forget the using derivative for the IInputClickHandler, using HoloToolkit.Unity.InputModule; 

Also add the Instantiate code. You would have seen something similar already in the other tutorial

 Now we will add a TapScript to the button such that when the user taps on the button, it does something. In this example, it will instantiate a Cube. Click on AddComponent of the HomeButton and create and add a new script. Rename it TapScript or something similar. Add the OnInputClicked method as indicated in the screenshot of the code. Do not forget the using derivative for the IInputClickHandler, using HoloToolkit.Unity.InputModule; Also add the Instantiate code. You would have seen something similar already in the other tutorial

In the Inspector of the HomeButton under TapScript Cursor drag and drop the DefaultCursor

 In the Inspector of the HomeButton under TapScript Cursor drag and drop the DefaultCursor Let's test this out so far. Build and run on the HoloLens or the Emulator. You should see a Cube instantiated when you tap on the button. (If you get the TextMeshPro error while deploying, then remove the package Unity TextMeshPro from your project and rebuild. Go to Unity-Window-Package Manager and a pop up appears.Click on TextMeshPro and Remove. While you are at it, also remove the Analytics Library, Ads and In App purchasing)

Let’s test this out so far. Build and run on the HoloLens or the Emulator. You should see a Cube instantiated when you tap on the button.

(If you get the TextMeshPro error while deploying, then remove the package Unity TextMeshPro from your project and rebuild. Go to Unity->Window->Package Manager and a pop up appears.Click on TextMeshPro and Remove. While you are at it, also remove the Analytics Library, Ads and In App purchasing)

Now let’s add the Tag Along and Billboarding logic to the menu:

This would mean that the menu moves or tags along with the gaze of the user. Look for the Simple Tagalong script from the MRTK in the Add Component of the Menu GameObject and attach it to the Menu. It adds an Interpolator. Let this be, we need not concern ourselves with this.

 Now let's add the Tag Along logic to the menu. This would mean that the menu moves or tags along with the gaze of the user. Look for the Simple Tagalong script from the MRTK in the Add Component of the Menu GameObject and attach it to the Menu. It adds an Interpolator. Let this be, we need not concern ourselves with this. Let's test this out. Build and deploy. By now the Menu should move with your gaze.

Let’s test this out. Build and deploy. By now the Menu should move with your gaze.

You would have noticed that the menu doesn’t always face you when it tags along. We are going to fix that with Billboarding

On the Inspector of the Menu, click on Add Component. Look for the script Billboard from the MRTK and attach it to the Menu. As easy as that.

Let’s build and deploy and try out this feature.

Now you will see that the Menu will always face you when it tags along. This is because we pivoted it on the XY Axis through the Pivot Axis Inspector value. if you want to change this, you could change the value of the same

 You would have noticed that the menu doesn't always face you when it tags along. We are going to fix that with Billboarding On the Inspector of the Menu, click on Add Component. Look for the script Billboard from the MRTK and attach it to the Menu. As easy as that. Let's build and deploy and try out this feature. Now you will see that the Menu will always face you when it tags along. This is because we pivoted it on the XY Axis through the Pivot Axis Inspector value. if you want to change this, you could change the value of the same

Now for the part where we want to pin the menu to a location of our choice:

So, this tag along and the billboarding feature is great. But it disturbs us when we want to interact with the actual GameObject in the scene. Therefore, we could write a simple script to pin and unpin the menu whenever we want.

Let’s duplicate our HomeButton because it’s easier. Rename this button PinButton. Move this button below the HomeButton on the Menu. Change it’s Label Text to Pin This 

Now we want to add a script to this PinButton. On it’s Inspector’s Add Component, create and add a new script. Call it PinScript

In the PinScript, we want to be able to add the logic that when the PinButton is tapped on, the Menu is pinned to that location where the user is currently gazing, and it remains there until it’s unpinned.

This would mean, that once the PinButton is tapped, the TagAlong and the BillBoarding features are disabled.

We also need to display an Unpin button to the user, when the Pin Button has been tapped and it’s logic has been executed. Let’s head over to Unity and duplicate the Pin Button. Rename it as UnpinButton. Change it’s Label Text to Unpin. Don’t move it under the PinButton, it needs to be shown in the same place as the PinButton. Just hide this button for now. You’d need to go over to the UnpinButton’s Inspector and deselect the checkbox on top.

 In the PinScript, we want to be able to add the logic that when the PinButton is tapped on, the Menu is pinned to that location where the user is currently gazing, and it remains there until it's unpinned. This would mean, that once the PinButton is tapped, the TagAlong and the BillBoarding features are disabled. We also need to display an Unpin button to the user, when the Pin Button has been tapped and it's logic has been executed. Let's head over to Unity and duplicate the Pin Button. Rename it as UnpinButton. Change it's Label Text to Unpin. Don't move it under the PinButton, it needs to be shown in the same place as the PinButton. Just hide this button for now. You'd need to go over to the UnpinButton's Inspector and deselect the checkbox on top.

Coming back to the PinScript, add the OnInputClicked method. Since we are dealing with this here, there is no need for the TapScript on both the PinButton and the UnpinButton. Remove the TapScript component from them.

We will write the same function for both pin and Unpin logic, but we will pass different values. We will add two public GameObjects to it which will be our PinButton and UnpinButton.

value indicates whether Tag Along and Billboarding would be enabled or disabled. 

pin indicates whether the PinButton needs to be active or not

unpin indicates whether the UnpinButton needs to be active or not

So we will pass value, pin and unpin accordingly.

See the screenshot of the code.

 Coming back to the PinScript, add the OnInputClicked method. Since we are dealing with this here, there is no need for the TapScript on both the PinButton and the UnpinButton. Remove the TapScript component from them. We will write the same function for both pin and Unpin logic, but we will pass different values. We will add two public GameObjects to it which will be our PinButton and UnpinButton. value indicates whether Tag Along and Billboarding would be enabled or disabled. pin indicates whether the PinButton needs to be active or not unpin indicates whether the UnpinButton needs to be active or not So we will pass value, pin and unpin accordingly. See the screenshot of the code.

You’ll see two more errors of SimpleTagAlong and BillBoard. They are missing a using directive,

using HoloToolkit.Unity; Add this in the code. The final code is shown below.

 You'll see two more errors of SimpleTagAlong and BillBoard. They are missing a using directive, using HoloToolkit.Unity; Add this in the code. The final code is shown below.

Go back to Unity and in the Inspector values of both the PinButton and the UnpinButton under PinScript, drag and drop the PinButton and the UnpinButton

Now let’s build and deploy and try out the pin logic.

That’s it! You can make a prefab of your Menu and port it to whichever project you want. This would help you extend its features to add more buttons to do more things like, reset objects, switch scenes, etc

 Go back to Unity and in the Inspector values of both the PinButton and the UnpinButton under PinScript, drag and drop the PinButton and the UnpinButton Now let's build and deploy and try out the pin logic.

The video of the expected output is shown below.

12 thoughts on “Tag Along, Billboarding and other menu elements

  1. I have a video textured applied to a cube.
    Simple Tagalong works fine.
    When I add Billboard and leave Target Transform set to none. It flips my video vertically, but Billboard works correctly. If I set my Target to the cube or my Video_billboard (which is what I used to load the texture onto my cube) The video does not flip but Billboard does not work the cube is locked in front view and does not face the camera.

    1. Hi James, can you send me a screenshot or even better a video of what’s happening? Tough to imagine this case with all the Inspector settings. You may also want to check which face of the cube the video is being rendered onto.. Could be why it’s flipping.

  2. Hi,
    I have followed the steps as you suggest including the codes but when I airtap, the home button disappears and no cube is created. Please Help

  3. Hey and thank you, I would love if you could explain why it only rotates a lil and not 360 around me… im using the sphere based tagalong aswell becaue the simple tagalong didn’t work. I feel the problem is that its rotateing on a center of gravity that’s ahead of me and that’s why it wont face me on my left or right. any advice you could give would be super awesome!!!

  4. I lied the buttons are not responding… do I need to set the box colliders in the buttons to is trigger or might it be a cursor issue? sadness… menu moves like a jewel thought and the script checks out, just no response from my clicks…

  5. Hi,

    I tried to follow your script for creating a gameobject. However, in my case I want a gameobject and it’s child objects to be set active by clicking on a gameobject in my current scene. Lets say i have a cube in my scene, I want to navigate to the cube by gaze, airtap it and that should lead to another gameobject appearing. Could you talk me through how to do that please?

    Thank you

    1. Hi Habeeb, So add a collider to the cube. Add the code to do something when user taps on it. So in the OnInputClicked, if the GameObject is this cube, then write the code to secondObject.SetActive(true). Does this help you?

Leave a Reply

Your email address will not be published. Required fields are marked *