Tag Along, Billboarding and other menu elements

In this article before the year ends, I bring to you another useful set of elements which you’ll need almost in every HoloLens application.

To be specific, I’m talking about a menu which can help you perform the following actions:

1. Menu appears as a tag along to the user’s gaze (Tag along)

2. Menu always present with it’s front facing the user (Billboarding)

3. Menu which can be pinned (remove tag along) and unpinned when it starts disturbing a user

With the MRTK,  it has become exceptionally easy to achieve objectives 1 and 2. For 3 there will be simple scripts which would be available in the article

Let’s start off with the usual things:

1. Create a new project. I have Unity version 2018.2.13f1

2. I’m using the latest stable release of the HoloToolkit 2017.4.3.0 Import it into your project

3. Do the usual tweaks:

a. Delete the main Camera.  Add the MixedRealityCameraParent.prefab in the Hierarchy. This will serve as the Main Camera

b. Change the Camera Settings to Hololens preferred:

Skybox to Solid Color

Background to 0,0,0,0

In the MixedRealityCameraManager script which is attached to the MixedRealityCamera object, change the Clear Flags value to Solid Color

Ensure that the Background Color is black

c. Go to Edit ->Project Settings -> Player -> Other Settings -> and change the Scripting Runtime version to .NET 4.x Equivalent and Scripting Backend to .Net

Make sure also that in XR Settings, Virtual Reality Supported is checked

We need both the InputManager and the DefaultCursor in the scene

4. InputManager

Drag and drop the InputManager.prefab into your scene

5. DefaultCursor

Drag and drop the DefaultCursor.prefab into your scene. In the InputManager’s Inspector Settings, drag and drop the DefaultCursor into the Cursor field of the SimpleSinglePointerSelector script

d. Save the scene

Now let’s create our menu:

I used a simple Cube and set the following scale ( 1,1,0.1) to make it look like a menu. Rename it to Menu or something similar.

Make sure the Menu is in front of the camera

For the material, we could use something which looks dark and almost transparent to achieve a good UI. I borrowed a material called UIDarkGray from the MixedRealityToolkit-Examples (btw, you can find a lot of other UX based prefabs and materials in there- it’s loaded). Just copy this to the Assets folder of your project. And then click on the circle next to the Element 0 on the Materials and choose this UIDarkGray

Let’s add a heading for it:

Create a 3D Text object. Call it Title or something sensible. Make sure you move it to sit on top of the Menu like in the screenshot shown. Adjust it’s Scale, Text and other attributes according to what suits the app the best.

After you’ve adjusted, make sure you make it a child object of the Menu, such that whenever you move the Menu, it’s children move with it.

Let’s add some Menu buttons. Typically, you will have many scenes in your HoloLens application and may want to switch between them. So usual buttons would be a Home, About, and some other scenes like Level1, Level2 etc. Let’s for the sake of our example add some dummy buttons.

Create a Cube. Rename it HomeButton or something similar. Change it’s Material to HolographicButtonPlate. Add a 3D Text element which serves as it’s Label. Rename it to Label or something sensible. Adjust it’s scale and other attributes to suit the button best. Move it as a child object of Menu.

The screenshot of my Label as a reference is below.

Now we will add a TapScript to the button such that when the user taps on the button, it does something.

In this example, it will instantiate a Cube.

Click on AddComponent of the HomeButton and create and add a new script. Rename it TapScript or something similar.

Add the OnInputClicked method as indicated in the screenshot of the code. Do not forget the using derivative for the IInputClickHandler, using HoloToolkit.Unity.InputModule; 

Also add the Instantiate code. You would have seen something similar already in the other tutorial

In the Inspector of the HomeButton under TapScript Cursor drag and drop the DefaultCursor

Let’s test this out so far. Build and run on the HoloLens or the Emulator. You should see a Cube instantiated when you tap on the button.

(If you get the TextMeshPro error while deploying, then remove the package Unity TextMeshPro from your project and rebuild. Go to Unity->Window->Package Manager and a pop up appears.Click on TextMeshPro and Remove. While you are at it, also remove the Analytics Library, Ads and In App purchasing)

Now let’s add the Tag Along and Billboarding logic to the menu:

This would mean that the menu moves or tags along with the gaze of the user. Look for the Simple Tagalong script from the MRTK in the Add Component of the Menu GameObject and attach it to the Menu. It adds an Interpolator. Let this be, we need not concern ourselves with this.

Let’s test this out. Build and deploy. By now the Menu should move with your gaze.

You would have noticed that the menu doesn’t always face you when it tags along. We are going to fix that with Billboarding

On the Inspector of the Menu, click on Add Component. Look for the script Billboard from the MRTK and attach it to the Menu. As easy as that.

Let’s build and deploy and try out this feature.

Now you will see that the Menu will always face you when it tags along. This is because we pivoted it on the XY Axis through the Pivot Axis Inspector value. if you want to change this, you could change the value of the same

Now for the part where we want to pin the menu to a location of our choice:

So, this tag along and the billboarding feature is great. But it disturbs us when we want to interact with the actual GameObject in the scene. Therefore, we could write a simple script to pin and unpin the menu whenever we want.

Let’s duplicate our HomeButton because it’s easier. Rename this button PinButton. Move this button below the HomeButton on the Menu. Change it’s Label Text to Pin This 

Now we want to add a script to this PinButton. On it’s Inspector’s Add Component, create and add a new script. Call it PinScript

In the PinScript, we want to be able to add the logic that when the PinButton is tapped on, the Menu is pinned to that location where the user is currently gazing, and it remains there until it’s unpinned.

This would mean, that once the PinButton is tapped, the TagAlong and the BillBoarding features are disabled.

We also need to display an Unpin button to the user, when the Pin Button has been tapped and it’s logic has been executed. Let’s head over to Unity and duplicate the Pin Button. Rename it as UnpinButton. Change it’s Label Text to Unpin. Don’t move it under the PinButton, it needs to be shown in the same place as the PinButton. Just hide this button for now. You’d need to go over to the UnpinButton’s Inspector and deselect the checkbox on top.

Coming back to the PinScript, add the OnInputClicked method. Since we are dealing with this here, there is no need for the TapScript on both the PinButton and the UnpinButton. Remove the TapScript component from them.

We will write the same function for both pin and Unpin logic, but we will pass different values. We will add two public GameObjects to it which will be our PinButton and UnpinButton.

value indicates whether Tag Along and Billboarding would be enabled or disabled. 

pin indicates whether the PinButton needs to be active or not

unpin indicates whether the UnpinButton needs to be active or not

So we will pass value, pin and unpin accordingly.

See the screenshot of the code.

You’ll see two more errors of SimpleTagAlong and BillBoard. They are missing a using directive,

using HoloToolkit.Unity; Add this in the code. The final code is shown below.

Go back to Unity and in the Inspector values of both the PinButton and the UnpinButton under PinScript, drag and drop the PinButton and the UnpinButton

Now let’s build and deploy and try out the pin logic.

That’s it! You can make a prefab of your Menu and port it to whichever project you want. This would help you extend its features to add more buttons to do more things like, reset objects, switch scenes, etc

The video of the expected output is shown below.