Changing scenes and viewing UI elements rightly in HoloLens apps

Recently I came across an interesting problem when developing a HoloLens app. I was working with changing scenes with UI elements. And I just wasn’t able to see the UI elements when changing scenes…

I did resort to the HoloLens forums to ask the experts there, on how they did it.

And since this is slightly confusing, especially when you consider UI elements, I thought I’d write an article about this.

1. Start by creating a new Unity project. The version I’m using is Unity 2018.1.5f1 and using Mixed Reality toolkit-Unity for the HoloLens related assets.

I’ve also upgraded to OS RS4

2. Import the Mixed Reality Toolkit-Unity from https://github.com/Microsoft/MixedRealityToolkit-Unity/releases/tag/2017.4.0.0 

Now, before we step into the usual tweaks for the HoloLens apps, let me explain the whole project setup for us to effectively carry out this example. I thought a pictorial representation would help better:

Scene 0: Let’s call it Base Scene

Delete the Directional Light. We don’t need it.

As indicated in the pictorial representation, we need a base scene which contains the assets which are recurrent and necessary in other scenes. In this example, they will be:

1. MixedRealityCameraParent

a. Delete the main Camera.  Add the MixedRealityCameraParent.prefab in the Hierarchy. This will serve as the Main Camera

b. Change the Camera Settings to Hololens preferred:

Skybox to Solid Color

Background to 0,0,0,0

On the MixedRealityCamera, look for the MixedRealityCameraManager script and change the Clear Flags to Solid Color

Ensure that the Background Color is black

c. Go to Edit ->Project Settings -> Player -> Other Settings -> and change the Scripting Runtime version to .NET 4.x Equivalent and Scripting Backend to .NET

Make sure also that in XR Settings, Virtual Reality Supported is checked

2. InputManager

Drag and drop the InputManager.prefab into your scene

3. DefaultCursor

Drag and drop the DefaultCursor.prefab into your scene. In the InputManager’s Inspector Settings, drag and drop the DefaultCursor into the Cursor field of the SimpleSinglePointerSelector script

4. BaseObject – which has a script attached to it, which loads Scene 1

Add an empty GameObject. Call it BaseObject or something similar.

To the BaseObject, add a script and call it LoadMyScene or something similar

Open the LoadMyScene in Visual Studio and edit it to add the following code:

If you notice, I have added Scenes/Scene 1 in the code. This is because I have created a folder in my Assets called Scenes, where in I will be moving all my Scenes in the project. This is just for better organization.

Now, save the scene as Base Scene or something similar

Scene 1:

Go to the Scenes folder and create a new scene. Call it Scene 1

Open this scene

Delete the Main Camera

Add a Canvas object to the scene. In the Canvas’ Inspector settings, change the Canvas Render Mode to World Space

Drag and drop the Canvas object into the  Canvas Inspector field of the Canvas Helper script

Now we need to tell the Canvas that everytime it is seen, it needs to use the MixedRealityCameraParent.prefab as the right Camera to be used in the scene. Remember our Base Scene is loading the MRCameraParent and the InputManager for each scene in the background. Therefore, the Canvas needs to be seen by this MRCameraParent which is loaded via the BaseScene.

We also cannot have two Cameras, InputManagers etc in our scenes because Unity in general destroys objects on loading. Therefore, even if you add a Camera in your Scene 1, this will be destroyed upon loading of the scene as you switch back and forth between scenes. Therefore, we will apply the following fix to tell the Canvas to use the Camera of the Base Scene

Look for the Canvas Helper script and add it to the Scene 1’s Canvas. Drag and drop the Canvas object into the  Canvas Inspector field of the Canvas Helper script .This is fortunately provided to us in the HoloToolkit itself to make our life easier.

Let’s take a minute to look at this script:

So if you see in the if statements, the Canvas Camera is set to the Instance of the MRCamera at runtime. i.e this is the fix for the problem we are trying to achieve

What I’m talking about is that in the Canvas, you usually set the Inspector field of Camera to the Camera that you are using in the scene. Since this cannot be done directly where you refer to an object in the other scene, we fix this in runtime

Add a DefaultCursor into the scene

Let’s try and see what happens so far.

Firstly, go to File->Build Settings and add both your scenes. NOTE the order. You should always have your Base Scene built first followed by your subsequent scenes in that order

(You can drag and drop the Scenes into the Scenes in Build window)

Build the project.

Before you deploy, let’s just hit Play in Unity and observe what happens.

1. The Base Scene is loading – but you can’t see it since it is transitioning very quickly to Scene 1 (remember our LoadMyScene script)

2. In the Project’s Hierarchy, the Scene 1 appears

3. Below it DontDestroyOnLoad objects appear. This is from our Base Scene

4. If you click on the Camera Inspector field of the Canvas in Play Mode, you’ll see that the MRCamera of the DontDestroyObjectsOnLoad will appear highlighted. So our Canvas is indeed using the right Camera 🙂

Now let’s carry on to add some UI elements to our Scene 1

I’m going to add two Buttons to the Canvas. Right click on Canvas and add UI->Button

Rename this to ButtonScene2

I’m just making some UI changes like changing the colors and size of the Buttons and adding some color transition effects- so that you know it changes color when tapped. This is optional but I like visual feedback 🙂

Go to the Text part of the Button and edit the Text to ButtonScene2 so that we can see it on the Button

Right click on this button and click Duplicate. Now you have two Buttons

Rename the second one to ButtonScene3. Similarly make the changes with it’s Text to Button Scene 3 and add color transitions etc

Move around the Buttons to fit in the Canvas however you like (the idea is sufficient distance between them to tap on them)

Now we add the Tap Logic to the Buttons to transition to new scenes (I have covered the Tap tutorial in detail here:

https://codeholo.com/2018/01/26/using-gaze-and-tap-to-select-the-objects-of-your-choice/ so I’m not going to explain that part in detail)

On ButtonScene2, add a script and call it TaptoChangeScene or something similar

Edit this in VS to add the following code, We use a switch case here since we have many buttons and we can reuse the same script for these buttons that we will use across scenes

We will also add the logic to jump to the right switch case based on the tapped object’s name

So we see that if ButtonScene2 is tapped, then Scene 2 is loaded and if ButtonScene3 is tapped, then Scene 3 is loaded

Scene 2 and Scene 3:

Before we test this out, go to your Scenes folder and create two new scenes. Call them Scene 2 and Scene 3

Open Scene 2 and delete the Main Camera

Add a Canvas and add a UI Button

Call it Button Scene 1

Change the Canvas Render Mode to World Space

Add the Canvas Helper script to the Canvas

Drag and drop the Canvas object into the  Canvas Inspector field of the Canvas Helper script

Add a 3D Text element to the Canvas and in it’s Text Mesh Inspector field, add text which says “You are in Scene 2” or anything you want, really. This will give you visual feedback

Add a DefaultCursor into the scene

Do all of the above for Scene 3 too

Okay, now let’s build and deploy. 

When you do deploy, you probably won’t see anything. This took me a lot of time to figure out why. The problem was that we are adding the Camera at runtime and we need to see the Canvas in front of the Camera.

Let’s fix it scene by scene.

For Scene 1:

Note that in the Base Scene, the MRCamera is at Position 0,0,0. We need to position the Canvas so that it is seen by this Camera. The easiest way to do this is adding an MRCamera in Scene 1 just temporarily and moving the Canvas around in front of it, so that you see it.

The values I have used are:

Canvas: 100, 150, 430

MRCamera (Base Scene) : 0,0,0

After you’ve used this MRCamera to set your Canvas right, simply uncheck the box to hide the object. We don’t need it. You could also delete it.

For Base Scene:

Go to the MixedRealityCamera and increase the Field of View to 30. This will ensure a wider view of the Canvas elements in the other scene

For Scene 2 & Scene 3:

Canvas: 100,150,430

Let’s add the last tap logic part to both scenes 2 and 3,

Just drag and drop the TapToChangeScene.cs script to the buttons in Scene 2 and Scene 3

Change the colors and transitions for the buttons so that you have visual feedback

Add the following logic to the switch cases. Since both buttons in Scene 2 and Scene 3 are named ButtonScene1, there is only one switch case for this

Ok, now that we have put them all together let’s try it out.

I have attached the video of the output below:

Let me know if you have any problems with switching scenes and viewing UI elements on the HoloLens apps!

Errors and solutions:

Error 1: Scene ‘Scenes/Scene 1’ couldn’t be loaded because it has not been added to the build settings or the AssetBundle has not been loaded.

Solution 1: Add the scene which you’re trying to load to the build settings use the menu File->Build Settings…

Error 2: When you deploy, if you see nothing, your UI elements are not appearing in front of the Camera at an optimum distance

Solution 2: See my detailed write up about scene by scene fixes earlier on in this article