In this post, you’ll learn:
1. How to import the Mixed Reality Toolkit to your project
2. How to use the Speech Input Source and Speech Input Handler scripts provided in the MR Toolkit to use voice commands in your app
Prerequisites:
Make sure that you have the environment set up as described in my previous post
Start with creating a new Unity Project
Import the MR Toolkit which you previously have downloaded to your hard disk (if not, check here)
In your Unity project, go to Assets->Import Package->Custom Package
This opens up the explorer. Navigate to where you saved your MR Toolkit previously downloaded
Select it and click on Open
Click on Import when you see this screen. The import will take a few minutes to complete.
You’ll know it’s succesful when you see the HoloToolkit folder in your Assets folder in the Project window in the Unity Project
Let’s first look at how we can use the Mixed Reality Camera parent prefab. In the older HoloToolkit, you had to make a few adjustments to your camera in the scene so that your app can look better. This involved changing the Skybox on the Camera to Solid Color, changing the clipping planes and so on.
However, with the Mixed Reality Camera parent prefab, this got easier. The prefab is what it implies- it is a pre-done object which one can drag and drop in the Hierarchy window of a project so that it can be readily used. Start by deleting your Main Camera which is present already in your Hierarchy window
Start typing Mixedrealitycamera in the filter tab in your Project window. Once you find the MixedRealityCameraParent, drag and drop it to your Hierarchy window.
Okay, now let’s get to what we want to do – rather what I want to do 🙂 I’m fond of warriors and strategy games. And this project is a tiny bit attempt to get back to my childhood, But I digress. What I want to say is, let’s use voice commands to make a warrior go forward, backward, left and right and also attack. Don’t expect a full fledged game here. It is just a simple example to focus on using voice commands on the Hololens.
What I’ve done here is use free assets from the Unity Asset Store. If you want the warrior I am using, then simply download the assets from here https://www.assetstore.unity3d.com/en/#!/content/47320
To download into your project, first add the Asset Store into your project. (Of course you need to first create a Unity Account and sign into this on the link that you just opened above in the Asset Store. It’s free and you can use other free assets in the future for your projects.) This will then appear as a tab next to your scene tab.
Type in ‘warrior pro bundle 3’ in the search field and press enter
Scroll down to see this asset which is free. Click on it
Once you see this screen, click on Import. The import takes a few minutes.
Make sure to click on Import once you see this screen. You’ll know the import is successful when you see the ‘Warrior Pack Bundle 3 Free’ folder in your Assets folder in your project
Now similar to how you added the MRcameraprefab into your Hierarchy window, search for ‘Crossbow Warrior’ and drag and drop this to your Hierarchy window. This will show the warrior asset in the scene window as well.
Click on the Crossbow Warrior object in the Hierarchy window and in it’s Inspector, click on the gear symbol on the Transform and reset its position. We want to make sure that our camera sees him
Now expand the MixedRealityCameraParent and click on the MixedRealityCamera on the Hierarchy window and you’ll see a box saying Camera Preview on the left. If you’re able to see the warrior in it, you’ll know you have set the position of both the warrior and the camera right
Now click on the Crossbow Warrior gameobject and you’ll see that it has 2 components attached previously. We will use the animations which came with the asset.
For now, simply click on Add Component -> New Script and type in the name of your script. I call it AnimateWarrior. Click on Create and Add
This will be added to the Crossbow Warrior object. Right click on it, and click on Edit Script it will open in Visual Studio (or MonoDevelop if you use that)
In your AnimatorWarrior script, replace the code by what I have provided here. We are simply going to change the position transforms to move forward, backward, left and right.
For attack, we will use what the Warrior asset already comes with. We don’t need to understand what the warrior does in this tutorial. We just reuse what has been provided. In place of the warrior, you can also use a cube and use the same code to transform its positions. Our objective here is just to use speech to navigate whatever GameObject we use in the Hololens app.
Save your script.
Now for the fun speech part.
First right click on your Hierarchy window and Create empty GameObject. Rename it in the Inspector as Manager
Now on the Manager, right click and create empty GameObject. This should be the child of the Manager
In the new MR Toolkit, voice commands can be added by 2 components: The Speech Input Source and the Speech Input Handler.
Our GameObject in the Manager is where the Speech Input Source will be added. This maintains all the keywords which can be used to generate voice commands in our app.
The Speech Input Handler will be added to the GameObject we are trying to manipulate or react to the voice commands that we provide. In this example, it is the Crossbow Warrior. If you have used a Cube instead of a Crossbow Warrior, then add the Speech Input Handler to that.
Click on Add Component on the GameObject and search for Speech Input Source (this will be available in the MR Toolkit) and add it.
Drag and drop the CrossBow Warrior to the Animator field. This is to use the animations that came with the warrior asset when we downloaded it
Click on the Crossbow Warrior object and Add Component. Search for Speech Input Handler and add it. This will also be added from the MR Toolkit.
Now search in the Assets folder for InputManager and add it as a child to the Manager (drag and drop)
In the Animator sript, we had refered to ISpeechHandler which the script requires and gets from this InputManager object to function properly
Check the Is Global Listener flag – this ensures that the app is listening and waiting for your speech input
Now add all the commands in the Speech input Handler. Click on the + to add more. We need to have five in total for each of the keywords that we included in the Speech Input Source
When you’re done with the Speech Input Handler , it should look like the screenshot.
Drag and drop your Crossbow Warrior object into the box on the left. (This is the object to be manipulated upon speech input)
Select the dropdown and navigate to AnimateWarrior and choose the UpdateWarrior method. (This is the method to be called upon speech input)
Since all our commands use the same method, you are calling the same method for every keyword. But in your code you can write methods for each keyword that you want to use
Now you are all set to build and deploy your app.
I’ll show it again in this article. But the steps to build and deploy are essentially the same.
Go to Edit – > Project Settings -> Player and navigate to Other Settings and change scripting backend to .NET
Under Edit -> Project Settings -> Player -> Publishing Settings, scroll down and check Microphone. Of course we need the microphone to detect our voice input
Under Edit -> Project Settings -> Player -> XR Settings, check Virtual Reality Supported and ensure that it shows Windows Mixed Reality
Go to File -> Build Settings -> select Universal Windows Platform.
Set Target device to Hololens
Build Type to D3D
SDK to Latest Installed
Build and run on Local Machine
Check Unity C# Projects
Add Open Scenes and choose your current scene (you have to save the scene before by clicking on Save as and to any place in your local hard drive)
Once you click on build, the explorer opens. Add new folder called app and Select Folder
Once the build is complete, double click on the solution (this will be the .sln file in the app folder) and go to visual studio and select
Release. x86, Device in the top menu (if you have a Hololens)
if not, select Release, x86, Holographic Emulator
Now press Ctrl-F5 or Debug -> Start without Debugging to start deployment
I’ve added a Gif of what should happen when you say Forward, Left, Right, Back and Attack (in that order)
*I apologize for the poor Gif quality.
Errors and Solutions
Error: Windows Mixed Reality toolkit only wiorks with Windows Fall Creators Update –
Solution: That means you don’t have the Fall Creator’s Update which is a prerequisitie to run MR Toolkit. Download it from here: https://support.microsoft.com/en-us/help/4028685/windows-10-get-the-fall-creators-update
Error: Voice input does not work
Solution: First check under Edit -> Project Settings -> Player and under Publishing Settings if microphone is checked.
If it is checked, delete the app folder and build it again. I have seen that sometimes the capabilities are not included in the packageappmanifest when you try to rebuild on the same folder. What I check is directly in the .sln file in Visual Studio, look for packageappmanifest and under capabilities, microphone should be present. check package.appmanifest and see if the microphone capability is enabled. if not, then player settings and checkbox microphone. If on the emulator, see if your default microphone on the PC is enabled
Error: Null Pointer exception to SpeechInputHandler
Solution: You have attached the script which does something to manupulate the GameObject in context upon speech input wrongly. You need to ensure that the SpeechInputSource is attached to the Manager’s GameObject, The SpeechInputHandler is attached to the GameObject which you want to manipulate via voice commands and that InputHandler is also a child of the Manager that you created. All 3 components need to be present and attached rightly to the objects as shown in this example.
That’s it! Let me know if you made the warrior move on your command 🙂
