You must have come across the problem when you use Holograms or objects in your HoloLens apps, you’d need to do either of three things at some point:
1. Move it around
2. Make the object bigger or smaller
3. Rotate the object
That is exactly what I’m going to cover in this article.
Microsoft makes it very easy for us to do this by providing what they call the App bar and the Bounding Box.
The above link should quickly explain to you about the concepts
So, let’s get started…
1. Create a new project in Unity.
It’s better if you get Unity 2017.2 version as you’d need to make it compatible with the next must have – Mixed Reality Toolkit- Unity
The Mixed Reality Toolkit- Unity is using the base code from the normal MR Toolkit, but it has scripts and prefabs specially designed to be used with ease within Unity.
2. Import the Mixed Reality Toolkit-Unity from https://github.com/Microsoft/MixedRealityToolkit-Unity Latest Master branch Release (as of today 6th May, 2018 is https://github.com/Microsoft/MixedRealityToolkit-Unity/releases/tag/2017.2.1.4 )
3. Do the usual tweaks:
a. Delete the main Camera. Add the MixedRealityCameraParent.prefab in the Hierarchy. This will serve as the Main Camera
b. Change the Camera Settings to Hololens preferred:
Skybox to Solid Color
Background to 0,0,0,0
c. Save the scene
Now let’s get some objects within the scene which we can manipulate.
I downloaded and imported the Furniture Pack 3D Asset pack from the Asset Store for free: https://assetstore.unity.com/packages/3d/props/furniture/pack-gesta-furniture-1-28237
If you want, you can even start with a simple 3D object like a cube in Unity. The furniture assets just make it a bit more fun 🙂 It will be like setting up your furniture in your living room in 3D
Once you import it, it will show up as Furniture_ges1 under your Assets
Now let’s get some furniture into the scene. You can use whatever prefab came with the pack (or your own). But I used the following:
Bed, TV, Printer, Cupboard, A big table, A little table. Simply drag the drop these prefabs from the furniture_ges1 asset pack into the scene. I also grouped them under MovableObjects, which is an empty GameObject- just to keep things organized
Now let us check if we are able to see the furniture in the scene.
Build and deploy. (It is best if you use a HoloLens for this experience, but the Emulator is also ok)
So if everything has been followed step by step, then you should be able to see the furniture in the space. Never mind for now that it is hanging in the air or is too big.
Once you’re sure that everything is visible, then adjust the scale to whatever you want. I reduced things to 0.5 and for the smaller things like printer and TV to 0.2
I just made quick camera adjustments to face the furniture and set up things in a presentable way so that it doesn’t look weird when you see through the HoloLens the first time (this is completely optional because you’ll pull and arrange objects or make bigger or smaller anyway)
Now for the part where we can move the objects around using the pinch and drag gesture
1. First add the Default Cursor.prefab from the HoloToolkit. Search for it in the Assets and drag and drop it into the Hierarchy
2. Search for the InputManager.prefab in the Assets and drag and drop it into the Hierachy as well
3. On the Inspector settings of the InputManager, under SimpleSingerPointerSelector component, drag and drop the Default Cursor into the Cursor field (if you haven’t seen the article Gaze and Tap on objects, then this is a good time to refer to it)
4. For every object that you want to pinch and move, you need to ensure that there is a corresponding Collider of some sort. I added Mesh Colliders to the individual objects which are movable
5. Now we add the script to each movable object which makes the objects draggable by pinch and move. The MR Toolkit-Unity has already provided us with this script. It is as easy as attaching this script to the object. On the object’s Add Component, search for HandDraggable.cs and click to add it
Quickly build and deploy and test it. The objects should be movable now. Great going! Now let’s add the BoundingBox part
This is the logic which will create a box around the object which you can click on to make bigger or smaller. You can also rotate the objects with the Boundingbox
1. Again, the MR Toolkit-Unity provides us with the script to do all this with ease. In the Assets folder, search for BoundingBoxRig.cs script and add it to each movable object. You can drag the drop the script to the object directly
2. In the BoundingBoxRig script, there is a BoundingBoxBasicPrefab field. Search for the BoundingBoxBasicPrefab in the Assets folder and drag and drop this into the field.
In case you are using a 2D object, make sure you select the Flattened Axis field to Flatten Auto. But for 3D objects, the default Do Not Flatten should do
Also if you are using the settings as is, you’ll see that the AppBar, which appears in front of each object is way too tiny. So let’s make that bigger.
Search for the AppBar.prefab in the Assets. This is already referenced in the BoundingBoxRig script on every movable object. So we need to change it only once in the prefab.
Change the scale of the AppBar to 1,1,1. When you test it, you can also increase it to suit your preference.
The handles on the BoundingBox are too small for me. Tapping on them is stressful with the sizes already existing and so I headed over to the BoundingBoxRig.cs Script and increased the scale
I changed the scaleHandleSize and the rotateHandleSize to 0.07f,0.07f,0.07f and 0.06f,0.06f,0.06f respectively. This way they are bigger and easier to tap and grab
I also increased the Scale Rate on the BoundingBoxRig script to 4. 1 is too low in my opinion, but you can try whatever suits you
Okay, now you need to just build and deploy and you can see the App bar in front of the movable objects.
On the AppBar’s Adjust button, Tap to see the Bounding Box.
Tap and Grab the square handles and pull up to make bigger and push down to make smaller
Tap and Grab the round handles to rotate the object
The objects can be pinched and moved around in BoundingBox mode or not
Have fun setting up the furniture!!
Attached is a video of how the output should be like.
(P.S: it is extremely tough to test out the controls on the Emulator, but if you have no other go you can. Try with smaller objects so that you don’t have to move the mouse much to adjust your gaze)
Questions, Comments? Let me know below!
Errors and Solutions:
Error: Object not moving
Solution: Check if Cursor, InputManager is in the scene
Reference Cursor in the SimpleSinglePointerSelector component of the InputManager
Make sure the object has a Mesh Collider
Make sure the HandDraggable.cs is attached to the object
Amazing, thanks!
I’m glad it helped! Thanks for the feedback Marco!
Hi we just tried you manual (10x by the way its awesome!)
but we stopped at the point when you need to drag BoundingBoxRig.prefab in to the bounding box rig script component under the bounding box prefab filed ().
For some reason it doesn’t let us to choose it, any ideas?
Thanks:)
Hey ArielF,
Thanks for the feedback! 🙂 You need to drag in BoundingBoxBasic.prefab into the Bounding Box Rig script and not the BoundinfBoxRig.prefab.
Hi Nischita, in your first demo it appears that you are rotating the object in the Y axis, can you explain how did you manage to do that? Thanks
Hi Nischita, in your first demon with the wooden furniture it appears you rotate the object in the Y axis, can you explain how you did that?
In my test the object rotates from the right side only.
Hi Erick, If you are using the BoundingBox, there should also be the rotate (round) handles on the top for Y Axis rotation. Do you see them? If not, maybe your object is sort of flattened which may hide the rotate handles on the top. Maybe send a screenshot and I can help you more?
Dear Nischita, In the latest MRTK, I cannot seem to find the BoundingBoxRig.cs script and the BoundingBoxBasic prefab. Is this the case or am I missing something? Thanks
Hi Joshua, Did you try the MRTK or the MRTK-Unity? https://github.com/Microsoft/MixedRealityToolkit-Unity The Unity version of the MRTK should have BoundingBox stuff for sure.
Thanks for your reply, Nischita. The newest toolkit version 2017.4.0.0 that I downloaded from https://github.com/Microsoft/MixedRealityToolkit-Unity/releases does have the app bar and bounding box, but for some reason my old toolkit did not. Anyhoo, I have it now! Cheers
Another question if I may: I am using the recommended Unity 2017 LTS with the latest MRTK 2017.4.0.0 but I have noticed that when you use the native menu option to configure your scene, firstly the camera’s Clear Flags do not change from Skybox to Solid Color (so we have to do it manually) and also the little hands are missing for air-tapping. These were always sorted when you configured the project via that menu option. Have you noticed these?
If you’re talking about the MRTK – Configure option and Apply MRTK settings and scene settings, then yes -> I have also noticed that Clear Flags needs to be manually changed to Solid Color. Even in Unity’s 2018 version this happens. About the hands, I’ve really never needed them so not observed this. It’s so complicated to keep track of these tiny changes since each upgrade something new pops up or something old gets discarded 😉
The bounding box isn’t working for me. When I click on Adjust, nothing happens and the console displays this error:
NullReferenceException: Object reference not set to an instance of an object
HoloToolkit.Unity.InputModule.InputManager.RaiseBoundingBoxRigActivated (UnityEngine.GameObject boundingBoxRiggedObject) (at Assets/HoloToolkit/Input/Scripts/Utilities/Managers/InputManager.cs:852)
Any help?
Hi Rupe, can you send me a screenshot of your project’s inspector? Especially the object’s inspector on which the BoundingBox is added. That’ll help me see what else could be wrong. thanks!
hey.. after I followed the tutorial that you provided, everything went well when playing on unity, but why can’t build to hololens emulator?
Hi Wawan, Did you try the build procedure explained in the article https://codeholo.com/2017/11/23/get-started-with-hololens-development-with-mr-toolkit/. In this article, look for the section where it specifically says…”To deploy to the Emulator, Choose Release, x86, Hololens Emulator on the top menu. If you don’t see the Hololens emulator, then download it from here (https://go.microsoft.com/fwlink/?linkid=852626) and restart Visual Studio.”
yes, i understand about that, but why the object can’t move after i deploy in hololens ?
Hi, Not sure if you are trying it on the HoloLens or the emulator. If you are building it and deploying it onto the HoloLens then you need to use pinch and drag gesture for the move part. The article has a video attached where I demo on how to move the object. Also I’d suggest to try out the tutorial https://codeholo.com/2018/06/19/twohandmanipulatable-part-2-of-moving-scaling-rotating-objects/ since this is a more updated, latest version to move, scale or rotate objects in HoloLens apps
how to order the movement can faster when i pinch ?
As I said, try the https://codeholo.com/2018/06/19/twohandmanipulatable-part-2-of-moving-scaling-rotating-objects/ it is much faster and the transition is smoother.
My application works almost perfectly.
But the AppBar does not face the front. How do i fix it?
Hi, The AppBar should by default appear in front of the user. If it doesn’t just play around with it’s settings in the Inspector. Maybe you are moving and the AppBar moves with your gaze, so it then appears as not in front of the object. If you can, also send me a screenshot of the object and the settings in the Inspector and I can also check what the problem could be.
Dear Nischita,
I had a problem with your tutorial. Indeed, it works as you when I apply what you did on a simple object. However, I have a problem when I create a game object which is composed by an object and a button (which is on it). When I want to apply your tutorial to my game object, it does’nt work and I don’t know why… I can’t seperate my button and my object so could you help me ? Thank you very much.
Hi Mathilde, Please send me a screenshot of both your object in Unity and where you are trying to manipulate- also the inspector settings on the object would be good to see before I can help debug. Thanks!
Hi,
I need some help from you. When we load a 3D object in a space/room it doesn’t retain its initial position if we move any other new room/space.
How can i fix these issue. Please help
You need to anchor the model in the space to “fix” the model in it’s position. Read more here: https://docs.microsoft.com/en-us/windows/mixed-reality/spatial-anchors . I will also write an article in the future about anchoring.
Hope this helps!
Hello Nischita,
I am trying to use the bounding box to reshape an object and then remove the bounding box app bar but can not figure out how to remove the app bar without removing the object as well. I would appreciate any help to this problem.
Hi James, If I understand you right, then all you have to do is go to the Bounding Box rig script on the Inspector. Under the field ‘App bar Prefab’ just select and choose None. I hope I have understood your problem, if not a screenshot would help me better understand.
This works great! However, I want the object to stop moving or rotating if it collides with a room mesh created with spatial mapping or spatial understanding. Colliders don’t seem to accomplish what I would expect intuitively. Any tips or tutorials out there to add this functionality? Thanks for your help and the great tutorial!
Hi Gary, thanks for the feedback! I haven’t yet explored the spatial mesh/mapping part, so I can’t tell you for sure. But, I saw this: https://github.com/MicrosoftDocs/mixed-reality/blob/master/mixed-reality-docs/spatial-mapping-in-unity.md and also the https://docs.microsoft.com/en-us/windows/mixed-reality/holograms-230 . Maybe these can help?
Hi Gary, Another thing I stumbled upon which could help you: https://docs.microsoft.com/en-us/windows/mixed-reality/mr-azure-310 – check the spatial mapping and the GazeCursor part. That would show you how to set the SpatialMapping colliders corerctly and then you can disable moving when the collider of your GameObject interacts with the SpatialMapping collider
Thanks, Nischita. I successfully completed the Mixed Reality 230 tutorial, but the setup doesn’t do exactly what I want. By adding a spatial mapping collider to the scanned room mesh and adding a rigid body to each object, I could get physical interaction object to object and object to floor or wall. When I move a cube object against the wall it will bounce off or if I hold onto it, it sort of squirms around. If I constrain the rigid body in space in the x, y, and z direction it will stay put when I let go, but it again twitches and squirms rotationally. Plus, I can drag it right through the wall if I want to. What I’d like it to do is orient itself flat against a surface when I move it there, but I am prohibited from dragging it through the wall. Basically, I want it to behave like all the standard app windows in HoloLens, where one can place them anywhere in midair or against a surface. Disabling movement once the colliders interact seems to be the problem….
Thank you so much Nischita, this was exactly what I was looking for!
Glad to know Enrique 🙂 Welcome!
I do not get the bounding box with handles it just blinks.
Sorry should have been a little more clear. I have the objects and can drag them around the room I just cannot scale them. I am just trying to scale the dresser for now. The AppBar appears and seems to work. when clicked that is when the dresser just blinks.
Hey Nischita!
Thanks for the awesome tutorial. I did this once before and it worked perfectly. However, I’m building another app and the bounding box doesn’t show up which i think is because the app bar doesn’t respond to my inputs at all. Could you please let me know how I can go about fixing this? Thanks in advance!
Hey Godfrey, Thanks for the feedback. Hmm, have you included the BoundingBoxBasicPrefab in the Bounding Box Prefab Inspector field of the BoundingBoxRig script? Also you can drop that in the AppBar inspector value in the AppBar component. If not, send me a screenshot at codeholo@gmail.com and I can help you better.
Hi James, For scaling, the handles need to be pinched and then you need to increase distance between both fingers while pinching the object in context which needs to be scaled. In any case, see my part 2 of the Moving Scaling, Rotating apps https://codeholo.com/2018/06/19/twohandmanipulatable-part-2-of-moving-scaling-rotating-objects/ … This is a cleaner way to perform manipulation with the TwoHandManipulatable script. I’d recommend this method.
Hi Nischita,
I want to thank you for this tutorial. But i have a question about the rotate and the resize of objects: why these operations are very slowly, and there is any parameters that i can it access to fast the rotation and resize of object. In contrast, the moving of object is perfectly.
Thank you
Hi Haythem, yes- I agree too that these operations are quite slow. Have a look at the second part of the tutorial- https://codeholo.com/2018/06/19/twohandmanipulatable-part-2-of-moving-scaling-rotating-objects/ This is much faster with movements and scaling.
Hi Nischita, Well, the second one is more helpful. Therefore, I want to ask you about communication between HoloLens and a server via TCP/IP or other protocols. Have you an idea concerning this communication or can you indicate me a tutorial explain the steps to release this communication on sending data from HoloLens to server.
Thank you
Hi Haythem, I’m yet to explore protocols with HoloLens. But you could ask here: https://holodevelopers.slack.com/messages/C1AK8E6CS/ it’s the new HoloLens forums which has moved to the Slack group.
Hi Nischita. Ok thank you, so if you want can I send you some samples about this field when I find a good results. Have a nice day.
Yes please! That would be great. Thanks Haythem! 🙂
Hi Nischita, where do I go to add the mixed reality parent prefab? I can’t seem to find it anywhere. Thanks for your help.
Hi Arav,
Are you looking for MixedRealityCameraParent prefab? And which MRTK are you using?
I downloaded the MRTK but i dont have MixedRealityCameraParent.prefab and all the other things all i have are scripts.
I have the newest MRTK version
Hi Charles, I haven’t yet upgraded to the newest MRTK (v2) because there are lots of changes and features, especially catering to HoloLens 2. I’m waiting for the device and when it does come, I will upgrade everything to MRTK v2.
I followed the tutorial and it works. However now I am trying to delete the object by having a function in the script. I managed to delete the object itself, but BoundingBoxBasic(Clone), center and AppBar(Clone) persist as game objects in the scene. Any advice about how to cleanly delete the full instance of the prefab?
Thanks Nischita for the tutorial. It worked for me but now I have struggles when trying to destroy the objects from code. I can remove the objects but the center, BoundingBox and AppBar persist. When having multiple objects I even have multiple AppBar(Clone) etc which makes it even harder. Do you have any advice how to cleanly remove the full used prefab? Thanks in advance!
Hi Ziska, Welcome. Glad the tutorial helped. Could you reference the whole prefabObject as a separate GameObject in your script and then do a Destroy on it? Maybe you are just destroying the single object and not the whole prefab with the BoundingBox and AppBar? Else try to GetComponent() on that object and call Destroy on it… If not send me the script, and I will try it here.
How do I correctly reference the whole prefab Object?
Currently i did it this way:
public class Obstacle : Singelton
{
public GameObject obstaclePrefab; //corresponding to prefab from tutorial
private GameObject[] obstacle;
public static int counter;
void Start(){
counter = 0;
obstacle = new Gameobject[100];}
public void SetObstacle(){
obstacle[counter] = Instantiate(obstaclePrefab);
obstacle[counter].name = “obstacle” + counter;
obstacle[counter].SetActive(true);
counter++;}
public void ResetObstacles() {
for (int i = 0; i < counter; i++){
GameObject buffer;
buffer = GameObject.Find("obstacle" + (i));
Destroy(buffer);}
counter = 0;}
}
I tried to reduce the script to basic components. I also directly used the array to destroy the object (Destroy(obstacle[i])) which had the same result. Furthermore I tried
BoundingBoxRig test = obstacle[i].GetComponent();
Test.OnDestroy();
and
Destroy(obstacle[i].GetComponent());
Both seem to not have any effect at all. To be honest by now I think I have lost track of all the stuff I tried but apparently I am getting something wrong. Already thanks for your help and I hope you have more advice for me.
Best regards, Ziska
Hey Ziska, make sure you destroy the BBrig and the AppBar first and then the prefabObject. Also, Why are you deriving from Singleton? Try MonoBehaviour. Thirdly, check with logs at every place if there is something to destroy. For example: GameObject.Find may not be giving a proper result. Similarly, check if obstacle array is containing the GameObjects. Debug at every stage to see if there is no null object etc.
Thanks, Leonor Chelmsford for codeholo.com