Take photos and videos from within a HoloLens application

Hey there! I have been talking about dynamically taking pictures/videos from the HoloLens in my newsletters for quite some time now. If you have needed an app where you could trigger taking a photo or recording videos dynamically, then this tutorial is for you.

Let’s start with part 1: Taking pictures from an app via voice input

The usual things to do:

1. Create a new project. I have Unity version 2018.3.12f1

2. I’m using the latest stable release of the HoloToolkit 2017.4.3.0 Import it into your project

3. Do the usual tweaks:

a. Delete the main Camera. Add the MixedRealityCamera.prefab in the Hierarchy. This will serve as the Main Camera

b. Change the Camera Settings to Hololens preferred:

Skybox to Solid Color

Background to 0,0,0,0

In the MixedRealityCameraManager script which is attached to the MixedRealityCamera object, change the Clear Flags value to Solid Color

Ensure that the Background Color is black

c. Go to Edit ->Project Settings -> Player -> Other Settings -> and change the Scripting Runtime version to .NET 4.x Equivalent and Scripting Backend to .Net. Ensure that UWP is selected as the Build Platform.

Make sure also that in XR Settings, Virtual Reality Supported is checked

We need both the InputManager and the DefaultCursor in the scene.

4. InputManager

Drag and drop the InputManager.prefab into your scene

5. DefaultCursor

Drag and drop the DefaultCursor.prefab into your scene. In the InputManager’s Inspector Settings, drag and drop the DefaultCursor into the Cursor field of the SimpleSinglePointerSelector script

d. Save the scene

We are going to use Unity’s PhotoCapture API for the same. As the page says, WebCam and Microphone capabilities should be enabled in the project. So under Player Settings->Capabilities-> check Microphone and WebCam.


 Now we will write a script that will take a picture.  We are going to use Unity's PhotoCapture API for the same. As the page says, WebCam and Microphone capabilities should be enabled in the project. So under Player Settings-Capabilities- check Microphone and WebCam. 

Now we will write a script that will take a picture

The PhotoCapture API already shows how to take a picture and display it on a Unity GameObject. We will first create a Quad in the scene to render the image onto.

Adjust this Quad so that the MRCamera can see it.

Let’s create an empty GameObject and call it PhotoCaptureManager. To this we will add a script and call it TakePhoto.

 The PhotoCapture API already shows how to take a picture and display it on a Unity GameObject. We will first create a Quad in the scene to render the image onto.  Adjust this Quad so that the MRCamera can see it.  Let's create an empty GameObject and call it PhotoCaptureManager. To this we will add a script and call it TakePhoto. In the script copy the code from the PhotoCapture API

In the script copy the code from the PhotoCapture API




In this example, we will take a photo when the user says “Click”. So let’s configure the speech commands for that via Speech Input Manager and Speech Input Source . I won’t be explaining this in detail here. Please visit my Speech tutorial for the same.

Create an empty GameObject and call it SpeechManager or something similar. Before we proceed, let’s organize the scene a bit. Create an empty GameObject and call it Managers. Move the InputManager, PhotoCaptureManager and the SpeechManager as child objects below Managers.


 In this example, we will take a photo when the user says Click. So let's configure the speech commands for that via Speech Input Manager and Speech Input Source . I won't be explaining this in detail here. Please visit my Speech tutorial for the same.  Create an empty GameObject and call it SpeechManager or something similar. Before we proceed, let's organize the scene a bit. Create an empty GameObject and call it Managers. Move the InputManager, PhotoCaptureManager and the SpeechManager as child objects below Managers. 

Add the SpeechInputSource and SpeechInputHandler to the SpeechManager.


 Add the SpeechInputSource and SpeechInputHandler to the SpeechManager. Going back to our TakePhoto script, let's pack the code of the photo stuff in a function. I call this function Click() I also create a public variable where I will reference the Quad object in our scene to which the taken video would be rendered onto. Make those code changes. The new script code is shown below.

Going back to our TakePhoto script, let’s pack the code of the photo stuff in a function. I call this function Click()

I also create a public variable where I will reference the Quad object in our scene to which the taken video would be rendered onto. Make those code changes. The new script code is shown below.


 Do also remove the line where you set the Shader of the quad. We don't need that because the line, quadRenderer.material.SetTexture(_MainTex, targetTexture); will already do the trick of setting the Texture2D of the captured picture on the Quad. 

Do also remove the line where you set the Shader of the quad. We don’t need that because the line,

quadRenderer.material.SetTexture(“_MainTex”, targetTexture); 

The above line will already do the trick of setting the Texture2D of the captured picture on the Quad.



Now go back to the Unity scene and drag and drop your Quad into the Inspector field Quad of the TakePhoto script.


 Now go back to the Unity scene and drag and drop your Quad into the Inspector field Quad of the TakePhoto script. Go to the SpeechInputHandler and reference the Click() function on the PhotoCaptureManager Object.   

Go to the SpeechInputHandler and reference the Click() function on the PhotoCaptureManager Object.



Ok now let’s test this.

Build and deploy it onto the HoloLens. So by the way, the TextMeshPro problem has been resolved by Unity (in the version 3.12f1) in the meantime. So you don’t need to remove the Package now before building 🙂

Accept the popups which ask you to access the Microphone and the Camera of the HoloLens

Once you say Click, (tip: wait a second or two once you say this, else your picture will be distorted because the Camera is still taking the picture), you will see the picture rendered onto the Quad. Of course, you can provide some visual feedback like “Photo taken successfully” etc via 3DText or so.

Moving on to part 2: How to record a video from the app…

Let’s write another script for that. Luckily for us, Unity has provided the HoloLens VideoCapture API- so it got easier for us.

Go to Managers and add an empty GameObject. Call it VideoCaptureManager or something similar.

Add a new script to the VideoCaptureManager via the Add Component and call it TakeVideo or something similar.

The code already comes with the StartVideoCaptureTest() function. We just remove the Start() function and make the StartVideoCaptureTest() public.

Add a speech command to the SpeechInputSource script and call it Record or something similar.

Add the command Record in the SpeechInputHandler script and refer the VideoCaptureManager object and the corresponding function StartVideoCaptureTest() from the TakeVideo script.


 Open the script in Visual Studio and add the contents of the VideoCapture API onto it.  The code already comes with the StartVideoCaptureTest() function. We just remove the Start() function and make the StartVideoCaptureTest() public.  Add a speech command to the SpeechInputSource script and call it Record or something similar. Add the command Record in the SpeechInputHandler script and refer the VideoCaptureManager object and the corresponding function StartVideoCaptureTest() from the TakeVideo script. 

Open the script in Visual Studio and add the contents of the VideoCapture API onto it.

If you look at the API code, in the OnStartedVideoCaptureMode() function, it stores the video as TestVideo_{0}.mp4 in the persistentdataPath, which will be on the HoloLens in a path like this: C:/Data/Users/nisch/AppData/Local/Packages/Template3D_pzq3xp76mxafg/LocalState\TestVideo_9163378.mp4

But let’s take the recorded video and play it on the Quad.

Make the following code changes.

First add a public variable GameObject which will reference our quad. Go back to the script in the Inspector and reference our Quad object in the Inspector field quad of the TakeVideo script.

We also need a VideoPlayer object. In the Start() function, initialize this.

It will throw an error for the VideoPlayer. So add the using directive,

using UnityEngine.Video;

Make the filepath also a public global variable, we will use it later.

Write a separate function called PlayVideo() or something similar. First the quad needs a VideoPlayer component and an AudioSource component. Add that in the code.

We need to copy the movie from the filepath and copy this to our Quad’s Videoplayer.

Also make it loop (optional)

Don’t forget to attach an AudioSource to play the audio.

The PlayVideo() function code is shown below. If you need to see how we play videos with VideoPlayer, see my full tutorial here

 If you look at the API code, in the OnStartedVideoCaptureMode() function, it stores the video as TestVideo_{0}.mp4 in the persistentdataPath, which will be on the HoloLens in a path like this: C:/Data/Users/nisch/AppData/Local/Packages/Template3D_pzq3xp76mxafg/LocalState\TestVideo_9163378.mp4 But let's take the recorded video and play it on the Quad. Make the following code changes. First add a public variable GameObject which will reference our quad. Go back to the script in the Inspector and reference our Quad object in the Inspector field quad of the TakeVideo script. We also need a VideoPlayer object. In the Start() function, initialize this. It will throw an error for the VideoPlayer. So add the using directive, using UnityEngine.Video; Make the filepath also a public global variable, we will use it later.  Write a separate function called PlayVideo() or something similar. First the quad needs a VideoPlayer component and an AudioSource component. Add that in the code. We need to copy the movie from the filepath and copy this to our Quad's Videoplayer. Also make it loop (optional) Don't forget to attach an AudioSource to play the audio.  The PlayVideo() function code is shown below. See my full tutorial on how to play videos in the HoloLens apps.  Go back to the SpeechInputSource and add a keyword for Play We will call PlayVideo() from the TakeVideo script via the SpeechInputHandler  Let's test this. Build and deploy again onto the HoloLens.  Once you say Record, the video is recording for 5 seconds (you can change the time on top with the variable MaxRecordingTime) and then stored onto the persistentDataPath. Then say Play to see the video playing on your Quad. Of course adding visual feedback will help you figure out when you say Play() so that you don't get any errors even before it has finished recording. 

Go back to the SpeechInputSource and add a keyword for “Play”

We will call PlayVideo() from the TakeVideo script via the SpeechInputHandler

Let’s test this. Build and deploy again onto the HoloLens.

Once you say Record, the video is recording for 5 seconds (you can change the time on top with the variable MaxRecordingTime) and then stored onto the persistentDataPath. Then say Play to see the video playing on your Quad. Of course adding visual feedback will help you figure out when you say Play() so that you don’t get any errors even before it has finished recording.

Go to GitHub for my full project.

I cannot post a video output here because the camera can only be used either for taking pictures/videos or recording the live streaming of the app. 

Errors and solutions

Error: Null reference object while displaying picture

Solution: Make sure the Quad is referenced in the Inspector field of the TakePhoto script

Error: Null reference object while playing video

Solution: Make sure the Quad is referenced in the Inspector field of the TakeVideo script

Error: Speech not working

Solution: Microphone capability is not on

Keywords have not been set or functions have not been set for keywords

Error: Taking photos or videos is not working

Solution: WebCam capability is not on

Keywords have not been set or functions have not been set for keywords

Error: Picture looks distorted

Solution: Wait a second or two once you say Click, else your picture will be distorted because the Camera is still taking the picture. So don’t move until a few seconds or provide visual feedback to know when to move again 🙂

Error: Video not playing

Solution: Play is called before record is finished. Provide visual feedback via 3DText or so on the HoloLens view so that you know when to say play.

10 response to "Take photos and videos from within a HoloLens application"

  1. By: Dat Posted: April 29, 2019

    Nice tut Nischita 😀 I did try with this unity code earlier but did not success. I do remember its because the TMPro problem. So for someone who use earlier Unity version, which could not use this code you provide. I suggest to use this repo , which use the MediaCapture class from microsoft. You can take Photo with this but Recording Video is not available.

    https://github.com/MSAlshair/HoloLensMediaCapture/blob/master/README.md

    btw Nitischa, have you do experience with the topic video call from hololens with pc before ?

    • By: Nischita Posted: May 2, 2019

      Hi, This would still work with earlier Unity versions. Simply remove the TextMeshPro package from the Unity project and then build and deploy it onto the HoloLens. With respect to the video call from HoloLens and PC do you mean, you are calling HoloLens with a PC ? If yes, then both WebRTC and VNC would work. Here are some examples: https://github.com/Microsoft/WebRTC-universal-samples

  2. By: Dat Posted: May 2, 2019

    Thanks for the answer, well I do not know about VNC til now. Probably will check that later. However I do succesfully bring WebRTC to work on my HoloLens App. Come to this point I would use this recording features to record the call. However I got some error and I believe its because of the saving process to the camera directory.
    I did test your work on github and it works well but when I try to save the video content to camera rolls, the app crashed. Could you offer a solution to save the video to the camera roll place ?

  3. By: Dat Posted: May 14, 2019

    Today I tried the solution of patrick, However I got alot exceptions because I cant move the file from persistencedatapath to picturelibrary.
    I am not able to resolve that problem yet.
    Even though if I am able to move the file , I think I would face the problem recording the video at the same time when having a call. There might be some access violation because the camera already activated.

    • By: Nischita Posted: May 17, 2019

      Hi Dat, Thanks for getting back and letting us know the results of your trial. Recording video at the same time of having a call is not possible because the camera can be used at a time only for one thing. At least with HoloLens 1. I don’t know if this has changed with HoloLens2 because I face the same issue when I want to use it for 2 simultaneous purposes. I will try this out at some point to save recordings onto Picture Library on the HoloLens.

  4. By: Oluwafemi Posted: June 19, 2019

    Hello,
    When I try to build using the .Net backend, it displays an error that the currently selected scripting backend is not installed.

    I have both 3.5 and 4.5 .Net installed, please help

    • By: Nischita Posted: June 20, 2019

      Hi, which Unity version are you using?

  5. By: Oluwafemi Posted: June 20, 2019

    Unity 2018.2.11f1

    • By: Nischita Posted: June 21, 2019

      Could you upgrade to 2018.3.12f and try? some things might break or are differently handled in the older versions.

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: