Take photos and videos from within a HoloLens application

Hey there! I have been talking about dynamically taking pictures/videos from the HoloLens in my newsletters for quite some time now. If you have needed an app where you could trigger taking a photo or recording videos dynamically, then this tutorial is for you.

Let’s start with part 1: Taking pictures from an app via voice input

The usual things to do:

1. Create a new project. I have Unity version 2018.3.12f1

2. I’m using the latest stable release of the HoloToolkit 2017.4.3.0 Import it into your project

3. Do the usual tweaks:

a. Delete the main Camera. Add the MixedRealityCamera.prefab in the Hierarchy. This will serve as the Main Camera

b. Change the Camera Settings to Hololens preferred:

Skybox to Solid Color

Background to 0,0,0,0

In the MixedRealityCameraManager script which is attached to the MixedRealityCamera object, change the Clear Flags value to Solid Color

Ensure that the Background Color is black

c. Go to Edit ->Project Settings -> Player -> Other Settings -> and change the Scripting Runtime version to .NET 4.x Equivalent and Scripting Backend to .Net. Ensure that UWP is selected as the Build Platform.

Make sure also that in XR Settings, Virtual Reality Supported is checked

We need both the InputManager and the DefaultCursor in the scene.

4. InputManager

Drag and drop the InputManager.prefab into your scene

5. DefaultCursor

Drag and drop the DefaultCursor.prefab into your scene. In the InputManager’s Inspector Settings, drag and drop the DefaultCursor into the Cursor field of the SimpleSinglePointerSelector script

d. Save the scene

We are going to use Unity’s PhotoCapture API for the same. As the page says, WebCam and Microphone capabilities should be enabled in the project. So under Player Settings->Capabilities-> check Microphone and WebCam.

Now we will write a script that will take a picture

The PhotoCapture API already shows how to take a picture and display it on a Unity GameObject. We will first create a Quad in the scene to render the image onto.

Adjust this Quad so that the MRCamera can see it.

Let’s create an empty GameObject and call it PhotoCaptureManager. To this we will add a script and call it TakePhoto.

In the script copy the code from the PhotoCapture API

In this example, we will take a photo when the user says “Click”. So let’s configure the speech commands for that via Speech Input Manager and Speech Input Source . I won’t be explaining this in detail here. Please visit my Speech tutorial for the same.

Create an empty GameObject and call it SpeechManager or something similar. Before we proceed, let’s organize the scene a bit. Create an empty GameObject and call it Managers. Move the InputManager, PhotoCaptureManager and the SpeechManager as child objects below Managers.

Add the SpeechInputSource and SpeechInputHandler to the SpeechManager.

Going back to our TakePhoto script, let’s pack the code of the photo stuff in a function. I call this function Click()

I also create a public variable where I will reference the Quad object in our scene to which the taken video would be rendered onto. Make those code changes. The new script code is shown below.

Do also remove the line where you set the Shader of the quad. We don’t need that because the line,

quadRenderer.material.SetTexture(“_MainTex”, targetTexture); 

The above line will already do the trick of setting the Texture2D of the captured picture on the Quad.

Now go back to the Unity scene and drag and drop your Quad into the Inspector field Quad of the TakePhoto script.

Go to the SpeechInputHandler and reference the Click() function on the PhotoCaptureManager Object.

Ok now let’s test this.

Build and deploy it onto the HoloLens. So by the way, the TextMeshPro problem has been resolved by Unity (in the version 3.12f1) in the meantime. So you don’t need to remove the Package now before building 🙂

Accept the popups which ask you to access the Microphone and the Camera of the HoloLens

Once you say Click, (tip: wait a second or two once you say this, else your picture will be distorted because the Camera is still taking the picture), you will see the picture rendered onto the Quad. Of course, you can provide some visual feedback like “Photo taken successfully” etc via 3DText or so.

Moving on to part 2: How to record a video from the app…

Let’s write another script for that. Luckily for us, Unity has provided the HoloLens VideoCapture API- so it got easier for us.

Go to Managers and add an empty GameObject. Call it VideoCaptureManager or something similar.

Add a new script to the VideoCaptureManager via the Add Component and call it TakeVideo or something similar.

The code already comes with the StartVideoCaptureTest() function. We just remove the Start() function and make the StartVideoCaptureTest() public.

Add a speech command to the SpeechInputSource script and call it Record or something similar.

Add the command Record in the SpeechInputHandler script and refer the VideoCaptureManager object and the corresponding function StartVideoCaptureTest() from the TakeVideo script.

Open the script in Visual Studio and add the contents of the VideoCapture API onto it.

If you look at the API code, in the OnStartedVideoCaptureMode() function, it stores the video as TestVideo_{0}.mp4 in the persistentdataPath, which will be on the HoloLens in a path like this: C:/Data/Users/nisch/AppData/Local/Packages/Template3D_pzq3xp76mxafg/LocalState\TestVideo_9163378.mp4

But let’s take the recorded video and play it on the Quad.

Make the following code changes.

First add a public variable GameObject which will reference our quad. Go back to the script in the Inspector and reference our Quad object in the Inspector field quad of the TakeVideo script.

We also need a VideoPlayer object. In the Start() function, initialize this.

It will throw an error for the VideoPlayer. So add the using directive,

using UnityEngine.Video;

Make the filepath also a public global variable, we will use it later.

Write a separate function called PlayVideo() or something similar. First the quad needs a VideoPlayer component and an AudioSource component. Add that in the code.

We need to copy the movie from the filepath and copy this to our Quad’s Videoplayer.

Also make it loop (optional)

Don’t forget to attach an AudioSource to play the audio.

The PlayVideo() function code is shown below. If you need to see how we play videos with VideoPlayer, see my full tutorial here

Go back to the SpeechInputSource and add a keyword for “Play”

We will call PlayVideo() from the TakeVideo script via the SpeechInputHandler

Let’s test this. Build and deploy again onto the HoloLens.

Once you say Record, the video is recording for 5 seconds (you can change the time on top with the variable MaxRecordingTime) and then stored onto the persistentDataPath. Then say Play to see the video playing on your Quad. Of course adding visual feedback will help you figure out when you say Play() so that you don’t get any errors even before it has finished recording.

Go to GitHub for my full project.

I cannot post a video output here because the camera can only be used either for taking pictures/videos or recording the live streaming of the app. 

Errors and solutions

Error: Null reference object while displaying picture

Solution: Make sure the Quad is referenced in the Inspector field of the TakePhoto script

Error: Null reference object while playing video

Solution: Make sure the Quad is referenced in the Inspector field of the TakeVideo script

Error: Speech not working

Solution: Microphone capability is not on

Keywords have not been set or functions have not been set for keywords

Error: Taking photos or videos is not working

Solution: WebCam capability is not on

Keywords have not been set or functions have not been set for keywords

Error: Picture looks distorted

Solution: Wait a second or two once you say Click, else your picture will be distorted because the Camera is still taking the picture. So don’t move until a few seconds or provide visual feedback to know when to move again 🙂

Error: Video not playing

Solution: Play is called before record is finished. Provide visual feedback via 3DText or so on the HoloLens view so that you know when to say play.