QRCode tracking with Hololens 2, XR SDK and MRTK V2.5

Hello everybody! This article has been long overdue and most requested as well. I thought it would be also a good opportunity to combine this with how to use the latest XR plugin that replaces the Legacy XR with the advent of Unity 2020.x versions.

In this article, we will see a tutorial on how to do QR code scanning and tracking with the new XR SDK and MRTK v2.5.x and of course, Unity 2020.x version. If you have seen my articles before, I always use the penultimate release of MRTK, simply because the latest one will not yet be stable and could be buggy. Therefore, the MRTK v2.5.

Let’s get started!

1. Get Unity 2020.x version. I have specifically used, 2020.2.1f1, also because its not too recent and higher likelihood of it being more compatible with the MRTK and the code that we write as well. AFAIK, the QRcode tracking code that I have used to refer was tested successfully on Unity 2020.x versions. Feel free to use latest versions and let me also know if that works with this code here ūüôā

Create a new 3D project as usual.

2. Get MRTK v2.5: Specifically, I have used MRTK v2.5.4, we can get this with the new Mixed Reality Feature Tool which was out early March, this year. The link to download is here: https://www.microsoft.com/en-us/download/details.aspx?id=102778 Once you have downloaded, open the MixedRealityFeatureTool.exe and you will see a screen


 If you want to configure it in a specific way, see here. Else, just click on Start. In the next screen, provide the location of your project path

If you want to configure it in a specific way, see here. Else, just click on Start.

In the next screen, provide the location of your project path


 It should automatically pick up your project version as well.  Then click on Discover Features. It should show you standard packages which we may need for MRTK. For this project, we would need:  Mixed Reality Toolkit Foundation v2.5.4 If there are any dependencies we need, the tool will automatically recommend the needed packages. In this case, it needs the Standard Assets package as a dependancy. 

It should automatically pick up your project version as well.

Then click on Discover Features. It should show you standard packages which we may need for MRTK. For this project, we would need:

Mixed Reality Toolkit Foundation v2.5.4

If there are any dependencies we need, the tool will automatically recommend the needed packages. In this case, it needs the Standard Assets package as a dependency.


 You could here click on Import or Validate. Although the recommendation by MRTK guys is to click Validate- so that we can see if anything went wrong in the pop up dialog later. When no issues, click on Import.

You could here click on Import or Validate. Although the recommendation by MRTK guys is to click Validate– so that we can see if anything went wrong in the pop up dialog later. When no issues, click on Import.


 The next screen will be a Review and Approve. Click on Approve.  The final screen will be that the project has been successfuly updated. If you go to your project now in Unity, it will refresh and load all those packages. 

The next screen will be a Review and Approve. Click on Approve.

The final screen will be that the project has been successfully updated. If you go to your project now in Unity, it will refresh and load all those packages.



3. Now for the XR SDK: Go to Player-Project Settings and under XR Plugin Management, click on Install XR Plugin Management. This will take a few minutes and install the XR SDKs into your project


 3. Now for the XR SDK: Go to Player-Project Settings and under XR Plugin Management, click on Install XR Plugin Management. This will take a few minutes and install the XR SDKs into your project

Switch to the Windows Platform and enable the Windows Mixed Reality option. This again takes a few minutes to install the WindowsMR plugin into your project. Also make sure that Initialize XR on Startup is enabled.


 Switch to the Windows Platform and enable the Windows Mixed Reality option. This again takes a few minutes to install the WindowsMR plugin into your project. Also make sure that Initialize XR on Startup is enabled. 

4. Click on Mixed Reality Toolkit at the top and click on Add to Scene and Configure. This will add the Mixed Reality essentials to the project such as the Playspace and the Toolkit. We need to configure the MRTK to run with XR SDK. For this, choose DefaultXRSDKConfigurationProfile as the active profile.

Also clone the SpatialAwarenessProfile XR SDKWindowsMixedRealitySpatialMeshObserver and set the Display SettingsDisplay Option to None.

Ok, So we are now done with setup.

For the QRCode part:

Let’s make a simple UI panel which can start and stop QR code tracking with the help of a button. I won’t cover the UI panel in this tutorial. You can use the Pressable Button prefab that MRTK provides and make your own. I also have the latest QRCode scanned data shown in this panel- so that we get visual feedback directly within the app.

For the QRCode part, here’s where I started off with it:

https://docs.microsoft.com/en-us/windows/mixed-reality/develop/platform-capabilities-and-apis/qr-code-tracking

I also looked at the Sample QR Code app here:

https://github.com/chgatla-microsoft/QRTracking which was really helpful in showing how this is done. It also provides some prefabs and scripts which I reused such as QRCodesManager, QRCode.cs, QRCodesManafer.cs, QRCodesVisualizer.cs, etc. Whatever I have reused, I put them in a folder called QRCodeStuff in the project.

First, we need the Microsoft.MixedReality.QR package. This can be installed via Nuget for Unity. I downloaded Release 3.0.1 which is latest at this time. Once you have downloaded it into your PC, go to Assets-Import Package-Custom Package and navigate to where you saved Nuget package and import into your project.


 First, we need the  Microsoft.MixedReality.QR  package. This can be installed via Nuget for Unity. I downloaded Release 3.0.1 which is latest at this time. Once you have downloaded it into your PC, go to Assets-Import Package-Custom Package and navigate to where you saved Nuget package and import into your project.

Then you will see a menu item called Nuget when it has been successfully imported. Click on Manage Nuget Packages and search for Microsoft.MixedReality.QR package and install it.


 Then you will see a menu item called Nuget when it has been successfully imported. Click on Manage Nuget Packages and search for   Microsoft.MixedReality.QR package and install it.  

Once successfully installed, you will see the package in Assets-Packages


 Once successfully installed, you will see the package in Assets-Packages

So now in your scene, drag and drop the prefabs QRCodePanel and QRCodesManager. Note here that you need all those prefabs, scripts and materials which I have in the QRCodeStuff folder. Else, it breaks.

If you get a popup to Import TMP essentials, do it- as this is needed for the text items on QRCodePanel

Now in the QRCodesManager Inspector view, drag and drop the QRCode prefab into QRCodePrefab of QRCodesVisualizer

Drag and drop the LatestQRCodeDetails into the LatestQRCodeDetails field of QRCodesVisualizer

Make sure AutoStartQRTracking is disabled as we want to trigger the scan and stop it via our own script. I will come to this in a bit…


 So now in your scene, drag and drop the prefabs QRCodePanel and QRCodesManager. Note here that you need all those prefabs, scripts and materials which I have in the QRCodeStuff folder. Else, it breaks.  If you get a popup to Import TMP essentials, do it- as this is needed for the text items on QRCodePanel Now in the QRCodesManager Inspector view, drag and drop the QRCode prefab into QRCodePrefab of QRCodesVisualizer Drag and drop the LatestQRCodeDetails into the LatestQRCodeDetails field of QRCodesVisualizer Make sure AutoStartQRTracking is disabled as we want to trigger the scan and stop it via our own script. I will come to this in a bit...

To explain a few prefabs and scripts(these were used from the sample project indicated earlier as well)

QRCode: This is a prefab with a transparent Quad and an axis on the top left corner of the Quad to show the coordinates and the QRcode area whenever a new QRCode is scanned or updated. This is especially useful, when you want to perform tracking of the object with a QRCode on it.

QRCode itself has two scripts on it:

QRCode.cs: It shows what goes into a QRCode such as its unique ID, data, text, version, timestamp etc. It also has a NodeID which is used for the coordinate identification part. Therefore, this script requires the SpatialGraphCoordinateSystem.cs which does all of these coordinate calculations.


 To explain a few prefabs and scripts(these were used from the sample project indicated earlier as well) QRCode: This is a prefab with a transparent Quad and an axis on the top left corner of the Quad to show the coordinates and the QRcode area whenever a new QRCode is scanned or updated. This is especially useful, when you want to perform tracking of the object with a QRCode on it. QRCode itself has two scripts on it: QRCode.cs: It shows what goes into a QRCode such as its unique ID, data, text, version, timestamp etc. It also has a NodeID which is used for the coordinate identification part. Therefore, this script requires the SpatialGraphCoordinateSystem.cs which does all of these coordinate calculations.

The QRCodesManager script comes with its own StartQRTracking() and StopQRTracking() functions. We can call these directly when we press the StartScanButton and StopScanButton respectively. But to keep it neater, I will just create a new script which does that for us. I also will update the StatusText to inform us that QRCode scanning has started and stopped accordingly. Let’s call this script MyQRCodeManager.cs

It will be a simple script with two functions used to start and stop scanning and to change the StatusText.

The code is shown below.


 The QRCodesManager script comes with its own StartQRTracking() and StopQRTracking() functions. We can call these directly when we press the StartScanButton and StopScanButton respectively. But to keep it neater, I will just create a new script which does that for us. I also will update the StatusText to inform us that QRCode scanning has started and stopped accordingly. Let's call this script MyQRCodeManager.cs It will be a simple script with two functions used to start and stop scanning and to change the StatusText.  The code is shown below.

Add an empty GameObject to the scene and call it Manager. To it add the MyQRCodeManager script.

It has two public variables qRCodesManager and statusText which we will assign to the QRCodesManager prefab which contains the QRCodesManager script and the statusText in our QRCodePanel


 Add an empty GameObject to the scene and call it Manager. To it add the MyQRCodeManager script. It has two public variables qRCodesManager and statusText which we will assign to the QRCodesManager prefab which contains the QRCodesManager script and the statusText in our QRCodePanel

Let’s have a look at QRCodesManager script again.

For your ease of understanding, I have added in comments to explain the code better.

Looking at the Start() method, we first need to check if QRCode scanning is supported on the device that we use. Thankfully for us, this is in the QRCodeWatcher script with IsSupported(). Then we need to get a reference to the QRCodeWatcher class and call the RequestAccessAsync() on it which gets us the permission to use the webcam.

Once these are checked and there are no errors, we set the accessStatus and capabilityInitialized to true. These will be used later.


 Let's have a look at QRCodesManager script again.  For your ease of understanding, I have added in comments to explkain the code better.  Looking at the Start() method, we first need to check if QRCode scanning is supported on the device that we use. Thankfully for us, this is in the QRCodeWatcher script with IsSupported(). Then we need to get a reference to the QRCodeWatcher class and call the RequestAccessAsync() on it which gets us the permission to use the webcam.  Once these are checked and there are no errors, we set the accessStatus and capabilityInitialized to true. These will be used later. 

The next function to look at is the SetupQRTracking():

Here, we will initialize to check if the tracker is already running via a variable IsTrackerRunning.

Next, we will subscribe to events to alert us whenever a QRCode is added, updated, removed and enumeration completed. Enumeration completed is used to maintain a list of QR Codes which we add in the same session.


 The next function to look at is the SetupQRTracking(): Here, we will initialize to check if the tracker is already running via a variable IsTrackerRunning.  Next, we will subscribe to events to alert us whenever a QRCode is added, updated, removed and enumeration completed. Enumeration completed is used to maintain a list of QR Codes which we add in the same session.

So now, returning to MyQRCodesManager, let’s assign the start and stop functions to the button pressing accordingly.

We want to be able to do three things when we press the StartScanButton:

1. Call the StartScan() function on MyQRCodeManager

2. Don’t show the StartScanButton

3. Show the StopScanButton

Do all of these accordingly with the OnClick() event on the button


 So now, returning to MyQRCodesManager, let's assign the start and stop functions to the button pressing accordingly. We want to be able to do three things when we press the StartScanButton: 1. Call the StartScan() function on MyQRCodeManager 2. Don't show the StartScanButton 3. Show the StopScanButton Do all of these accordingly with the OnClick() event on the button

We also need to do three things when we press the StopScanButton:

1. Call the StopScan() function in MyQRCodeManager

2. Show the StartScanButton

3. Don’t show the StopScanButton

Now if we look at the flow of our code:

when we press the StartScanButton, the StartScan() function in MyQRCodeManager calls the StartQRTracking() function on QRCodesManager.

Simultaneously, the QRCodesVisualizer script sets up its dictionary to maintain QRCodes. It also subscribes to the events on adding, updating, removing, so that the visualizer can act accordingly.

You will notice in the Update() method, it calls HandleEvents() which will based on the corresponding event- such as updating of QRcode, removing or adding will handle the visualization and data change accordingly.

There are more comments within the scripts to explain what the code does.


 We also need to do three things when we press the StopScanButton: 1. Call the StopScan() function in MyQRCodeManager 2. Show the StartScanButton 3. Don't show the StopScanButton Now if we look at the flow of our code, when we press the StartScanButton, the StartScan() function in MyQRCodeManager calls the StartQRTracking() function on QRCodesManager.  Simultaneously, the QRCodesVisualizer script sets up its dictionary to maintain QRCodes. It also subscribes to the events on adding, updating, removing, so that the visualizer can act accordingly.  You will notice in the Update() method, it calls HandleEvents() which will based on the corresponding event- such as updating of QRcode, removing or adding will handle the visualization and data change accordingly. There are more comments within the scripts to explain what the code does.

In line 112 of QRCodesVisualizer, we also will update our LatestQRCodeDetails text to show the data of the most recent QRCode scanned.

If you do have any more questions about what the code does in these scripts, do let me know.


 In line 112 of QRCodesVisualizer, we also will update our LatestQRCodeDetails text to show the data of the most recent QRCode scanned.  If you do have any more questions about what the code does in these scripts, do contact me!

Don’t forget to add following capabilities to detect QR Codes and visualize them correctly

Webcam

InternetClient

SpatialPerception

GazeInput

Microphone(optional – if you use speech commands to control the starting and stopping of scanning)

Ok, so now that we have all setup, let’s build and try it out!

I also noticed that XR SDK builds are much faster! I don’t know if this was a feature of Unity 2020.x or the XR plugin.

So, I tried 3 different QR codes with different sizes and resolution. One with 300 px and 7x7cm size, one with 200 px and 4x4cm size and one with 300 px and 2x2cm size

1. The bigger and closer you are to the QRCode, the faster it gets recognized. 2x2cm does not get recognized.

2. Gets detected in good lighting conditions, I tried in twilight time and it was working well even with little light

3. This point is specifically important on how to manage QR codes from the Microsoft page here:

Managing QR code data

“Windows Mixed Reality devices detect QR codes at the system level in the driver. When the device is rebooted, the detected QR codes are gone and will be redetected as new objects next time.

We recommend configuring your app to ignore QR codes older than a specific timestamp. Currently, the API doesn’t support clearing QR code history.”

You can see the detailed best practices for QRCode tracking with Hololens2 here

I also noticed that the tracking is slower, so it takes a few seconds to get adjusted to the new positions of the QRCode. But this is not a deal breaker ūüôā

The project code is in GitHub here: https://github.com/NSudharsan/HoloLensExamples/tree/master/QRCodeXRSDK

Below is the output video:

I’m not sure why the video output shows that the QRCode displacement by a few cms. In the Hololens View, its perfectly on the QRCode.

By the way, I have tried this code with Legacy XR and it also works. The only change I had to make was in SpatialCoordinateSystem.cs. Comment out the XR related lines on 75 and 76 and add line 77,78. Thanks to Edd Smith from the HoloDevelopers Slack Community who gave me this solution!

 By the way, I have tried this code with Legacy XR and it also works. The only change I had to make was in SpatialCoordinateSystem.cs. Comment out the XR related lines on 75 and 76 and add line 77,78 Thanks to Edd Smith from the HoloDevelopers Slack Community who gave me this solution!

 

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: