QRCode tracking with Hololens 2, XR SDK and MRTK V2.5

Hello everybody! This article has been long overdue and most requested as well. I thought it would be also a good opportunity to combine this with how to use the latest XR plugin that replaces the Legacy XR with the advent of Unity 2020.x versions.

In this article, we will see a tutorial on how to do QR code scanning and tracking with the new XR SDK and MRTK v2.5.x and of course, Unity 2020.x version. If you have seen my articles before, I always use the penultimate release of MRTK, simply because the latest one will not yet be stable and could be buggy. Therefore, the MRTK v2.5.

Let’s get started!

1. Get Unity 2020.x version. I have specifically used, 2020.2.1f1, also because its not too recent and higher likelihood of it being more compatible with the MRTK and the code that we write as well. AFAIK, the QRcode tracking code that I have used to refer was tested successfully on Unity 2020.x versions. Feel free to use latest versions and let me also know if that works with this code here 🙂

Create a new 3D project as usual.

2. Get MRTK v2.5: Specifically, I have used MRTK v2.5.4, we can get this with the new Mixed Reality Feature Tool which was out early March, this year. The link to download is here: https://www.microsoft.com/en-us/download/details.aspx?id=102778 Once you have downloaded, open the MixedRealityFeatureTool.exe and you will see a screen


 If you want to configure it in a specific way, see here. Else, just click on Start. In the next screen, provide the location of your project path

If you want to configure it in a specific way, see here. Else, just click on Start.

In the next screen, provide the location of your project path


 It should automatically pick up your project version as well.  Then click on Discover Features. It should show you standard packages which we may need for MRTK. For this project, we would need:  Mixed Reality Toolkit Foundation v2.5.4 If there are any dependencies we need, the tool will automatically recommend the needed packages. In this case, it needs the Standard Assets package as a dependancy. 

It should automatically pick up your project version as well.

Then click on Discover Features. It should show you standard packages which we may need for MRTK. For this project, we would need:

Mixed Reality Toolkit Foundation v2.5.4

If there are any dependencies we need, the tool will automatically recommend the needed packages. In this case, it needs the Standard Assets package as a dependency.


 You could here click on Import or Validate. Although the recommendation by MRTK guys is to click Validate- so that we can see if anything went wrong in the pop up dialog later. When no issues, click on Import.

You could here click on Import or Validate. Although the recommendation by MRTK guys is to click Validate– so that we can see if anything went wrong in the pop up dialog later. When no issues, click on Import.


 The next screen will be a Review and Approve. Click on Approve.  The final screen will be that the project has been successfuly updated. If you go to your project now in Unity, it will refresh and load all those packages. 

The next screen will be a Review and Approve. Click on Approve.

The final screen will be that the project has been successfully updated. If you go to your project now in Unity, it will refresh and load all those packages.



3. Now for the XR SDK: Go to Player-Project Settings and under XR Plugin Management, click on Install XR Plugin Management. This will take a few minutes and install the XR SDKs into your project


 3. Now for the XR SDK: Go to Player-Project Settings and under XR Plugin Management, click on Install XR Plugin Management. This will take a few minutes and install the XR SDKs into your project

Switch to the Windows Platform and enable the Windows Mixed Reality option. This again takes a few minutes to install the WindowsMR plugin into your project. Also make sure that Initialize XR on Startup is enabled.


 Switch to the Windows Platform and enable the Windows Mixed Reality option. This again takes a few minutes to install the WindowsMR plugin into your project. Also make sure that Initialize XR on Startup is enabled. 

4. Click on Mixed Reality Toolkit at the top and click on Add to Scene and Configure. This will add the Mixed Reality essentials to the project such as the Playspace and the Toolkit. We need to configure the MRTK to run with XR SDK. For this, choose DefaultXRSDKConfigurationProfile as the active profile.

Also clone the SpatialAwarenessProfile XR SDKWindowsMixedRealitySpatialMeshObserver and set the Display SettingsDisplay Option to None.

Ok, So we are now done with setup.

For the QRCode part:

Let’s make a simple UI panel which can start and stop QR code tracking with the help of a button. I won’t cover the UI panel in this tutorial. You can use the Pressable Button prefab that MRTK provides and make your own. I also have the latest QRCode scanned data shown in this panel- so that we get visual feedback directly within the app.

For the QRCode part, here’s where I started off with it:

https://docs.microsoft.com/en-us/windows/mixed-reality/develop/platform-capabilities-and-apis/qr-code-tracking

I also looked at the Sample QR Code app here:

https://github.com/chgatla-microsoft/QRTracking which was really helpful in showing how this is done. It also provides some prefabs and scripts which I reused such as QRCodesManager, QRCode.cs, QRCodesManafer.cs, QRCodesVisualizer.cs, etc. Whatever I have reused, I put them in a folder called QRCodeStuff in the project.

First, we need the Microsoft.MixedReality.QR package. This can be installed via Nuget for Unity. I downloaded Release 3.0.1 which is latest at this time. Once you have downloaded it into your PC, go to Assets-Import Package-Custom Package and navigate to where you saved Nuget package and import into your project.


 First, we need the  Microsoft.MixedReality.QR  package. This can be installed via Nuget for Unity. I downloaded Release 3.0.1 which is latest at this time. Once you have downloaded it into your PC, go to Assets-Import Package-Custom Package and navigate to where you saved Nuget package and import into your project.

Then you will see a menu item called Nuget when it has been successfully imported. Click on Manage Nuget Packages and search for Microsoft.MixedReality.QR package and install it.


 Then you will see a menu item called Nuget when it has been successfully imported. Click on Manage Nuget Packages and search for   Microsoft.MixedReality.QR package and install it.  

Once successfully installed, you will see the package in Assets-Packages


 Once successfully installed, you will see the package in Assets-Packages

So now in your scene, drag and drop the prefabs QRCodePanel and QRCodesManager. Note here that you need all those prefabs, scripts and materials which I have in the QRCodeStuff folder. Else, it breaks.

If you get a popup to Import TMP essentials, do it- as this is needed for the text items on QRCodePanel

Now in the QRCodesManager Inspector view, drag and drop the QRCode prefab into QRCodePrefab of QRCodesVisualizer

Drag and drop the LatestQRCodeDetails into the LatestQRCodeDetails field of QRCodesVisualizer

Make sure AutoStartQRTracking is disabled as we want to trigger the scan and stop it via our own script. I will come to this in a bit…


 So now in your scene, drag and drop the prefabs QRCodePanel and QRCodesManager. Note here that you need all those prefabs, scripts and materials which I have in the QRCodeStuff folder. Else, it breaks.  If you get a popup to Import TMP essentials, do it- as this is needed for the text items on QRCodePanel Now in the QRCodesManager Inspector view, drag and drop the QRCode prefab into QRCodePrefab of QRCodesVisualizer Drag and drop the LatestQRCodeDetails into the LatestQRCodeDetails field of QRCodesVisualizer Make sure AutoStartQRTracking is disabled as we want to trigger the scan and stop it via our own script. I will come to this in a bit...

To explain a few prefabs and scripts(these were used from the sample project indicated earlier as well)

QRCode: This is a prefab with a transparent Quad and an axis on the top left corner of the Quad to show the coordinates and the QRcode area whenever a new QRCode is scanned or updated. This is especially useful, when you want to perform tracking of the object with a QRCode on it.

QRCode itself has two scripts on it:

QRCode.cs: It shows what goes into a QRCode such as its unique ID, data, text, version, timestamp etc. It also has a NodeID which is used for the coordinate identification part. Therefore, this script requires the SpatialGraphCoordinateSystem.cs which does all of these coordinate calculations.


 To explain a few prefabs and scripts(these were used from the sample project indicated earlier as well) QRCode: This is a prefab with a transparent Quad and an axis on the top left corner of the Quad to show the coordinates and the QRcode area whenever a new QRCode is scanned or updated. This is especially useful, when you want to perform tracking of the object with a QRCode on it. QRCode itself has two scripts on it: QRCode.cs: It shows what goes into a QRCode such as its unique ID, data, text, version, timestamp etc. It also has a NodeID which is used for the coordinate identification part. Therefore, this script requires the SpatialGraphCoordinateSystem.cs which does all of these coordinate calculations.

The QRCodesManager script comes with its own StartQRTracking() and StopQRTracking() functions. We can call these directly when we press the StartScanButton and StopScanButton respectively. But to keep it neater, I will just create a new script which does that for us. I also will update the StatusText to inform us that QRCode scanning has started and stopped accordingly. Let’s call this script MyQRCodeManager.cs

It will be a simple script with two functions used to start and stop scanning and to change the StatusText.

The code is shown below.


 The QRCodesManager script comes with its own StartQRTracking() and StopQRTracking() functions. We can call these directly when we press the StartScanButton and StopScanButton respectively. But to keep it neater, I will just create a new script which does that for us. I also will update the StatusText to inform us that QRCode scanning has started and stopped accordingly. Let's call this script MyQRCodeManager.cs It will be a simple script with two functions used to start and stop scanning and to change the StatusText.  The code is shown below.

Add an empty GameObject to the scene and call it Manager. To it add the MyQRCodeManager script.

It has two public variables qRCodesManager and statusText which we will assign to the QRCodesManager prefab which contains the QRCodesManager script and the statusText in our QRCodePanel


 Add an empty GameObject to the scene and call it Manager. To it add the MyQRCodeManager script. It has two public variables qRCodesManager and statusText which we will assign to the QRCodesManager prefab which contains the QRCodesManager script and the statusText in our QRCodePanel

Let’s have a look at QRCodesManager script again.

For your ease of understanding, I have added in comments to explain the code better.

Looking at the Start() method, we first need to check if QRCode scanning is supported on the device that we use. Thankfully for us, this is in the QRCodeWatcher script with IsSupported(). Then we need to get a reference to the QRCodeWatcher class and call the RequestAccessAsync() on it which gets us the permission to use the webcam.

Once these are checked and there are no errors, we set the accessStatus and capabilityInitialized to true. These will be used later.


 Let's have a look at QRCodesManager script again.  For your ease of understanding, I have added in comments to explkain the code better.  Looking at the Start() method, we first need to check if QRCode scanning is supported on the device that we use. Thankfully for us, this is in the QRCodeWatcher script with IsSupported(). Then we need to get a reference to the QRCodeWatcher class and call the RequestAccessAsync() on it which gets us the permission to use the webcam.  Once these are checked and there are no errors, we set the accessStatus and capabilityInitialized to true. These will be used later. 

The next function to look at is the SetupQRTracking():

Here, we will initialize to check if the tracker is already running via a variable IsTrackerRunning.

Next, we will subscribe to events to alert us whenever a QRCode is added, updated, removed and enumeration completed. Enumeration completed is used to maintain a list of QR Codes which we add in the same session.


 The next function to look at is the SetupQRTracking(): Here, we will initialize to check if the tracker is already running via a variable IsTrackerRunning.  Next, we will subscribe to events to alert us whenever a QRCode is added, updated, removed and enumeration completed. Enumeration completed is used to maintain a list of QR Codes which we add in the same session.

So now, returning to MyQRCodesManager, let’s assign the start and stop functions to the button pressing accordingly.

We want to be able to do three things when we press the StartScanButton:

1. Call the StartScan() function on MyQRCodeManager

2. Don’t show the StartScanButton

3. Show the StopScanButton

Do all of these accordingly with the OnClick() event on the button


 So now, returning to MyQRCodesManager, let's assign the start and stop functions to the button pressing accordingly. We want to be able to do three things when we press the StartScanButton: 1. Call the StartScan() function on MyQRCodeManager 2. Don't show the StartScanButton 3. Show the StopScanButton Do all of these accordingly with the OnClick() event on the button

We also need to do three things when we press the StopScanButton:

1. Call the StopScan() function in MyQRCodeManager

2. Show the StartScanButton

3. Don’t show the StopScanButton

Now if we look at the flow of our code:

when we press the StartScanButton, the StartScan() function in MyQRCodeManager calls the StartQRTracking() function on QRCodesManager.

Simultaneously, the QRCodesVisualizer script sets up its dictionary to maintain QRCodes. It also subscribes to the events on adding, updating, removing, so that the visualizer can act accordingly.

You will notice in the Update() method, it calls HandleEvents() which will based on the corresponding event- such as updating of QRcode, removing or adding will handle the visualization and data change accordingly.

There are more comments within the scripts to explain what the code does.


 We also need to do three things when we press the StopScanButton: 1. Call the StopScan() function in MyQRCodeManager 2. Show the StartScanButton 3. Don't show the StopScanButton Now if we look at the flow of our code, when we press the StartScanButton, the StartScan() function in MyQRCodeManager calls the StartQRTracking() function on QRCodesManager.  Simultaneously, the QRCodesVisualizer script sets up its dictionary to maintain QRCodes. It also subscribes to the events on adding, updating, removing, so that the visualizer can act accordingly.  You will notice in the Update() method, it calls HandleEvents() which will based on the corresponding event- such as updating of QRcode, removing or adding will handle the visualization and data change accordingly. There are more comments within the scripts to explain what the code does.

In line 112 of QRCodesVisualizer, we also will update our LatestQRCodeDetails text to show the data of the most recent QRCode scanned.

If you do have any more questions about what the code does in these scripts, do let me know.


 In line 112 of QRCodesVisualizer, we also will update our LatestQRCodeDetails text to show the data of the most recent QRCode scanned.  If you do have any more questions about what the code does in these scripts, do contact me!

Don’t forget to add following capabilities to detect QR Codes and visualize them correctly

Webcam

InternetClient

SpatialPerception

GazeInput

Microphone(optional – if you use speech commands to control the starting and stopping of scanning)

Ok, so now that we have all setup, let’s build and try it out!

I also noticed that XR SDK builds are much faster! I don’t know if this was a feature of Unity 2020.x or the XR plugin.

So, I tried 3 different QR codes with different sizes and resolution. One with 300 px and 7x7cm size, one with 200 px and 4x4cm size and one with 300 px and 2x2cm size

1. The bigger and closer you are to the QRCode, the faster it gets recognized. 2x2cm does not get recognized.

2. Gets detected in good lighting conditions, I tried in twilight time and it was working well even with little light

3. This point is specifically important on how to manage QR codes from the Microsoft page here:

Managing QR code data

“Windows Mixed Reality devices detect QR codes at the system level in the driver. When the device is rebooted, the detected QR codes are gone and will be redetected as new objects next time.

We recommend configuring your app to ignore QR codes older than a specific timestamp. Currently, the API doesn’t support clearing QR code history.”

You can see the detailed best practices for QRCode tracking with Hololens2 here

I also noticed that the tracking is slower, so it takes a few seconds to get adjusted to the new positions of the QRCode. But this is not a deal breaker 🙂

The project code is in GitHub here: https://github.com/NSudharsan/HoloLensExamples/tree/master/QRCodeXRSDK

Below is the output video:

I’m not sure why the video output shows that the QRCode displacement by a few cms. In the Hololens View, its perfectly on the QRCode.

By the way, I have tried this code with Legacy XR and it also works. The only change I had to make was in SpatialCoordinateSystem.cs. Comment out the XR related lines on 75 and 76 and add line 77,78. Thanks to Edd Smith from the HoloDevelopers Slack Community who gave me this solution!

 By the way, I have tried this code with Legacy XR and it also works. The only change I had to make was in SpatialCoordinateSystem.cs. Comment out the XR related lines on 75 and 76 and add line 77,78 Thanks to Edd Smith from the HoloDevelopers Slack Community who gave me this solution!

 

22 response to "QRCode tracking with Hololens 2, XR SDK and MRTK V2.5"

  1. By: Rajendra Posted: May 12, 2021

    Hi Nischita, nice article, have one question, do you know when the other events like qr code removed and updated is called. I understand that qr code added event is called when I bring a qr code in front of hololens camera but couldn’t understand when the other events like remove and update is called

    • By: Nischita Posted: May 28, 2021

      Hi Rajendra, The QRCodeAdded event is called whenever the system detects the QRcodes for the first time. QRCodeUpdated event is called each time the QR code location is changed (only applies to previously detected QRCodes). The remove event is called when you want to call it 🙂 i.e imagine a situation where you want to have a timer of 5 min and remove all those QRCodes detected before that timer. Then you call the QRCodesRemoved event. Hope that helps!

  2. By: Auguste Posted: May 28, 2021

    Hi !

    Thank you for this post, it was vrey usefull.

    I have some questions for you:

    – I do not understand how the Hololens succeed to replace QRCode prefab in the location and position in the space even if I move at the start of the app and even if I unistall and install thre app. I think with what I saw in scripts it is because the app scan and place in function of the environment but I am not sure.

    -Do you think it is possible to place gameobject in fuction of the location of the QRCode prefab? If yes, how because I tried without successe.

    Thank you for everything.

    • By: Nischita Posted: May 28, 2021

      Hi Auguste, Glad to help! To answer your questions:
      – The QRCodes with the Hololens are detected at OS level. What that means is the device remembers the QRCodes previously scanned unless you reboot the device itself. Then the QRCodes previously detected would be removed from OS level. Uninstalling the app does not help in resetting QRCodes to not scanned. So if you want the app to work in a way that you trick it to only scan QRCodes when you start the app, have a timer or so and only show those QRCodes which you have detected in the last 5 min or so…
      – Yes, this is possible. Just use the transforms on the qrCodeObject in the QRCodesVisualizer.cs and instantiate your object in the same location, rotation or scale of the QRCode detected.

  3. By: Savary Océane Posted: May 31, 2021

    First of all a huge thank you for your website and your articles it’s really very useful for novices like me.
    I use your QRcode Tracking example to create an HL2 application, but I have a problem I slightly modified the QRcode prefab and the associated script to display the coordinates of QRInfo;
    and I would like to get the coordinates of the QRcode to be able to create 3D objects positioned according to this QRcode.
    I saw that in the script SpatialGraphCoordinateSystem.cs there is a local variable of the function UpdateLocation which gives the position and the rotation of the QRCode.
    I thought that it was possible to recover these local variables and to pass them in “global” in order to transmit them to another script (script which creates my 3D objects).
    Unfortunately I don’t know how to do it, I tried to make a getter/setter but I’m not sure how to call this getter from another script.
    Could you help me ? I am completely new to C# and unity so sorry if my question is stupid.

  4. By: Savary Océane Posted: May 31, 2021

    First of all a huge thank you for your website and your articles it’s really very useful for novices like me.
    I use your QRcode Tracking example to create an HL2 application, but I have a problem I slightly modified the QRcode prefab and the associated script to display the coordinates of QRInfo;
    and I would like to get the coordinates of the QRcode to be able to create 3D objects positioned according to this QRcode.
    I saw that in the script SpatialGraphCoordinateSystem.cs there is a local variable of the function UpdateLocation which gives the position and the rotation of the QRCode.
    I thought that it was possible to recover these local variables and to pass them in “global” in order to transmit them to another script (script which creates my 3D objects).
    Unfortunately I don’t know how to do it, I tried to make a getter/setter but I’m not sure how to call this getter from another script.
    Could you help me ? I am completely new to C# and unity so sorry if my question is stupid.

    • By: Nischita Posted: July 1, 2021

      Hey Oceane, just access the qrCodesObjectsList from your other script. One way is to make that public and access it via getter and setters. You could watch this video to understand getters and setters in c# https://www.youtube.com/watch?v=-Bgt6HJEhf0

  5. By: HSD Hololens2 Posted: July 1, 2021

    Hey Nischita,

    For a project at our University we need to scan QR-Codes to switch in differnt scenes.
    First of all: is this possible?

    Moreover, we try to implement your how to. Unfortunately we can’t find the QRCodePanel and the QRCode_icon within the proposed download folder on GitHub.

    It would be great if you could help us getting our problem solved.

    Yours Sincerely,
    Ramon

  6. By: Ramon Posted: July 8, 2021

    Hi Nischita,

    thank you for your support. We now got the right files.

    But there came a new problem up. We are trying to assign the start and stop functions from the “StartScanButton”. But we can´t see the Onclick()-Event.

    Do you have an idea why there is only Touch Begin () and Touch End() etc.?

    Thank you very much in advance.

    BR
    Ramon

  7. By: HSD Hololens2 Posted: July 15, 2021

    Hey Nischita,

    thank you, not it worked.
    But we´ve got a new problem now. We would like to support a gripper change for a small robot with the Hololens. For this we would like to use two QR codes (for two different grippers) that show the user which gripper is currently mounted.
    By using your instructions, we are unfortunately only shown the link to the associated website after successfully scanning the QR code so far. Do you have any idea how it could be possible to switch to a new scene by scanning the QR code? So that our programme jumps to the corresponding next scene depending on the scanned gripper? I hope you now understand our problem better and can help us.

    BR
    Ramon

  8. By: Auguste Posted: July 30, 2021

    Hi Nischita,

    I do not know why but I cannot post under my first comment.
    I just want to say thank you for helping me, it allows me to finish my application. Sorry to respond just now but when I managed to read QRcode I was to concentrate on my project that I have forgotten to say thank you.

    I wish a great continuation

    Auguste

    • By: Nischita Posted: July 30, 2021

      Hello Auguste, Glad that this helped you. 🙂
      best, Nischita

  9. By: Cherise Tabbiner Posted: August 2, 2021

    Hi Nischita! thanks so much for this amazing guide, I managed to make it work thanks to you! I am now wondering where i should go to look for where the QRCode,Data is stored? i am having a few issues finding it. I essentially want to be able to specify which QR codes should make the program react and produce something and which QR codes should be ignored (so i’m hoping to specify a string that the QRCode.Data should look for)

    if this is probably the wrong way to do it though please let me know. Your help is much appreciated.

    • By: Nischita Posted: September 6, 2021

      Hi, check the article for the section on looking for In line 112 of QRCodesVisualizer, adding LatestQRCodeDetails text to show the data of the most recent QRCode scanned. This is the QRcode data.

  10. By: Elena Posted: September 29, 2021

    Hi Nischita,

    Do you know if this will work using OpenXR instead of the Windows Mixed Reality? (Under XR Plug-in Management)

  11. By: Ari Posted: October 10, 2021

    Hey Nichita!
    Your tutorial helped a lot. I now want to implement a reset button that would delete all scans along with the prefabs associated with them, but when I did that, the scanner would not scan the QR codes again and when I stop and start the scanning it doesn’t help and the old QR codes pop up once again. Could you help me perhaps?
    My reset function clears both SortedDictionary’s in the visualizer and manager.

  12. By: Ari Posted: October 12, 2021

    Hey,
    I want to be able to delete a QR Code after i scanned it and then rescan it. I see there is a RemoveQRCode function but I couldn’t figure out how to call it. Can you help me do so?

    • By: Nischita Posted: October 13, 2021

      Please see my response to your previous question. hope it helps

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: