HoloLens® Vuforia® Image Tracking for Unity®

The Vuforia Engine uses computer vision and object recognition to enable AR applications to correctly render virtual objects in space. This makes it very useful for correctly placing haptics in an AR scene, and for integrating with the Ultrahaptics Core Asset (UCA) for Unity.

Below we show the steps we took to use Vuforia® with Unity® and Microsoft®’s HoloLens® AR headset as well as outlining some of the tips and workarounds that we found.

Quick links

• Installation
  • HoloLens Development Prerequisites
  • Vuforia Unity Installation
    • Adding Vuforia to an existing Unity installation
    • Microsoft HoloLens users
• Project Setup
• Creating Tracked Image Targets
  • Vuforia License Key
  • Custom Images
  • Adding custom images to Unity
• Testing and deploying on HoloLens
• Troubleshooting Checklist

Note: these steps may differ for your individual set up. You should check documentation provided by Microsoft, Vuforia, and Unity. Ultrahaptics takes no responsibility for information provided and all usage is at your own risk.

Installation

These instructions apply to a Windows Installation, although Vuforia can be used with MAC OS, but without the HoloLens support.

HoloLens Development Prerequisites

Follow the installation checklist for the items found on Microsoft’s install the tools. You do not, however, need to install the HoloLens emulator, as its likely Vuforia will not work with it anyway (due to requiring the webcam feed).

You need to enable a few options on the HoloLens itself before developing:

1. Open the HoloLens Settings Application
2. Go to “Update & Security”
3. Go to “For developers”
4. Enable “Use developer features“
5. Enable “Enable Device Portal”. This lets you see device information in your web browser

• • Over WIFI this is https://<device IP address>, accept the certificate
  • Over USB it is http://127.0.0.1:10080
  • You may have to use the device discover and pair option to initially connect.
  • Refer to Microsoft’s using the windows device portal for more information.

An alternative and easier way to obtain the IP of your connected device is to install the “Holographic Remoting Player” from the Windows Store. This displays the IP once launched.

Vuforia Unity Installation

Vuforia integrates with Unity and is installed as a playback engine as part of the installation. You can use any version of Unity you wish. However, newer versions may be missing several key build options (Unity are planning to deprecate the UWP .NET backend, which will make building and deploying to the HoloLens significantly slower).

1. 1. Install Unity, making sure to tick “UWP Build Support (.NET)” (you do not need the IL2CPP option if .NET is available to you). (If you already have Unity installed, simply tick the UWP option only).
   2. For now, do not tick the Vuforia support option.

Adding Vuforia to an existing Unity installation

1. Run the installer (you may have to download it again from Unity download archive)
2. Select only the Vuforia Augmented Reality Support component.

Microsoft HoloLens users

Vuforia doesn’t always work with HoloLens on later versions of their SDK. Any version you install past those included with Unity 2017.4 will not work properly.

To solve this issue, you have to do some configuration editing:

Rather than allowing you to import a Unity package, Vuforia offers only its latest version on its website. Therefore, you must download and run the installer for an older Unity version

1. Download the Unity 2017.4.11f1 installer
2. Install the Vuforia AR component, setting the destination folder to your existing Unity destination, e.g. Unity 2018.2.
3. You will receive a warning about version mismatch, simply press yes to continue.
4. Navigate to your Unity installation folder, then “Editor\Data\PlaybackEngines\VuforiaSupport”
5. Open the ivy.xml file and change the instances of 2017.4.11 to your current version of Unity.
6. For both version and e:unityVersion, you should have something like version=”2018.2.11″ and e:unityVersion=”2018.2.11f1″ (e:unityVersion is the full version number).
   • Note that you may have to save the ivy.xml file to somewhere else, then copy it back to the folder.
7. Relaunch Unity. You’ll see the Vuforia options available when right-clicking in your hierarchy.

Project Setup

HoloLens is relatively limited in the amount of processing power you actually have available. This means there are a number of sacrifices that must be made from the get-go. The list below shows the settings we used to get our system to work. Not all of these will apply to your project.

1. Open Edit > Project Settings > Player
   • In the inspector, from XR Settings, enable Virtual Reality Supported and Vuforia Augmented Reality Support.
   • Go to Publishing Settings > Capabilities and enable InternetClient, WebCam, Microphone and SpatialPerception.
   • Go to Other Settings > Configuration and change the Scripting Backend to .NET
2. Open File > Build Settings, click on Universal Windows Platform and press the “Switch Platform” button.
   • Tick the Unity C# Projects box
   • Set your Target Device to HoloLens
3. Open Edit > Project Settings > Quality
   • Set the UWP default to Very Low
   • You can change a few features such as Texture Quality to Full Res. However, be aware that HoloLens has a whopping 600MB of VRAM available.
4. Right-click in the Inspector and select Vuforia > AR Camera
   • You’ll get a message asking to Import Vuforia Assets, click Import
   • Click on the ARCamera and change World Center Mode to DEVICE_TRACKING (newer versions of Vuforia do not have this option).
   • Change the background colour alpha to 0 (RGB should already be 0)
5. Open the Vuforia Configuration either from Assets/Resources/VuforiaConfiguration or from the ARCamera > Vuforia Behaviour.
   • for Digital Eyewear, change Device Type to Digital Eyewear and Device Config to HoloLens.
   • for Device Tracker, tick Track Device Pose.
   • Finally, change Tracking Mode to Positional.
6. Remove the Main Camera from your scene.

Creating Tracked Image Targets

Vuforia works by tracking predefined images, or markers, using an attached camera, be that a webcam or your AR headset. Vuforia provides a selection of default markers. These are selected by default when you first add the AR Camera to your scene.

Vuforia License Key

To add your own image targets or use the ones used in this article you must add an App License Key to your VuforiaConfiguration. When developing you can use a key from anyone’s account (we only need Image Targets). However, your shipped Unity application will need a fully paid license for each user.

• Register with Vuforia to create a free Vuforia developer account.
• Go to https://developer.vuforia.com/license-manager
• Click “Get Development Key”. You’ll see a long string of random characters. Copy and paste this into the App License Key section in the Vuforia Configuration.

Custom Images

Vuforia has extensive guidelines on how to create your own tracking images, as well as support for many of the steps we’ve taken in this article. However, as a rule of thumb, make sure your image has a non-repeating, large defining feature. For our application, we found that the Ultrahaptics text and logo make a pretty good start. It then required a “spiders web” of lines behind it. Images can be of any size, but they should have a 4:1 aspect ratio.

To add your image on developer.vuforia.com go to Target Manager

1. Create a new target database by clicking “Add Database”. Make sure Device type is selected.
2. Select your database and click Add Target
3. Select Single Image for type and upload your image file.

Use a 24-bit RGB (no alpha) PNG.

To convert an image to 24-bit PNG, use the free paint.net Chrome app and save it through that. This allows you to select which type of PNG you want.

4. Set the image width (in metres)
5. Give the target tracker a name. This will appear as the tracker’s name in Unity.

Once uploaded Vuforia takes a few minutes to process, then scores your image between 0 and 5 for tracking performance. You can see the targeting points used by Vuforia by selecting the target and clicking Show Features.

You can see two examples of high and low rated images here:

Rating	Uploaded Image	Image Features
5
+ High contrasting points   + Highly varied foreground and background + Feature points cover a large majority of the image
0
+ Varied design   – Not a clearly varied design – Lack of varied contrast (dark colours with dark lines) – Very few non-unique points – Limited unique shapes (no curves etc)

Adding custom images to Unity

Once you’ve created your image database you can use it in your Unity scene.

1. Download your custom image database from the Vuforia developer site and import the resulting .unitypackage file to your project.
2. Open your Vuforia Configuration, your database should be listed in the Databases
3. Click Load on your custom database, making sure to tick Activate (this step may not apply for all Unity installations).
4. In the Unity Hierarchy, right-click to add a Vuforia > Image to the top level.
5. In the Vuforia Image object’s inspector, the Image Target Behaviour should be pre-set with your database and target image. If not, go ahead and select these from the drop-down boxes (you can only use one chosen image in the scene).
6. Under the Advanced Options, tick “Preserve Child Size” and “Extended Tracking”.

This will make a standard, working Image Target. You can then build your Unity scene around this, making it either a fixed anchor point or attaching moving AR objects. As an example, add a 3D object as a child of your Image Target.

By default, the target image will appear in play mode. To avoid this

1. Download the “Vuforia HoloLens Sample” from the asset store
2. Import the script, and ONLY the script “SampleResources/Scripts/CustomTurnOffBehaviour.cs”
3. Attach the script to your image target.
4. Remove the “cs” script from the image target
5. Set the Turn Off Rendering option to “Play Mode and Device”

Testing and deploying on HoloLens

Microsoft provide detailed instructions for deploying HoloLens applications on their website. However, you can also use the steps we found that worked best for us below.

Without Vuforia, the default usage of HoloLens is with the Holographic Remoting Player. This allows you to test your application remotely, without having to deploy to the device.

Vuforia does not work with this, as the Remoting Player does not give you the webcam feed. This means you must deploy to the device every time you wish to test features or changes. Fortunately, the .NET backend is relatively quick to compile, though somewhat slower for IL2CPP.

You can deploy to the HoloLens over USB or WIFI (for large projects, USB is recommended).

1. Follow the steps outlined in “Project Setup” above.

1. Make sure to add all the scenes you want to include within the project. Remember the larger your project, the longer it will take to build and deploy to the device each time.
2. Press Build. Once completed, you will have a folder with a Visual Studio Solution (.sln) file.
3. Open the .sln file with Visual Studio
4. Set your configuration to Release x86.
5. For USB deployment, set the build target to “Device”.
6. For WIFI deployment
   • Go to Project > “Project Name” properties
   • Go to the Debug panel
   • Change Target Device to Remote Machine
   • Under Remote Machine click the “Find…” button
   • Either enter the IP of the device or use the “Auto Detected” option
   • You may need to pair the device (see “HoloLens Development Prerequisites” above)
7. Once setup, simply press the Play button
   • Do not run or interact with anything on the HoloLens during this process, the deploy will error if any user interaction is performed (other than physical movement)
8. Once deployed, the app will automatically run.
   • Each time you deploy, it will ask for permissions required by Vuforia

Troubleshooting Checklist

These are some troubleshooting tips, for common issues you may encounter with Vuforia:

Image targets not tracking.

• Is your image too small?
• Is the sizing of the image in Unity too disparate from the real world?
• Have you made sure to enable the database in the Vuforia Configuration for your project?
  • You will still be able to select custom images within Unity, even if you do not enable and activate the related database.
• Have you made sure to include an API key in the Vuforia Configuration?

Vuforia does not seem to be working properly.

• Do you have the correct version?
  • The current version that comes with Unity 2018.2 does not correctly work with the HoloLens
• Have you made sure to delete other cameras in the scene?
• Have you made sure the Vuforia Behaviour is attached to your ARCamera?
• Is the Vuforia Behaviour set to DEVICE_TRACKING on World Center Mode?
• Is your project colour space set to Gamma?
  • Vuforia and HoloLens do not work with Linear
• Have you made sure to set all the correct Capabilities?
  • Refer to Project Settings steps above

Vuforia is not an option in the right-click menu.

• Are you using the correct version of Unity?
• Have you correctly set the version number in ivy.xml?
  • Refer to Vuforia Installation steps above.