HoloLens – Interacting with Holograms

Just a few weeks ago, Microsoft announced the new generation of HoloLens, with its new hardware, features, and capabilities bringing a revolution to Mixed Reality as we know it. However, HoloLens 2 is not yet fully available, while the HoloLens 1 is still getting updates and we have yet to exploit its full potential.

In today’s article, I am going to show you how to position holograms in the 3D physical space and interact with them, adding animations and gestures. Developers who are still exploring the device’s true potential can use this demo as a reference.

At the bottom of the post, there will be a link to download the source code of this project.

What is Microsoft HoloLens

Microsoft HoloLens debuted in 2016 as a cutting-edge Mixed Reality head-mounted display. HoloLens maps the 3D space around you and projects holograms in front of your eyes, allowing you to interact with them in a natural way.

Prerequisites

In order to develop a HoloLens app, the following tools are required.

• Windows 10 Pro or Enterprise
• Visual Studio 2017 with the following components installed:
  • Universal Windows Platform development
  • USB Device Connectivity
• Unity LTS 2017.4
• Mixed Reality Toolkit for Unity (at the time of writing the HoloToolkit 2017.4.3.0 is available)

Got the tools downloaded? Great, we are ready to start!

Setting up the working environment

First, create a new Unity project, ensuring the 2017.4 version is selected, and selecting the 3D template.

When Unity fires up, there is a standard procedure to follow to develop an app targeting HoloLens:

First, set the Camera’s Clear Flags property to Solid Color and specify the 0,0,0,0 color. HoloLens understands this as “blank” and “see-through”. Reset its position to 0,0,0, and set the Near clip to 0.85. This is your head!

Next, change the Default Quality Settings of the UWP platform to Very Low.

Finally, switch the platform to Universal Windows Platform and select HoloLens as the Target Device. Then go to the Player Settings to make sure that the Scripting Backend is selected to .NET, and enable the Virtual Reality Supported in the XR Settings.

Displaying and interacting with holograms

Since everything is ready, we can now add some holograms to interact with.

Showing a 3D object

Create a 3D Object Cube in the Hierarchy, and position it to 0,0,2 in front of the camera. Then, rotate it to 45,45,45 and scale it to 0.25,0.25,0.25 to better view its 3D volume.

You are ready to build this to your device. When a project is built from Unity for Universal Windows Platform, a Visual Studio project is created. In Visual Studio, open the generated Solution (.sln) file and change the Platform to x86 and the solution configuration to Release.

In the Target Machine option, try the HoloLens Emulator first. From the Debug option select Start without Debugging to avoid debug messages that may slow down the procedure.

The first time the Emulator starts it will be slow, but it can be left open for the succeeding builds. When the virtual OS starts, your app will be installed and ran.

You can explore the cube using the WASD keys, and look around with the left mouse button. When you are happy with the result, change the target machine to Device, connect your HoloLens headset to a USB port, and start another build.

That would be the result on your device:

Adding interactivity

With just the barebones you were able to develop a HoloLens app. In order to make the development process smoother, the Mixed Reality Toolkit is recommended. So, go ahead, download and import the package to the Unity Editor.

When the toolkit is imported you’ll have a new “Mixed Reality Toolkit” option in the Unity menu bar. Select Apply Mixed Reality Scene Settings and Apply Mixed Reality Project Settings.

These tools add utilities to the scene to simulate the device in Play mode.

Now, create a script named InteractiveObject.cs to add some interactivity to the cube and add two interfaces to implement to the class IFocusable and IInputClickHandler.

Define some variables to use for the interaction:

private bool hasFocus = false;
private float rotationSpeed = 2f;
private float scaleModifier = 1.025f;

Then, implement the IFocusable interface to be notified whenever the object has focus.

public void OnFocusEnter()
{
    hasFocus = true;
}
public void OnFocusExit()
{
    hasFocus = false;
}

Also, implement the IInputClickHandler interface to scale up the object when you air-tap it.

public void OnInputClicked(InputClickedEventData eventData)
{
    Vector3 currentScale = transform.localScale;
    currentScale *= scaleModifier;
    transform.localScale = currentScale;
}

Finally, rotate the object when it has focus, in the Update method.

private void Update()
{
    if (hasFocus)
        transform.Rotate(new Vector3(0f, rotationSpeed, 0f), Space.World);
}

For the final touch, add the above script to the Cube object. When you press Play in Unity, the cube should start rotating.

Build the project again and see the results. The cube should rotate whenever you look at it and scale up when you air-tap it.

Source code

The source code of the project is available on GitHub. Instead of the cube, the project also includes a rigged 3D model of the Codeman you saw in the video.

Remember to share this post if you found it useful!

Resources

Want to become an expert in Mixed Reality and Cloud computing? Experts start from the basics:

Getting Started Tutorials

• Getting Started with Azure App Services
• Getting Started with HoloLens
• Getting Started with Unity3D

Software Development Kits

• Azure SDK for Unity3D
• Mixed Reality Toolkit for Unity3D

LightBuzz Tutorials and Guides

• The Definitive Azure SDK for Unity3D
• Speech Recognition in Unity3D
• A Pokemon Go clone for HoloLens
• HoloLens – The Ultimate Introduction
• Azure + HoloLens Source Code

Summary

At this point, you have created a HoloLens app you can interact with by using your gaze and the air-tap gesture. Try expanding it by replacing the cube with something more interesting, like a sculpture or a vintage car.

What would you like to create in Mixed-Reality? What are you expecting from the next generation of Mixed-Reality hardware? Let me know in the comments below!