HoloLens Spatial Mapping – A Developer’s Guide

In my previous HoloLens post, we talked about showing a holographic object that retains its position while you keep moving. We also implemented a way to rotate the digital object when looking at it, and interact with it using the Air-Tap gesture. Although this demo by itself is incredible at showcasing the capabilities of Microsoft HoloLens, the device can do much more.

In this article, we’ll explore one of the most impressive features of Microsoft HoloLens: Spatial Mapping. Spatial Mapping is a process that allows HoloLens to understand its environment and surroundings. Spatial Mapping is achieved using the integrated depth cameras, inherited by the Kinect. Depth cameras are measuring the distance between the device and its surroundings. This way, the device can detect objects, such as volumes or planes, and significantly enhance the user experience.

As software developers, we’ll use the Spatial Mapping capabilities of the device to place virtual objects on top of real objects!

Source code

The source code of the project is available on GitHub.

Remember to share this post if you found it useful!

What is Microsoft HoloLens

Microsoft HoloLens debuted in 2016 as a cutting-edge Mixed Reality head-mounted display. HoloLens maps the 3D space around you and projects holograms in front of your eyes, allowing you to interact with them in a natural way.

What is Spatial Mapping

Spatial Mapping is the representation of real-world surfaces in 3D. HoloLens 1 is equipped with 4 environment understanding cameras, an RGB camera and a depth camera, thus giving it the ability to understand surfaces in the real world and reconstruct them in 3D.

Prerequisites

In order to develop a HoloLens app, the following tools are required.

• Windows 10 Pro or Enterprise
• Visual Studio 2017 with the following components installed:
  • Universal Windows Platform development
  • USB Device Connectivity
• Unity LTS 2017.4
• Mixed Reality Toolkit for Unity (at the time of this writing, HoloToolkit 2017.4.3.0 is available)

Got the tools downloaded? Great, we are ready to start!

Setting up the working environment

First, launch Unity3D and import the Mixed Reality Toolkit plugin. Upon importing the plugin, you should see the Mixed Reality Toolkit menu item. From there, click the Apply Mixed Reality Project Settings and Apply Mixed Reality Project Settings options. These options allow you to use the Spatial Mapping Prefabs. The following prefabs must be present in your scene:

At the Spatial Mapping Manager that sits on the SpatialMapping GameObject, make sure that the option to Draw Visual Meshes is enabled. This will add a visual representation of the meshes scanned in our Scene. The visual representation is that cool grid of triangles you saw in the video.

You have now properly set up the scene for the environment understanding!

Placing a hologram on a real-world surface

Up to this point, almost everything is done automatically by the Mixed Reality Toolkit. Let’s type some C# code to position holograms on top of the physical surfaces detected by HoloLens. To start, create a new script named PlacingManager and declare a variable to assign the object you want to place.

[SerializeField] private GameObject objectToPlace;

This variable will later be assigned the prefab you want to spawn.

Now, you need to extend the class with the IInputClickHandler interface. This interface declares a method that listens to the air-tap gesture. To implement this interface write the following code:

public void OnInputClicked(InputClickedEventData eventData)
{
    PlaceObject();
}

The PlaceObject method will be created later. The OnInputClicked won’t have any collider like the cube in the previous tutorial to listen to this event. You just have to listen to any event dispatched from the FallbackInputHandler of the InputManager, so implement the following methods:

private void Start()
{
    InputManager.Instance.PushFallbackInputHandler(gameObject);
}
private void OnDestroy()
{
    if (InputManager.Instance != null)
        InputManager.Instance.PopFallbackInputHandler();
}

The FallbackInputHandler will call the OnInputClicked in case no other script uses this event. In order to avoid spawning too many objects in the same place, we will use this functionality later on.

For spawning the prefab add this code:

public static bool GetLookAtPosition(Vector3 headPosition, Vector3 gazeDirection, out Vector3 hitPoint)
{
    float maxDistance = 10f;
    if (SpatialMappingManager.Instance == null)
        throw new System.Exception("Missing spatial mapping manager");
    RaycastHit hit;
    if (Physics.Raycast(headPosition, gazeDirection, out hit, maxDistance, SpatialMappingManager.Instance.LayerMask))
    {
        hitPoint = hit.point;
        return true;
    }
    hitPoint = Vector3.zero;
    return false;
}
public void PlaceObject()
{
    Vector3 positionToPlace;
    Transform camTransform = Camera.main.transform;
    Vector3 lookBack = camTransform.rotation.eulerAngles;
    lookBack.x = lookBack.z = 0f;
    lookBack.y += 180f;
    if (GetLookAtPosition(camTransform.position, camTransform.forward, out positionToPlace))
    {
        InstantiateObject(positionToPlace, lookBack);
    }
}
private void InstantiateObject(Vector3 position, Vector3 rotation)
{
    Instantiate(objectToPlace, position, Quaternion.Euler(rotation));
}

The GetLookAtPosition checks where the camera (or HoloLens) is looking at and saves the position to the hitPoint variable. If there is no surface in front of the camera it returns false.

The PlaceObject method is responsible for 3 tasks:

1. Checks where the camera is at,
2. Calls the GetLookAtPosition method to check if there are surfaces in front of the camera, and
3. Calculates the rotation of the spawned object to look directly at the camera by adding 180 degrees to its rotation.

Finally, the InstantiateObject creates the hologram by passing it the position and rotation it should have. With this, the script is finalized.

Back to Unity, in the Hierarchy create an Empty GameObject, and name it PlacingManager. Add the script PlacingManager as a component to the GameObject PlacingManager. Then, add a prefab to the Object To Place reference. In the project’s repository, we have added our very own Codeman, but if you haven’t downloaded it, you can create an empty cube to test it. Just create a cube in the Hierarchy and drag it in your Assets folder.

Avoiding spawning objects on top of another

Right now you should be able to spawn multiple objects everywhere even in the same place. In order to avoid that, in the prefab you have created, add a simple script to consume the event.

public class ConsumeInputClickHandler : MonoBehaviour, IInputClickHandler
{
    public void OnInputClicked(InputClickedEventData eventData)
    {
        eventData.Use();
    }
}

This prevents calling the FallbackInputHandler from invoking this event and spawning another prefab.

Finally, build and deploy the project to the device and check the result.

Source code

The source code of the project is available on GitHub.

Remember to share this post if you found it useful!

Resources

Want to become an expert in Mixed Reality and Cloud computing? Experts start from the basics:

Getting Started Tutorials

• Getting Started with Azure App Services
• Getting Started with HoloLens
• Getting Started with Unity3D

Software Development Kits

• Azure SDK for Unity3D
• Mixed Reality Toolkit for Unity3D

LightBuzz Tutorials and Guides

• A Pokemon Go clone for HoloLens
• Interacting with holograms in HoloLens
• HoloLens – The Ultimate Introduction
• The Definitive Azure SDK for Unity3D
• Azure + HoloLens Source Code
• Speech Recognition in Unity3D

Summary

You have created an app that understands objects of the environment (desks, walls, floors, etc.) and creates a virtual representation of them. If you would like a more realistic approach without the triangles, disable the Draw Visual Meshes option from the Spatial Mapping Manager and share your app with others.

So, what’s your opinion on Microsoft’s HoloLens Spatial Mapping? Is it like you imagined it would be or are there features you would like to see included in the next generation of HoloLens? Use the comment section below to share your thoughts.