Body tracking for every camera
LightBuzz has created the world’s most accurate body-tracking software solution. Companies and universities worldwide use our SDK to develop commercial apps for desktop and mobile devices.
This article is based on a previous ARKit 3 tutorial & source code by Konstantinos Egkarchos.
Apple has officially entered the AI-powered body-tracking industry! With its new pose estimation capabilities, ARKit 3.5 is a Kinect alternative for iOS mobile devices. People occlusion and human pose estimation are now core parts of the latest ARKit 3.5 framework. More importantly, the new iPad Pro 2020 (as well as the upcoming iPhone PRo) is now equipped with a LiDAR depth camera.
Why is that important? It’s crucial because depth-sensing allows the camera to better understand its surroundings and estimate their distances. In terms of body-tracking, LiDAR camera allows developers to capture human motion in 3D.
So, without further ado, I am going to show you how to develop body-tracking apps for iPad (and future iPhone) devices in Unity3D!
More specifically, we’ll detect and visualize the following human body joints:
Prerequisites
Since we are developing for the Apple ecosystem, we need the proper Mac computer to develop our applications and the proper iOS device to run them.
Hardware
In terms of hardware, you need a MacOS computer that is compatible with MacOS Catalina. Also, body tracking applications need the powerful Apple A12 Bionic processors to run properly. The following Mac computers and iOS devices would be eligible:
Computers | Mobile devices |
12-inch MacBook | iPad Pro 2020 with LiDAR |
MacBook Air, 2012 and later | iPhone 2020 with LiDAR (expected later this year) |
MacBook Pro, 2012 and later | |
Mac mini, 2012 and later | |
iMac, 2012 and later | |
iMac Pro | |
Mac Pro, 2013 and later |
For this guide, I am using a MacBook Pro computer with an 11” iPad Pro 2020.
Software
To run the demos, you need to install the following software on your Mac computer:
- Unity3D 2020.2.07a with iOS build target
- MacOS Catalina 10.15
- XCode 11.4
Your iOS device should be updated to iOS 13.4 or iPadOS 13.4.
If you are in a hurry, download the complete source code on GitHub. Keep reading to understand how to create your own body-tracking apps!
Did you know?
For the past 10 years, LightBuzz has been helping Fortune-500 companies and innovative startups create amazing body-tracking applications and games. If you are looking for a reliable contractor to develop your next Motion Tracking project, get in touch with us.
Body Tracking step-by-step
Enough said… Let’s dive right into the ARKit magic. On your computer, launch Unity3D 2020.2 and create a new project.
Step 1 – Set up the main scene
Unity3D will start with an empty scene. Before adding any visual objects or writing any code, we first need to import the proper dependencies. The skeleton-tracking functionality is part of the ARKit toolkit. As a result, we need to import the ARKit XR 4.0.0 and ARFoundation 4.0.0 dependency packages.
Troubleshooting
At the time of this writing, you should use the preview versions 4.0.0-preview.1. If you can’t see those packages in Unity 2020, do the following:
- Open the Package Manager view (Window Package → Manager).
- Click Advanced.
- Check “Show preview packages”.
The AR Session Origin
Now, create a new scene and add an AR Session and an AR Session Origin object. These objects are controlling the iOS camera while providing a ton of ARKit goodies.
Also, add an empty game object, name it e.g. Human Body Tracking, and attach a new C# script (HumanBodyTracking.cs).
The structure of the scene should look like this:
Step 2 – Set up the Skeleton
Since the visual elements are in place, we can now start adding some interactivity. Open the HumanBodyTracking.cs script and add a reference to the ARHumanBodyManager class. The ARHumanBodyManager is the primary script that analyzes the camera data to detect human bodies.
[SerializeField] private ARHumanBodyManager humanBodyManager;
To display the joints, we’ll use some simple Unity3D spheres. Each sphere will correspond to a specific joint type. Add a C# Dictionary class to update the joint data, frame-by-frame.
private Dictionary<JointIndices3D, Transform> bodyJoints;
Finally, add references to the user interface elements of the skeleton. We’ll need a sphere object for the joints and a line object for the bones.
[SerializeField] private GameObject jointPrefab; [SerializeField] private GameObject lineRendererPrefab; private LineRenderer[] lineRenderers; private Transform[][] lineRendererTransforms;
You can find the complete C# code in the HumanBodyTracking.cs class on GitHub.
Step 3 – Detect the Tracked Bodies
This is the most important part of the tutorial! ARKit has made body-tracking incredibly easy and accessible. All you need to do is use the ARHumanBodyManager object and subscribe to the humanBodiesChanged event.
private void OnEnable() { humanBodyManager.humanBodiesChanged += OnHumanBodiesChanged; } private void OnDisable() { humanBodyManager.humanBodiesChanged -= OnHumanBodiesChanged; }
The event handler is where the magic happens. The information about the tracked bodies is part of the event arguments. This is how to acquire the bodies:
private void OnHumanBodiesChanged(ARHumanBodiesChangedEventArgs eventArgs) { foreach (ARHumanBody humanBody in eventArgs.added) { UpdateBody(humanBody); } foreach (ARHumanBody humanBody in eventArgs.updated) { UpdateBody(humanBody); } }
Piece of cake, right? So, let’s bring everything together and display the skeleton in the Unity user interface we created in the previous steps.
Note: as of the time of this writing, the ARKit only supports one tracked body.
Step 4 – Display the Skeleton
The following lines of code update the positions of the joints in the camera space. The spheres and lines are overlayed on top of the iOS camera feed.
private void UpdateBody(ARHumanBody arBody) { if (jointPrefab == null) return; if (arBody == null) return; if (arBody.transform == null) return; InitializeObjects(arBody.transform); NativeArray<XRHumanBodyJoint> joints = arBody.joints; foreach (KeyValuePair<JointIndices3D, Transform> item in bodyJoints) { UpdateJointTransform(item.Value, joints[(int)item.Key]); } for (int i = 0; i < lineRenderers.Length; i++) { lineRenderers[i].SetPositions(lineRendererTransforms[i]); } }
Apple supports 92 joint types (indices). However, not all of these joint types are actually tracked! Most of them are inferred, based on the positions of their neighboring joints. For your convenience, I have selected 14 joint types, so I can have a fair comparison with the Kinect camera.
This is how to connect the proper joints and form the human bones:
private void InitializeObjects(Transform arBodyT) { if (bodyJoints == null) { bodyJoints = new Dictionary<JointIndices3D, Transform> { { JointIndices3D.head_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.neck_1_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.left_arm_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.right_arm_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.left_forearm_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.right_forearm_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.left_hand_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.right_hand_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.left_upLeg_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.right_upLeg_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.left_leg_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.right_leg_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.left_foot_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.right_foot_joint, Instantiate(jointPrefab, arBodyT).transform } }; lineRenderers = new LineRenderer[] { Instantiate(lineRendererPrefab).GetComponent<LineRenderer>(), // head neck Instantiate(lineRendererPrefab).GetComponent<LineRenderer>(), // upper Instantiate(lineRendererPrefab).GetComponent<LineRenderer>(), // lower Instantiate(lineRendererPrefab).GetComponent<LineRenderer>(), // right Instantiate(lineRendererPrefab).GetComponent<LineRenderer>() // left }; lineRendererTransforms = new Transform[][] { new Transform[] { bodyJoints[JointIndices3D.head_joint], bodyJoints[JointIndices3D.neck_1_joint] }, new Transform[] { bodyJoints[JointIndices3D.right_hand_joint], bodyJoints[JointIndices3D.right_forearm_joint], bodyJoints[JointIndices3D.right_arm_joint], bodyJoints[JointIndices3D.left_arm_joint], bodyJoints[JointIndices3D.left_forearm_joint], bodyJoints[JointIndices3D.left_hand_joint]}, new Transform[] { bodyJoints[JointIndices3D.right_foot_joint], bodyJoints[JointIndices3D.right_leg_joint], bodyJoints[JointIndices3D.right_upLeg_joint], bodyJoints[JointIndices3D.left_upLeg_joint], bodyJoints[JointIndices3D.left_leg_joint], bodyJoints[JointIndices3D.left_foot_joint] }, new Transform[] { bodyJoints[JointIndices3D.right_arm_joint], bodyJoints[JointIndices3D.right_upLeg_joint] }, new Transform[] { bodyJoints[JointIndices3D.left_arm_joint], bodyJoints[JointIndices3D.left_upLeg_joint] } }; for (int i = 0; i < lineRenderers.Length; i++) { lineRenderers[i].positionCount = lineRendererTransforms[i].Length; } } }
ARKit is giving us the position and rotation of the joints in the 3D space! This is how to update the scale, position, and rotation of the sphere in the 2D screen space:
private void UpdateJointTransform(Transform jointT, XRHumanBodyJoint bodyJoint) { jointT.localScale = bodyJoint.anchorScale; jointT.localRotation = bodyJoint.anchorPose.rotation; jointT.localPosition = bodyJoint.anchorPose.position; }
This is it! Let’s build and run our project on an actual iOS device!
Step 5 – Build and Deploy
Finally, we need to build and run the project on an actual device. Given that ARKit is part of iOS and iPadOS, we cannot test our code on MacOS (I would love to see a simulator, though).
In Unity, select File → Build Settings. Click the iOS build target and hit the Build button. You’ll need to specify a location to store the generated project. Wait patiently until Unity finishes with the build process.
Unity will create an XCode project (.xcodeproj). Open the project with XCode 11.4. If you use a previous version of XCode, you’ll get an error and your project will not run properly.
When the project is launched, provide your iOS Development credentials, connect your iOS 13.4 device, and click the Run button. This way, the project will be deployed to the device.
When finished, you should point the camera to a person and you’ll start seeing the 3D overlay on top of the tracked body!
By the way…
For the past 10 years, LightBuzz has been helping Fortune-500 companies and innovative startups create amazing body-tracking applications and games. If you are looking for a reliable contractor to develop your next Motion Tracking project, get in touch with us.
If you liked this article, remember to share it on social media, so you can help other developers, too! Also, let me know your thoughts in the comments below. ‘Til the next time… keep coding!
Hello. Would this work for Lidar and Non-Lidar based iOS devices that contains >A12 chip?
Yes. The previous generations of the iPhone and iPad devices would work, too.
Hi. Is it possible to get this working with the front facing camera?
Hi Khalid. Using the latest ARFoundation Toolkit, you can switch cameras using the requestedFacingDirection property.
Vangos. I can’t seem to get it to work with the front facing camera. When I configure the Scene.AR Session Origin.AR Camera.AR Camera Manager (Script) with the parameter Facing Direction set to User it builds an IOS project just fine, but when it runs it outputs:
Using session configuration 0x1eb8b5500
Requested Features: User Facing Camera, Rotation and Orientation, 3D Body Tracking, Auto-Focus
Supported Features: Rotation and Orientation, 3D Body Tracking, Auto-Focus
Requested features not satisfied: User Facing Camera
UnityEngine.XR.ARSubsystems.XRSessionSubsystem:DebugPrintConfigurationChange(Configuration, Feature)
UnityEngine.XR.ARSubsystems.XRSessionSubsystem:Update(XRSessionUpdateParams)
UnityEngine.XR.ARFoundation.ARSession:Update()
I don’t thing the AR Foundation understands how to use the front facing camera on the ipad with lidar sensor.
It seems that not every camera supports every scenario.
Hello Vangos! Have you seen any improvements of the body tracking with the LiDAR camera. I have not but apple claims it shall have been improved. What’s your findings? Many thanks! Barre
Yes. I have compared ARKit on iPad 2018 and iPad 2020. The LiDAR version is less jittery.
When I build and run the project on an iOS device (iPad Pro), the program starts, but only shows a black screen, meaning I can’t see the camera feed or the overlay. Any idea why?
PS: I can successfully run the AR-Foundation examples (https://github.com/Unity-Technologies/arfoundation-samples), where the 2D body tracking is working, if a bit flakey.
Hi Harry. We are preparing an updated article for body-tracking on iOS. For now, you can import the current code to the latest AR Foundation sample projects.
this is great !
is is possible to export the captured motion data as bvh or fbx file ?
Is there that much improvement using Lidar vs without ? are there any videos showing the increased capture quality using lidar ?
thanks !
Hi Tom. There is a significant improvement in the Z (depth) axis when using a LiDAR camera. LiDAR is a type of depth camera. This means it can “see” in 3D. Older iPhone/iPad models rely on plain-old RGB cameras.
We are offering the BVH export as a service to our customers. Feel free to contact us if you are interested to learn more about it.
I am working on ipad mini 5, but it keeps showing a black screen, is there any solution?
Check the Unity logs to ensure it’s properly connected. Also, check Unity’s AR Foundation Samples for the newest version.