Create an AI Personal Trainer with a few lines of Code [Tutorial]

TL;DR

In this article, I’m showing you how to develop a virtual fitness trainer using the mighty power of AI and the ancient wisdom of Maths. By the end of this article, you’ll learn how to analyze human motion, measure angles between joints, and count the number of bicep curls performed!

Let’s start today’s tutorial with a quiz.

💪 Arnold Schwarzenegger

Holywood actor, former governor of California, Mr. Olympia champion, elite bodybuilder, and celebrated fitness icon.

📐 Euclid of Alexandria

Ancient Greek mathematician and author of the Elements, the most fundamental textbook in the history of Mathematics.

🤔 Can you guess what these two gentlemen have in common (apart from being so undeniably jacked)? The answer is Geometry, of course! Wait, what?! Let me explain: Euclid laid the foundations of Geometry, allowing scientists to objectively measure the relative position, distance, and orientation of objects. Millennia later, Arnold subconsciously used geometry to apply proper form to his training and maximize his muscle gains. For example, when doing bicep curls, he ensured his wrists traveled the whole range of motion, bringing the dumbbells all the way up to his shoulders.

So, let’s dive into the world of AI-powered motion analysis!

What is Motion Analysis?

Motion analysis involves measuring and understanding a person’s movements, whether in sports, physical therapy, or even video games (does anyone remember Kinect?). In our case, we want to analyze the motion of the human body during fitness exercises. Motion analysis helps evaluate movement quality, detect errors, provide feedback, and improve performance. In other words, motion analysis allows athletes to bring their A-game to the floor.

The biggest benefit of motion analysis is that we can objectively analyze our workouts instead of relying on guesswork. To achieve that, we’ll combine modern AI technology and some old-school (yet powerful) math formulas.

Basketball player (Vangos Pterneas) motion analysis

Vangos Pterneas while missing some easy hoops in California. Left: no AI – Right: with AI.

AI Magic

Body Tracking (Pose Estimation) definition

One crucial aspect of motion analysis is body tracking or pose estimation. It’s an AI-based technology that allows computers to analyze and understand human body movements. Body tracking systems use Machine Learning and Deep Neural Networks trained on annotated images and videos to detect human body landmarks.

The internal structure of such a system is quite complex. For example, LightBuzz has trained a Neural Network using millions of annotated images and videos. Based on the quality of its training procedure, the software can then detect human body landmarks on unknown data from cameras and sensors.

The system’s output, though, is pretty simple — it produces an array of skeletons, each containing a set of joint landmarks. Here are the joints included in a typical skeleton structure:

LightBuzz Body Tracking Skeleton Model (34 landmarks)

The detected human body joints and landmarks.

A word on accuracy

Accuracy is essential for Body Tracking systems as it can impact the overall user experience. If the system is inaccurate, it can result in incorrect tracking or interpretation of the user’s movements, leading to frustration and reduced engagement. Moreover, we need a system capable of performing its magic in real-time with minimal lag. Typical cameras serve between 30 and 60 frames per second. Therefore, developers should opt for a Body Tracking system that combines high accuracy with low latency, such as the LightBuzz Body Tracking SDK. That’s precisely what I’ll be using during this tutorial.

Body Tracking SDK

The LightBuzz Body Tracking SDK is known for its high accuracy and low latency, making it an excellent choice for motion analysis. Companies, research centers, and universities use our SDK to develop commercial apps for desktop and mobile devices.

Learn more

Working with camera data

From a developer’s perspective, you can think of the system as a black box: you feed it with photos, videos, or live camera frames, and the system detects and combines the visible human body joints into meaningful skeleton structures. The process is straightforward:

The system opens the camera
The camera generates frames
Each frame contains color and skeleton data
Each skeleton contains joints
Each joint contains position and orientation information.

This is how we can use the LightBuzz Body Tracking SDK to open the camera, capture a frame, and detect the body skeleton data:

// Connect to the first available webcam.
Sensor camera = Sensor.Create(SensorType.Webcam);
camera.Open();
// Event raised whenever a new frame is available.
camera.FrameDataArrived += (sender, frame) =>
{
    List<Body> bodies = frame.BodyData;
};
// Close the camera when finished.
camera.Close();

Sensor camera = Sensor::Create(SensorType::Webcam);
camera.Open();
sensor.FrameDataArrived = [&](FrameData frame) {
    std::vector<Body> bodies = frame.body_data;
};
camera.Close();

As you can see, the LightBuzz SDK produces frame objects, each containing a list of detected bodies. We can loop into the list of skeletons and review their landmarks. Here’s how to access, e.g., the neck joint and capture its position, orientation, and tracking confidence.

foreach (Body body in bodies)
{
    Joint neck = body.Joints[JointType.Neck];
    
    // Screen position (X, Y in pixels).
    Vector2D position2D = neck.Position2D;
    // World position (X, Y, Z in meters).
    Vector3D position3D = neck.Position3D;
    // Orientation (X, Y, Z, W).
    Quaternion orientation = neck.Orientation;
    // Tracking confidence level (0 to 1).
    float confidence = neck.Confidence; 
}

for (Body& body : bodies)
{
    Joint neck = body.joints.at(JointType::Neck);
    
    // Screen position (X, Y in pixels).
    Vector2D position2D = neck.position2D;
    // World position (X, Y, Z in meters).
    Vector3D position3D = neck.position3D;
    // Orientation (X, Y, Z, W).
    Quaternion orientation = neck.orientation;
    // Tracking confidence level (0 to 1).
    float confidence = neck.confidence; 
}

That’s pretty easy, right? Now, let’s get back to our app. Since we’ll be measuring bicep curls, we need to focus on the movement of the arms and forearms in the 3D space. Here’s how to capture the required positions:

// Use the default body of the list.
Body body = bodies.Default();
// Capture the shoulder, elbow, and wrist joints.
Joint shoulder = body.Joints[JointType.ShoulderLeft];
Joint elbow = body.Joints[JointType.ElbowLeft];
Joint wrist = body.Joints[JointType.WristLeft];
// Find their positions in the 3D space.
Vector3D shoulder3D = shoulder.Position3D;
Vector3D elbow3D = elbow.Position3D;
Vector3D wrist3D = wrist.Position3D;
// Repeat for the right arm and forearm...

// Use the default body of the list.
Body body = Body::Default(bodies);
// Capture the shoulder, elbow, and wrist joints.
Joint shoulder = body.joints.at(JointType::ShoulderLeft);
Joint elbow = body.joints.at(JointType::ElbowLeft);
Joint wrist = body.joints.at(JointType::WristLeft);
// Find their positions in the 3D space.
Vector3D shoulder3D = shoulder.position3D;
Vector3D elbow3D = elbow.position3D;
Vector3D wrist3D = wrist.position3D;
// Repeat for the right arm and forearm...

If we use a graphics engine to visualize the joints in the 2D screen and the 3D world spaces, here’s what it would look like (I’m using Unity):

Applying Math formulas

So far, so good! We’ve captured the positions of the human joints in the 3D space. Now what? Well, it’s time to do the Math! As you move your forearm up and down, the angle between the shoulder, elbow, and wrist changes. When the wrist is down, the angle is approximately 180 degrees. As the wrist moves up, the angle decreases to about 30 degrees.

Measuring angles

How are we supposed to measure that angle? Unfortunately, most software developers haven’t encountered geometric problems since high school. Thankfully, that’s a problem mathematicians solved centuries ago: we’ll use the law of cosines. According to the law of cosines, if we know the lengths of the sides of a triangle, we can calculate the angle between any two sides!

“Wait, Vangos; we are doing AI and motion analysis – where’s the triangle?” In our case, the triangle is formed between the shoulder, elbow, and wrist joints! The angle we need is the one formed by the arm and forearm “sides.”

Here are the trigonometric formulas that describe the relations between the sides of the triangle and the angles. We’ll convert those formulas to code and use them in our app.

If you want to learn more about why these formulas are valid, check links 1 and 2 for some interactive examples. Of course, if you are in a hurry, you can always copy-paste the code below. Be my guest and steal it right away!

using System;
public double Angle(Vector3D start, Vector3D middle, Vector3D end)
{
    double ab = Math.Sqrt(
        Math.Pow(start.X - middle.X, 2) +
        Math.Pow(start.Y - middle.Y, 2) +
        Math.Pow(start.Z - middle.Z, 2));
    double cb = Math.Sqrt(
        Math.Pow(end.X - middle.X, 2) +
        Math.Pow(end.Y - middle.Y, 2) +
        Math.Pow(end.Z - middle.Z, 2));
    double ca = Math.Sqrt(
        Math.Pow(end.X - start.X, 2) +
        Math.Pow(end.Y - start.Y, 2) +
        Math.Pow(end.Z - start.Z, 2));
    if (ab == 0.0 || cb == 0.0) return 0.0;
    double angle = Math.Acos(
        (ab * ab + cb * cb - ca * ca) /
        (2 * ab * cb));
    return angle * 180.0 / Math.PI;
}

#include <cmath>
double Angle(Vector3D start, Vector3D middle, Vector3D end)
{
    double ab = std::sqrt(
        std::pow(start.X - middle.X, 2) +
        std::pow(start.Y - middle.Y, 2) +
        std::pow(start.Z - middle.Z, 2));
    double cb = std::sqrt(
        std::pow(end.X - middle.X, 2) +
        std::pow(end.Y - middle.Y, 2) +
        std::pow(end.Z - middle.Z, 2));
    double ca = std::sqrt(
        std::pow(end.X - start.X, 2) +
        std::pow(end.Y - start.Y, 2) +
        std::pow(end.Z - start.Z, 2));
    if (ab == 0.0 || cb == 0.0)
        return 0.0;
    double angle = std::acos(
        (ab * ab + cb * cb - ca * ca) /
        (2 * ab * cb));
    return angle * 180.0 / M_PI;
}

Now, we can call the Angle method and pass the coordinates of joints we captured earlier:

Measuring the angle is now as easy as calling the method we implemented above with the shoulder, elbow, and wrist coordinates we captured before.

double angle = Angle(shoulder, elbow, wrist);

const double angle = Angle(shoulder, elbow, wrist);

Counting repetitions

OK, we have the angle value in real-time, but how will we use this information to measure the reps? A bicep curl is a simple exercise where you bend your elbow and bring your hand close to your shoulder. To count how many times you do this, we need to define some criteria:

The start position is when your arm is fully extended and your elbow angle is close to 180 degrees. To account for small movement variations, we are going to use a threshold of 160.
The end position is when your arm is fully bent, and your elbow angle is close to the shoulder. That should be about 40 degrees.
A repetition is counted when you go from the start position to the end position and back.

To implement this logic, we must track the elbow angle over time. We also need to keep track of the movement direction (up or down) and avoid counting false positives (such as when you move your arm sideways).

Let’s start by defining the variables and thresholds.

// Forearm is down
const float MaxAngle = 160.0f;
// Forearm is up
const float MinAngle = 40.0f;
// Checks whether the wrist is going up (-> angle decreases)
bool _isDecreasing = false;
// The rep count
int _repCount = 0;

// Forearm is down
const float MaxAngle = 160.0f;
// Forearm is up
const float MinAngle = 40.0f;
// Checks whether the wrist is going up (-> angle decreases)
bool _isDecreasing = false;
// The rep count
int _repCount = 0;

The algorithm is pretty simple: it first checks if the angle has surpassed the lower threshold. This condition indicates that the arm is in the upward phase of a bicep curl. Upon meeting this condition, we signify that the angle is now expected to decrease in future frames.

Subsequently, the algorithm checks if the calculated angle is less than the predefined MinAngle and has been decreasing. This condition indicates that the arm is in the downward phase of a bicep curl, moving towards the body. Meeting this condition implies that the angle has decreased as expected, marking the completion of a bicep curl repetition. Consequently, _isDecreasing is set to false to signify that the angle is expected to increase again in the next upward phase.

double angle = Angle(shoulder, elbow, wrist);
if (angle > MaxAngle && !_isDecreasing)
{
    _isDecreasing = true;
}
else if (angle < MinAngle && _isDecreasing)
{
    _isDecreasing = false;
    _repCount++;
    Console.WriteLine($"Repetition count: {_repCount}");
}

double angle = Angle(shoulder, elbow, wrist);
if (angle > MaxAngle && !_isDecreasing)
{
    _isDecreasing = true;
}
else if (angle < MinAngle && _isDecreasing)
{
    _isDecreasing = false;
    _repCount++;
    std::cout << "Repetition count: " << _repCount << std::endl;
}

And this is it! If you run the demo, you’ll see that the counter increases as the user performs bicep curls. As homework, I’ll leave the UI implementation to you and Copilot, using the graphics engine of your choice. If you are curious, implement the equivalent rep count for the other arm. When you master this technique, move on to more challenging exercises, such as squats.

Bringing it all together

Here’s the result of what we accomplished today:

And here’s the complete code for your reference:

const float MaxAngle = 160.0f;
const float MinAngle = 40.0f;
bool _isDecreasing = false;
int _repCount = 0;
Sensor camera = Sensor.Create(SensorType.Webcam);
camera.Open();
camera.FrameDataArrived += (sender, frame) =>
{
    List<Body> bodies = frame.BodyData;
    Body body = bodies.Default();
    if (body != null)
    {
        Joint shoulder = body.Joints[JointType.ShoulderLeft];
        Joint elbow = body.Joints[JointType.ElbowLeft];
        Joint wrist = body.Joints[JointType.WristLeft];
        Vector3D shoulder3D = shoulder.Position3D;
        Vector3D elbow3D = elbow.Position3D;
        Vector3D wrist3D = wrist.Position3D;
        double angle = Angle(shoulder, elbow, wrist);
        if (angle > MaxAngle && !_isDecreasing)
        {
            _isDecreasing = true;
        }
        else if (angle < MinAngle && _isDecreasing)
        {
            _isDecreasing = false;
            _repCount++;
            Console.WriteLine($"Repetition count: {_repCount}");
        }
    }
};
camera.Close();

const float MaxAngle = 160.0f;
const float MinAngle = 40.0f;
bool _isDecreasing = false;
int _repCount = 0;
Sensor camera = Sensor::Create(SensorType::Webcam);
camera.Open();
sensor.FrameDataArrived = [&](FrameData frame) {
    std::vector bodies = frame.body_data;
    Body body = Body::Default(bodies);
    if (body)
    {
        Joint shoulder = body.joints.at(JointType::ShoulderLeft);
        Joint elbow = body.joints.at(JointType::ElbowLeft);
        Joint wrist = body.joints.at(JointType::WristLeft);
        Vector3D shoulder3D = shoulder.position3D;
        Vector3D elbow3D = elbow.position3D;
        Vector3D wrist3D = wrist.position3D;
        double angle = Angle(shoulder, elbow, wrist);
        if (angle > MaxAngle && !_isDecreasing)
        {
            _isDecreasing = true;
        }
        else if (angle < MinAngle && _isDecreasing)
        {
            _isDecreasing = false;
            _repCount++;
            Console.WriteLine($"Repetition count: {_repCount}");
        }
    }
};
camera.Close();

Did you like this article? If so, share it on social media to help your fellow developers! Have questions? Let me know in the comments below!

Summary

In this article, we explored the world of developing a virtual fitness trainer using AI and mathematics. Motion analysis and body tracking play key roles, enabling precise measurement and understanding of human movements. Leveraging tools like the LightBuzz Body Tracking SDK ensures high accuracy and low latency for an immersive fitness experience. By analyzing joint positions and applying mathematical principles, we can precisely track movements, count reps, and provide valuable feedback.

Here’s what we covered:

Virtual fitness trainer development using AI and mathematics.
Real-time body tracking (pose estimation) with the LightBuzz SDK.
Measuring angles.
Counting bicep curl repetitions.

Body tracking

LightBuzz has created the world’s most accurate body tracking software solution. Companies, research centers, and universities use our SDK to develop commercial apps for desktop and mobile devices.

Try it now

3 Comments

Jerry says:
May 22, 2023 at 3:24 am
Look very nice and stable. Is it any way to give a trail on lightbuzz SDK, it will help more people to evaluate it and pay for it. Thx. Really good tutorial.
- Vangos Pterneas says:
  May 22, 2023 at 5:25 pm
  Hi Jerry! You can definitely try the LightBuzz SDK! Just drop us a line here, and we’ll send you your free trial.
Jerry says:
May 23, 2023 at 4:58 am
Thx for your comment. Already send out and looking forward to give it a try. 🙂