Speech Recognition in Unity

Voice Recognition is one of the hottest trends in the era of Natural User Interfaces. When used wisely, speech recognition is an effective and intuitive means of communication. As Virtual and Augmented Reality emerge, voice recognition is becoming a vital communication method between the human and the computer. Microsoft is now bringing Cortana, the high-tech virtual assistant, to every platform, including PCs, phones, XBOX, and, of course, HoloLens.

Today, I’ll show you how to develop a simple and open-source voice recognition game using C# and Unity3D. Here’s the end result:

To run the project, you’ll need the following software components:

• Unity3D 5.5 or later
• Visual Studio 2015 or later
• Windows 10

In my setup, I am using Unity 2019 with Visual Studio Community 2019.

Source code

The source code of the project is available in our LightBuzz GitHub account. Feel free to download, fork, and even extend it!

Step-by-step tutorial

To demonstrate the speech recognition capabilities, we’ll create a very simple demo. In a nutshell, here’s what we are going to do:

The user will be prompted to move a visual element on the screen. To move the element, she’ll need to say the direction: up, down, left, or right. The app will be listening for voice commands and will change the direction of the object when a voice command is recognized.

Step #1 – Create the User Interface

Launch Unity, create a new project, and add the following elements inside a Unity Canvas:

• A background image
• A movable object (in our case, a bee)
• Some text prompts

Specify the desired keywords

A speech recognition engine works best when you feed it with as few words as possible. Instead of searching the whole spectrum of the English language, we shall limit it to a small range. For our game, we need only four keywords: up, down, left, and right.

We can also specify the confidence level of the speech recognition engine. The confidence level is a value that indicates how ambiguous words should be treated. Use Medium or High for native English speakers, and Low for non-native speakers.

The third variable is the speed of the object.

All of these members are declared public, so you can edit them right into the Unity Editor.

public string[] keywords = new string[] { "up", "down", "left", "right" };
public ConfidenceLevel confidence = ConfidenceLevel.Medium;
public float speed = 1.0f;

Finally, we declare a variable for the word that was recognized:

protected string word = "right";

Step #3 – Use the Keyword Recognizer

Now, I would like to introduce you to the KeywordRecognizer class. The Keyword Recognizer encapsulates the voice recognition engine.

When the application starts, we initialize an instance of the recognizer by providing the keywords and confidence level to its constructor. Whenever a word is recognized, the OnPhraseRecognized event handler shall be called. To enable voice recognition, you need to call the Start() method:

private void Start()
{
    if (keywords != null)
    {
        recognizer = new KeywordRecognizer(keywords, confidence);
        recognizer.OnPhraseRecognized += Recognizer_OnPhraseRecognized;
        recognizer.Start();
    }
}

Remember to call the Stop() method when the application closes:

private void OnApplicationQuit()
{
    if (recognizer != null && recognizer.IsRunning)
    {
        recognizer.OnPhraseRecognized -= Recognizer_OnPhraseRecognized;
        recognizer.Stop();
    }
}

Here is the OnPhraseRecognized event handler. The PhraseRecognitionEventArgs parameter provides us with the exact text that was recognized:

private void Recognizer_OnPhraseRecognized(PhraseRecognizedEventArgs args)
{
    word = args.text;
    results.text = "You said: <b>" + word + "</b>";
}

Step #4 – Add some game logic

Since we know the recognized word, we can now move the visual element on our Canvas. All we need to do is change the position of the game object:

private void Update()
{
    var x = target.transform.position.x;
    var y = target.transform.position.y;
    switch (word)
    {
        case "up":
            y += speed;
            break;
        case "down":
            y -= speed;
            break;
        case "left":
            x -= speed;
            break;
        case "right":
            x += speed;
            break;
    }
    target.transform.position = new Vector3(x, y, 0);
}

Step #5 – Player Settings

This is a tricky one and it can be easily missed. To enable voice commands in your Windows app, you need to add the Microphone capability under the Player Settings.

• Go to File → Build Settings
• Select Player Settings
• Click the green Windows Store icon
• Find the Capabilities tab
• Check Microphone

Step #6 – Export

KeywordRecognizer is available for Windows Standalone and Windows Store Universal (Windows 10). The exported project will also work in Windows Store 8.1, however, the speech recognition features will not be available in Windows 8.1.

Of course, the API is compatible with Universal Windows Platform, so the exported app can run on:

• PC
• Phone
• XBOX
• HoloLens

The app can run offline, too, using Cortana’s speech infrastructure. Unlike Siri, no active Internet connection is required.

Summary

In this tutorial, we learned how to use Unity’s KeywordRecognizer to understand voice commands and integrate them into a Windows application.

Source code

You made it to this point? Awesome! Here is the source code for your convenience.