Voice Recognition is one of the hottest trends in the era of Natural User Interfaces. When used wisely, speech recognition is an effective and intuitive means of communication. As Virtual and Augmented Reality emerge, voice recognition is becoming a vital communication method between the human and the computer. Microsoft is now bringing Cortana, the high-tech virtual assistant, to every platform, including PCs, phones, XBOX, and, of course, HoloLens.

Today, I’ll show you how to develop a simple and open-source voice recognition game using C# and Unity3D. Here’s the end result:

Source Code

The source code of the project is available in our LightBuzz GitHub account. Feel free to download, fork, and even extend it.

Download Source Code

Prerequisites

To run the project, you’ll need the following software components:

  • Unity3D 5.5+
  • Visual Studio 2015+
  • Windows 10

Step-by-Step Tutorial

To demonstrate the speech recognition capabilities, we’ll create a very simple demo. In a nutshell, here’s what we are going to do:

The user will be prompted to move a visual element on the screen. To move the element, she’ll need to say the direction: up, down, left, or right. The app will be listening for voice commands and will change the direction of the object when a voice command is recognized.

Step #1 – Create the User Interface

Launch Unity, create a new project, and add the following elements inside a Unity Canvas:

  • A background image
  • A movable object (in our case, a bee)
  • Some text prompts

Speech Recognition Game UIStep #2 – Specify the desired keywords

A speech recognition engine works best when you feed it with as few words as possible. Instead of searching the whole spectrum of the English language, we shall limit it to a small range. For our game, we need only four keywords: up, down, left, and right.

We can also specify the confidence level of the speech recognition engine. The confidence level is a value that indicates how ambiguous words should be treated. Use Medium or High for native English speakers, and Low for non-native speakers.

The third variable is the speed of the object.

All of these members are declared public, so you can edit them right into the Unity Editor.

public string[] keywords = new string[] { "up", "down", "left", "right" };
public ConfidenceLevel confidence = ConfidenceLevel.Medium;
public float speed = 1;

Finally, we declare a variable for the word that was recognized:

protected string word = "right";

Step #3 – Use the Keyword Recognizer

Now, I would like to introduce you to the KeywordRecognizer class. The Keyword Recognizer encapsulates the voice recognition engine.

When the application starts, we initialize an instance of the recognizer by providing the keywords and confidence level to its constructor. Whenever a word is recognized, the OnPhraseRecognized event handler shall be called. To enable voice recognition, you need to call the Start() method:

private void Start()
{
    if (keywords != null)
    {
        recognizer = new KeywordRecognizer(keywords, confidence);
        recognizer.OnPhraseRecognized += Recognizer_OnPhraseRecognized;
        recognizer.Start();
    }
}

Remember to call the Stop() method when the application closes:

private void OnApplicationQuit()
{
    if (recognizer != null && recognizer.IsRunning)
    {
        recognizer.OnPhraseRecognized -= Recognizer_OnPhraseRecognized;
        recognizer.Stop();
    }
}

 

Here is the OnPhraseRecognized event handler. The PhraseRecognitionEventArgs parameter provides us with the exact text that was recognized:

private void Recognizer_OnPhraseRecognized(PhraseRecognizedEventArgs args)
{
    word = args.text;
    results.text = "You said: <b>" + word + "</b>";
}

Step #4 – Add some game logic

Since we know the recognized word, we can now move the visual element on our Canvas. All we need to do is change the position of the game object:

private void Update()
{
    var x = target.transform.position.x;
    var y = target.transform.position.y;

    switch (word)
    {
        case "up":
            y += speed;
            break;
        case "down":
            y -= speed;
            break;
        case "left":
            x -= speed;
            break;
        case "right":
            x += speed;
            break;
    }

    target.transform.position = new Vector3(x, y, 0);
}

Step #5 – Player Settings

This is a tricky one and it can be easily missed. To enable voice commands in your Windows app, you need to add the Microphone capability under the Player Settings.

  • Go to File → Build Settings
  • Select Player Settings
  • Click the green Windows Store icon
  • Find the Capabilities tab
  • Check Microphone

Step #6 – Export

KeywordRecognizer is available for Windows Standalone and Windows Store Universal (Windows 10). The exported project will also work in Windows Store 8.1, however, the speech recognition features will not be available in Windows 8.1.

Speech Recognition Unity Export

Of course, the API is compatible with Universal Windows Platform, so the exported app can run on:

  • PC
  • Phone
  • XBOX
  • HoloLens

The app can run offline, too, using Cortana’s speech infrastructure. Unlike Siri, no active Internet connection is required.

Speech Recognition - Unity GameSummary

In this tutorial, we learned how to use Unity’s KeywordRecognizer to understand voice commands and integrate it into a Windows Store application.

Ready to transform your own business using Speech Recognition?

Start a project with us
Michail Moiropoulos

Author Michail Moiropoulos

Michail is a Unity Specialist with significant experience in game development and motion technology. His professional experience includes Kinect, HoloLens, Oculus, Leap Motion, and RealSense.

More posts by Michail Moiropoulos

Join the discussion 19 Comments

  • Blair Adams says:

    This looks amazing! Thanks for sharing. Do you know if this would be compatible with a Vive application?

    • Hi Blair! Thanks for your kind words and support.
      Speech recognition is part of Unity and it runs on the application side, while VR is only a “projection on a headset monitor”. I strongly believe that running on Vive won’t be an issue. After all it’s a PC game and all you need is a microphone!

  • Blair Adams says:

    Ahh. Ok. I do remember seeing something about speech recognition a while back in Unity. I’m just surprised we haven’t seen more of it turn up inside of VR yet. Maybe the processing is heavy? I’ll have to try an experiment or two. Thanks.

  • Jake Aquilina says:

    Hello i seem to have an issue, im running unity 5.6 and windows 10, trying to run it in the editor and getting this error

    UnityException: Speech recognition is not supported on this machine.
    UnityEngine.Windows.Speech.PhraseRecognizer.CreateFromKeywords (System.String[] keywords, ConfidenceLevel minimumConfidence) (at C:/buildslave/unity/build/artifacts/generated/common/runtime/SpeechBindings.gen.cs:43)
    UnityEngine.Windows.Speech.KeywordRecognizer..ctor (System.String[] keywords, ConfidenceLevel minimumConfidence) (at C:/buildslave/unity/build/Runtime/Export/Windows/Speech.cs:221)
    UnityEngine.Windows.Speech.KeywordRecognizer..ctor (System.String[] keywords) (at C:/buildslave/unity/build/Runtime/Export/Windows/Speech.cs:201)

  • Brian says:

    Hello, I have an issue when running the source code on Unity 5.6.0f3 and windows 10. The error is

    UnityException: Speech recognition is not supported on this machine.
    UnityEngine.Windows.Speech.PhraseRecognizer.CreateFromKeywords (System.String[] keywords, ConfidenceLevel minimumConfidence) (at C:/buildslave/unity/build/artifacts/generated/common/runtime/SpeechBindings.gen.cs:47)
    UnityEngine.Windows.Speech.KeywordRecognizer..ctor (System.String[] keywords, ConfidenceLevel minimumConfidence) (at C:/buildslave/unity/build/Runtime/Export/Windows/Speech.cs:221)
    SpeechRecognitionEngine.Start () (at Assets/Scripts/SpeechRecognitionEngine.cs:23)

    And I had enabled the Microphone capability in Player Settings like this (image:http://imgur.com/a/X0mjC). But the error still occurred.

  • Massimo Fattorusso says:

    how one can set language? is inherited from the language of the unity project?

  • Amjad says:

    It can be applied to Android phones?

  • […] but Michael Moiropoulos at Lightbuzz did a much better tutorial on this than I could manage, so head there for details on integrating Unity and Windows 10 speech […]

  • Andrew says:

    I tried to run this project and it seems that Unity does not recognize my microphone ( or something like that) because it does not get any command. Any idea why and how I can fix that?

Leave a Reply