Furhat Software

The world’s most powerful software to bring robots to life

Furhat Software is the Conversational Operating System that runs on the robot, bringing together state of the art systems in speech recognition, computer vision, speech synthesis, animation and dialogue management.

Low latency conversations

All of Furhat‘s components interact together using an elegant and extendible Events system

Vision

SenseUserEnter User appearing in front of Furhat

SenseUserMove

SenseUserLeave

Speech synthesis

SenseUserEnter ”Hello there” MonitorSpeechEnd

”Goodbye”

Speech recognition

ActionListen Listening

SenseSpeechStart ”Hi”

Furhat gaze

ActionGesture Looking up

Shifting gaze

Looking down

Furhat face

ActionGesture Brows raise MonitorGestureEnd

ActionGesture Smiling ActionListen

Advanced user tracking and perception

Furhat’s computer vision system is the most advanced and robust system in any social robot.

Furhat‘s computer vision system provides real-time multi-user tracking, facial expression analysis, user distance and head pose estimation. This includes:

Highly accurate face detection system based on state-of-the-art deep learning (single-shot-detector) model
Accurate multi-user tracking, depth estimation, and spatial modelling
Supports face recognition using DNN-based face model, allows keeping track of users over time

Dialogue modelling

Conversations are modelled into State-Machines that are hierarchical and context-sensitive

Key features of Furhat‘s dialogue platform include the ability to:

Define powerful NLU modules, including generic intents and behaviours
Define intents over different hierarchies to allow interactions to scale very quickly
Define complex behaviours in reusable states as templates to reduce development time

Dialogue modelling Conversations are modelled into State-Machines that are hierarchical…

Context-aware understanding of input

Powerful multi-lingual natural language understanding engine

Furhat‘s in-house NLU engine, coupled with the dialogue platform, allows for:

Definition of multiple intents in each utterance
Intent classification is dynamically based on the current dialogue context
Entities can be programmatically defined and extended using dictionaries, grammars, knowledge graphs, wildcards, etc.
The Furhat Software comes with pre-defined common intents and entities for Date, Time, etc.

Context-aware understanding of input Powerful multi-lingual natural language understanding engine

Software Features and Specifications

Computer vision

Immediate face detection
Tracking of multiple faces
Users’ 3D location estimation
Users’ head rotation estimation
Facial expression analysis of users
Short-term memory of faces (for re-identification within the same interaction)
Ability to plug-in additional vision modules using the SDK

Speech Recognition

Support for more than 120+ languages using:

Google Cloud Speech-To-Text
Microsoft Azure Speech-To-Text
Support for utterances up to 60 seconds

Microphones and Audio Processing

Using Furhat‘s 4-mic array:

Speaker localisation
Noise cancellation
Echo cancellation

Speech Synthesis

Support for more than 35 languages and more than 200 voices using:

Amazon Polly cloud TTS (60 voices for 29 languages)
Acapela on-board TTS (35 languages, 136 voices including children, elderly voices, and regional accents)

Face Animation Engine

>35 muscles in the face
Automatic lip synchronisation
20+ built-in faces
20+ built-in gestures

Face Customisation

Ability to change textures
Ability to create and add new face textures
Ability to plug-in your own face and lip animation engine
Support for non-human faces including additional robot masks

External screen support

The officially supported monitors are:

ELO 1502L – Full HD (1080p version)
Dell P2418HT 24” Touch

More details

Download the Furhat Owners guide for extensive details including Furhat‘s hardware, software, and SDK

Discover the Furhat SDK

Design your own multi-modal and multi-party conversational applications with Furhat‘s SDK

Learn more