Furhat Software is the Conversational Operating System that runs on the robot, bringing together state of the art systems in speech recognition, computer vision, speech synthesis, animation and dialogue management.

Low latency conversations

All of Furhat‘s components interact together using an elegant and extendible Events system 

Vision
SenseUserEnter User appearing in front of Furhat
SenseUserMove
SenseUserLeave
Speech synthesis
SenseUserEnter ”Hello there” MonitorSpeechEnd
”Goodbye”
Speech recognition
ActionListen Listening
SenseSpeechStart ”Hi”
Furhat gaze
ActionGesture Looking up
Shifting gaze
Looking down
Furhat face
ActionGesture Brows raise MonitorGestureEnd
ActionGesture Smiling ActionListen

Advanced user tracking and perception

Furhat’s computer vision system is the most advanced and robust system in any social robot. 

Furhat‘s computer vision system provides real-time multi-user tracking, facial expression analysis, user distance and head pose estimation. This includes:

  • Highly accurate face detection system based on state-of-the-art deep learning (single-shot-detector) model
  • Accurate multi-user tracking, depth estimation, and spatial modelling
  • Supports face recognition using DNN-based face model, allows keeping track of users over time

Dialogue modelling

Conversations are modelled into State-Machines that are hierarchical and context-sensitive

Key features of Furhat‘s dialogue platform include the ability to:

  • Define powerful NLU modules, including generic intents and behaviours
  • Define intents over different hierarchies to allow interactions to scale very quickly
  • Define complex behaviours in reusable states as templates to reduce development time
Dialogue modelling Conversations are modelled into State-Machines that are hierarchical…

Context-aware understanding of input

Powerful multi-lingual natural language understanding engine 

Furhat‘s in-house NLU engine, coupled with the dialogue platform, allows for:

  • Definition of multiple intents in each utterance
  • Intent classification is dynamically based on the current dialogue context
  • Entities can be programmatically defined and extended using dictionaries, grammars, knowledge graphs, wildcards, etc.
  • The Furhat Software comes with pre-defined common intents and entities for Date, Time, etc.
Context-aware understanding of input Powerful multi-lingual natural language understanding engine 

Software Features and Specifications

Computer vision

  • Immediate face detection
  • Tracking of multiple faces
  • Users’ 3D location estimation
  • Users’ head rotation estimation
  • Facial expression analysis of users
  • Short-term memory of faces (for re-identification within the same interaction)
  • Ability to plug-in additional vision modules using the SDK

Speech Recognition

Support for more than 120+ languages using:

  • Google Cloud Speech-To-Text
  • Microsoft Azure Speech-To-Text
  • Support for utterances up to 60 seconds

Microphones and Audio Processing

Using Furhat‘s 4-mic array:

  • Speaker localisation
  • Noise cancellation
  • Echo cancellation

Speech Synthesis

Support for more than 30 languages and 200 voices using:

  • Amazon Polly cloud TTS (29 languages, 60 voices)
  • Acapela on-board TTS (35 languages, 136 voices including children, elderly voices, and regional accents)

Face Animation Engine

  • >35 muscles in the face
  • Automatic lip synchronisation
  • 20+ built-in faces
  • 20+ built-in gestures

Face Customisation

  • Ability to change textures
  • Ability to create and add new face textures
  • Ability to plug-in your own face and lip animation engine
  • Support for non-human faces including additional robot masks

External screen support

The officially supported monitors are:

  • ELO 1502L – Full HD (1080p version)
  • Dell P2418HT 24” Touch

More details

Download the Technical Product Overview for extensive details including Furhat‘s hardware, software and SDK

Discover the Furhat SDK Design your own multi-modal and multi-party…

Discover the Furhat SDK

Design your own multi-modal and multi-party conversational applications with Furhat‘s SDK

Learn more