Tutorial
Conversational interaction with social robots
Background
Spoken face-to-face communication is likely to be the most important means of interaction with robots in the future. In addition to speech technology, this also requires the use of visual information in the form of facial expressions, lip movement and gaze. Human-robot interaction is also naturally situated, which means that the situation in which the interaction takes place is of importance. In such settings, there might be several speakers involved (multi-party interaction), and there might be objects in the shared space that can be referred to. Recent years have seen an increased interest in modeling such communication for human-robot interaction.
However, modelling conversational interaction between humans and robots is non-trivial. Most platforms for conversational AI are targeted towards either written interaction (chatbots) or voice assistants, based on command-and control or question-answering dialogues. For multi-party, multi-modal conversational interaction, other aspects need to be addressed. Thus, the communities of human-robot interaction on the one hand, and conversational AI on the other are currently quite separated. In this tutorial, we would like to bridge that gap.
Tutorial overview
In this tutorial, we will start by providing the theoretical background of spoken face-to-face interaction and how this applies to human-robot interaction. We will then go through the state-of-the-art of the different technologies needed and how this kind of interaction can be modelled. To make the tutorial as concrete as possible, we will use the Furhat platform in order to show how different interaction patterns can be implemented, and (depending on the number of participants) give hands-on exercises on how to program human-robot interaction for a social robot. For this, we will use the Furhat SDK, which contains a virtual (3D animated) robot, which is generally available for anyone to download and supports all major operating systems.
Target audience
The ideal audience for this tutorial is academics and researchers from within the social robotics and conversational AI community, as well as those with an interest in developing conversational interactions with virtual agents.
Tutorial requirements
In the tutorial, you will be using the Virtual Furhat and the Blockly visual programming interface. To be able to participate in the tutorial we would like you to make sure you have the Virtual Furhat and the Blockly tool up and running. It should not take you more than 10-20 minutes to do this.
Download & Links
Here are the files you need to get going.
1. Request the Furhat SDK
Send us a request for access to the SDK. Please note it may take up to two working days to complete the request, but we aim to provide access as soon as possible.
2. Follow these video instructions:
SDK Setup Tutorial: Introduction to the Furhat SDK
SDK Setup Tutorial: Create an Account
SDK Setup Tutorial: Install and Test
SDK Setup Tutorial: Run a Skill
SDK Setup Tutorial: Run Blockly
Written instructions can be found here.
3. Run and test Blockly
Download it from furhat.io/downloads
Video instruction: Quick intro of Blockly
To get more acquainted with blockly you can check out a more comprehensive video of the Blockly visual programming interface.
Introduction to Conversational Interaction with Social Robots
In this lecture, we will give an overview of how conversational interaction with social robots is different from other forms of spoken interaction, such as interaction with smart speakers. We will also give a brief introduction to the different components needed for such interaction, and some of the different use cases for conversational social robots.
Please allow cookies to watch this video.
How to program conversational interactions with the Furhat SDK
In this lecture, we will give an introduction to the Furhat SDK and the different ways of programming interactions (by coding or graphical programming). We will briefly cover the dialog flow, NLU, user management, and Wizard-of-oz.
Please allow cookies to watch this video.
The ‘hands-on’ work
In this 1 hour tutorial, you will be guided through a tutorial on how to create a simple skill using the Blockly visual programming interface.
You can do the tutorial on your own from the written instructions or do a code-along with the video below.
The tutorial is targeted towards novices in human-robot interactions and no programming skills are required. If you are more experienced in creating human-robot interactions, you can skip ahead and jump to later in the interaction where it perhaps starts to get more interesting for you.
Download the step-by-step blockly flow files and some other example flows here.
For the participants that don’t mind writing code, we also encourage you to explore the written tutorials in our Developer Zone.
However that coding adventure might require more than one hour of your time, and we haven’t prepared a code-along video.
Best of luck!
Please allow cookies to watch this video.
Need support?
We’ve set up a channel in our Open Community Slack space. There you can ask questions and perhaps connect with other participants. Send us a request – and we’ll make sure you are invited to the slack channel.