SEMAINE-3.1: The final Sensitive Artificial Listener system
Released 13 December 2010.
Sensitive Artificial Listeners (SAL) are virtual dialogue partners who, despite their very limited verbal understanding, intend to engage the user in a conversation by paying attention to the user's emotions and non-verbal expressions. The SAL characters have their own emotionally defined personality, and attempt to drag the user towards their dominant emotion, through a combination of verbal and non-verbal expression.
The SEMAINE project has created an open-source implementation of fully autonomous SAL characters. It combines state-of-the-art emotion-oriented technology into a real-time interactive system. The SAL characters register your voice from a microphone, using a combination of speech recognition and vocal emotion recognition. They perceive your facial expressions and head movements through a video camera. They express themselves through a virtual animated face and a synthetic expressive voice. This video illustrates the concept.
New in version 3.1
This is essentially a cleanup release of the full SAL system. Its main emphasis is on improving the stability and performance, and simplifying the configuration of the system in order to make it easier to reuse the system in whole or in part for scientific research in real-time interactive, emotionally aware Embodied Conversational Agents.
This version of the system has the following new feature:
- User presence is detected automatically, based on face and voice detection.
- Embedded middleware: The middleware ActiveMQ is now started as part of the java components; no need to start a separate ActiveMQ server anymore.
Apart from this, all components have received minor improvements under the surface.
In total, 28 issues have been fixed: http://semaine.opendfki.de/query?status=closed&group=resolution&milestone=3.1
The open source parts of the system can be downloaded from the SEMAINE sourceforge project page. You can choose among two packages, each around 950 MB. (The files are so large because it includes four high-quality TTS voices as well as vocal emotion recognition models, which need a lot of space.)
- SEMAINE-3.1-windows includes binary versions of the full SEMAINE-3.1 system: the System manager component, the dialogue components, the speech synthesizer MARY TTS, the Greta agent components, the Opensmile speech analysis components, and the message-oriented middleware ActiveMQ.
- SEMAINE-3.1-source includes the source code for: the System manager component, the dialogue components, and the Opensmile speech analysis components, as well as binary versions of the speech synthesizer MARY TTS, the message-oriented middleware ActiveMQ and all dependencies needed to compile the code. It can be used to compile the system from source under Linux, Mac and Windows.
To run the full audio-visual SEMAINE-3.1 system, the windows package is required. The system can run on a single, fast machine (tested on a laptop with a 2.53 GHz Core2Duo CPU with 4 GB RAM), or you can set up SEMAINE-3.1 as a distributed system.
The Video analysis components are distributed as closed-source freeware from the website of the Intelligent Behaviour Understanding Group at Imperial College London: http://ibug.doc.ic.ac.uk/resources/semaine-visual-components-20092010/. Watch out for the Camera driver requirements if you are using a Firewire camera. If installed in the default location, the start.bat script will notice that the video analysis components are installed and will try to run them. Since they are computationally heavy, you may need an additional computer to run them.
The SEMAINE-3.1 system will work without the video analysis components, but will then not be able to pick up the same amount of information from the user.
Running the system
In its simplest form, the system can be run on a single (fast) Windows machine by installing all system components on the same computer as described above. The system is then run by starting the following batch file:
This will start all installed components. If the system does not start correctly, double-check that you have met all the requirements.
To stop all components of the system, call
The system with windows and java open source components runs OK on a Core2Duo with 2.53 GHz and 4 GB RAM. When the video analysis components are added, the system is running but very slow. Therefore, it is recommended to run SEMAINE-3.1 as a distributed system on several computers.
Whereas the video input and output components are available for Windows only, there is a configuration of the system that runs on all platforms -- the speech2speech system, with only speech input and speech output.
This system can be started as follows.
- In one shell, start the java components as SEMAINE-3.1/bin/semaine-speech2speech.sh (or .bat);
- In a second shell, start opensmile as SEMAINE-3.1/bin/run_components/start_component_tum.opensmile (linux/mac) or SEMAINE-3.1\Opensmile\start_openSMILE.bat (windows).
In this configuration the java components are started with a different config file, which loads a stack of audio-only output components in java.
The SEMAINE API for Java and C++, the SEMAINE dialogue components (in Java), and the speech synthesizer MARY TTS are distributed under the GNU Lesser General Public License (LGPL), version 3. The speech synthesis voices for the SAL agents are distributed under the Creative Commons ShareAlike - No Derivatives license.
The 3D agent animation software Greta and the speech analysis software Opensmile are distributed under the GNU General Public License (GPL).
The separately installable SEMAINE Video components for camera image analysis come as a freeware binary.
With some effort you can build the components from source.
Detailed documentation of the SEMAINE API is available in a number of documents:
- Online Developer documentation on the wiki;
- the deliverable report D7e, which is a snapshot of online documentation as of December 2010;
- A Journal article in Advances in Human-Machine Interaction The SEMAINE API: Towards a standards-based framework for building emotion-oriented systems;
- the deliverable report D1d Final SAL system
- the deliverable report D1c First full-scale SAL system
- section 3 of the deliverable report D1b: First integrated system
- Javadoc is available for the Java version of the SEMAINE API
- Doxygen is available for the C++ version of the SEMAINE API
There is a public SEMAINE-users mailing list at https://lists.sourceforge.net/lists/listinfo/semaine-users. Feel free to ask questions there.
Detailed information on the system and the underlying software architecture can be found in the following set of public project deliverable reports:
- D1d Final SAL system
- D2b Final feature extraction
- D3c Human behaviour interpreter
- D4b Final dialogue manager
- D5b Multimodal generation component
See also the slightly older set of reports on SEMAINE-2.0, which contain complementary information.
Furthermore, information about the data collected in the project can be found in the following reports: