Version 10 (modified by masc01, 9 years ago) (diff)


Building emotion-oriented systems with the SEMAINE API

The SEMAINE API is the component integration middleware created for the SEMAINE project, serving as the integration layer for system components in SEMAINE.

The SEMAINE API uses a message-oriented middleware (MOM) for all communication in the system. As a result, all communication is asynchronous, which decouples the various parts of the system. The actual processing is done in “components”, which communicate with one another over “Topics” below the named Topic*. Each component has its own “meta messenger”, which interfaces between the component and a central system manager. When a component is started, its meta messenger registers with the system manager over a special meta communication channel, the Topicsemaine.meta. At registration time, the meta messenger describes the component in terms of the data Topics that it sends data to and that it receives data from; if the component is an input or output component (in the sense of the user interface), that status is communicated as well. The system manager is keeping track of the components that have been registered, and checks at regular intervals whether all components are still alive by sending a “ping”. In reply to such a ping, each meta messenger confirms the respective component's status and sends debug information such as the average time spent processing incoming requests. The system manager keeps track of the information about registered components, and sends global meta messages informing all components that the overall system is ready or, if a component has an error or is stalled, that the system is not ready. Also, the system manager resets a global timer to zero when the system becomes ready. All components use this global time via their meta messenger, and thus can meaningfully communicate about timing of user and system events even across different computers with potentially unsynchronised hardware clocks.

A centralised logging functionality uses the Topics below semaine.log.*. By convention, messages are sent to semaine.log.<component>.<severity>, e.g. the Topic semaine.log.UtteranceInterpreter.debug would be used for debug messages of component UtteranceInterpreter. The severities used are “debug”, “info”, “warn” and “error”. Through this design, it is possible for a log reader to subscribe, e.g., to all types of messages from one component, or to all messages from all components that have at least severity “info”, etc. Furthermore, a configurable message logger can optionally be used to log certain messages in order to follow and trace them. Notably, it is possible to read log messages in one central place, independently of the computer, operating system or programming language used by any given component.

The following figure illustrates this system architecture. Components communicate with each other via Topics in the hierarchy (indicated by black arrows). Meta information is passed between each component's meta messenger and the system manager via the semaine.meta Topic (grey arrows). Optionally, components can write log messages, and a message logger can log the content messages being sent; a configurable log reader can receive and display a configurable subset of the log messages (dashed grey arrows).

Optionally, a system monitor GUI visualises the information collected by the system manager as a message flow graph. Input components are placed at the bottom left, output components at the bottom right, and the other components are sorted to the extent possible based on the data input/output relationships, along a half-circle from left to right. Component B comes later in the graph than component A if A's output is an input to B or if there is a sequence of components that can process A's output into B's input. This criterion is overly simplistic for complex architectures, especially with circular message flows, but is sufficient for simple quasi-linear message flow graphs. If a new component is added, the organisation of the flow graph is recomputed. This way, it is possible to visualise message flows without having to pre-specify the layout.

The following pages present three emotion-oriented example systems, in order to corroborate the claim that the SEMAINE API is easy to use for building new emotion-oriented systems out of new and/or existing components. Source code is provided in order to allow the reader to follow in detail the steps needed for using the SEMAINE API. The code is written in Java, and can be obtained from the SEMAINE sourceforge page [57]⁠. The SEMAINE API parts of the code would look very similar in C++.

A game driven by emotional speech: The swimmer's game

The third example system is a simple game application in which the user must use emotional speech to win the game. The game scenario is as follows. A swimmer is being pulled backwards by the stream towards a waterfall (Figure 19). The user can help the swimmer to move forward towards the river bank by cheering him up through high-arousal speech. Low arousal, on the other hand, discourages the swimmer and drives him more quickly to the waterfall.

The system requires the openSMILE components as in the Emotion mirror system; a component computing the swimmer's position as time passes, and considering the user's input; and a rendering component for the user interface. Furthermore, we will illustrate the use of TTS output in the SEMAINE API by implementing a commentator providing input to the speech synthesis component of the SEMAINE system 1.0 (Section 4.2).

The PositionComputer (Figure 20) combines areact()and anact()method. Messages are received via an EMMA receiver and lead to a change in the internal parameterposition(l. 22). Theact()method implements the backward drift (l. 29) and sends regular position updates (l. 30) as a plain-text message.

The SwimmerDisplay (Figure 21) implements the user interface shown in Figure 19. Its messaging part consist of a simple text-based Receiver (l. 5) and an interpretation of the text messages as single float values (l. 10).

Due to the separation of position computer and swimmer display, it is now very simple to add a Commentator component (Figure 22) that generates comments using synthetic speech, as a function of the current position of the swimmer. It subscribes to the same Topic as the SwimmerDisplay (l. 7), and sends BML output (l. 2) to the Topic serving as input to the speech synthesis component of the SEMAINE system 1.0 [47]⁠. Speech output is produced when the game starts (l. 18-20) and when the position meets certain criteria (l. 13-14). Generation of speech output consists in the creation of a simple BML document with a<speech>tag enclosing the text to be spoken (l. 25-28), and sending that document (l. 29).

The complete system consists of the Java components SystemManager, PositionComputer, SwimmerDisplay, Commentator, SpeechBMLRealiser and SemaineAudioPlayer, as well as the external C++ component openSMILE. The resulting message flow graph is shown in Figure 23.


This page is based on the following paper (an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited).

Schröder, M. (2010). The SEMAINE API: Towards a standards-based framework for building emotion-oriented systems. Advances in Human-Machine Interaction, Volume 2010, Article ID 319406. doi:10.1155/2010/319406.

It is one advantage of open access publishing that the information in the journal paper can be developed and updated on this wiki page.

Attachments (3)

Download all attachments as: .zip