Protocol for the Player in SEMAINE
Any player component in SEMAINE must follow the following protocol so that it supports ahead-of-time preparation of possible utterances. The player must keep a collection of "Animations" which can be played by a "playCommand".
This protocol is currently implemented by two players: The audio-visual Windows native PlayerOgre using the Greta agent, and the speech-only player in Java class QueuingAudioPlayer.
Low-level player data is sent to the player via the Topics
semaine.data.synthesis.lowlevel.audio semaine.data.synthesis.lowlevel.video.FAP semaine.data.synthesis.lowlevel.video.BAP
Incoming messages have the following properties:
- a message type specific to the payload format (currently: BytesMessage for audio, TextMessage for FAP and BAP).
- a data type (obtained by message.getDatatype()) identifying the type of message (current values are "AUDIO", "FAP" and "BAP").
- a content ID and a content creation time (obtained by message.getContentID() and message.getContentCreationTime()) which are used to assemble an Animation, to match data and command messages, and to identify the content item in callback and log messages.
- optionally, a contentType, such as "utterance" or "listener-vocalisation".
The idea is that a unit of player data (an "Animation") is assembled in the player from the individual data items that are coming in (currently, AUDIO, FAP and BAP). Certain data types are optional (currently: AUDIO). A message can either contain the complete data of the given type (currently the case for AUDIO) or it can contain a chunk of data (currently the case for FAP and BAP). A chunk contains information about its position in the Animation; it can be dynamically added even if the Animation is already playing.
There are two types of command messages: messages with data typesdataInfoandplayCommand.
- Data info commands: For a given content ID, define the data types that must be present in the Animation: HASAUDIO, HASFAP and HASBAP. Each can be 0 for "not needed" or 1 for "needed".
- Player commands: For a given content ID, define the playback conditions. This includes the following aspects:
- STARTAT: when to start the playback of the Animation (in milliseconds from the moment when the Animation becomes ready);
- PRIORITY: the priority of the Animation in case of competing Animations;
- LIFETIME: the lifetime of the Animation, counting from the moment when the animation becomes ready. When the lifetime is exceeded and the animation has not started playing, it will be marked as "dead" and removed.
Commands are sent to topic
and have the data type dataInfofor data info commands and playCommand for player start trigger commands.
For every content ID, a playCommand is required in order to play that animation. Without a matching playCommand, an animation will never be played.
A command has the following format:
- its content ID is identical to the content ID of the Animation for which it defines playback conditions;
- message format is TextMessage; the text consists of space-separated key-value pairs, one pair per line, where the keys are strings and the values are floating point numbers.
The following features are used:
- for playCommand:
- STARTAT (0 means start at the moment when all required parts are present, a positive number means milliseconds after that condition is met)
- LIFETIME (in milliseconds from the moment the animation is triggered; -1 means it will never expire)
- PRIORITY (a value between 0 and 1, where 0 is the lowest and 1 the highest possible priority)
- for dataInfo:
- HASAUDIO (a binary feature, 0 means the Animation does not have audio, 1 means the Animation has audio data)
- HASFAP (a binary feature, 0 means the Animation does not have FAP data, 1 means the Animation has FAP data)
- HASBAP (a binary feature, 0 means the Animation does not have BAP data, 1 means the Animation has BAP data)
Every player command must contain all features of its respective type.
Event-based callback messages are sent when certain conditions are met for a given Animation. The messages go to Topic
and have the following format:
<callback xmlns="http://www.semaine-project.eu/semaineml"> <event id="CONTENT_ID" contentType="CONTENT_TYPE" time="META_TIME" type="EVENT_TYPE"/> </callback>
where content ID and meta time are like before, and type is one of the following:
- ready means the Animation has received all required data, so it is ready for playing back. This event is triggered independently of the question whether a command has been received or not.
- deleted means the Animation was removed before it started playing, e.g. because it has exceeded its lifetime in the output queue.
- start means the Animation has started playing.
- stopped means the Animation was stopped while playing but before it was finished, e.g. because a request to change character was received.
- end means the Animation has finished playing.
The contentType attribute is present only if the incoming messages had a content type, and reproduces that content type.
The content ID must be unique for the lifetime of a system. This leads to the following error conditions.
It is an error condition...
- if a data chunk is received for an Animation that has already been discarded (because it finished playing, or exceeded its lifetime in the queue);
- if data is received for a data type that does not form chunks;
- if a playCommand is received for a content ID that has been started already, or that is already discarded;
An error condition should be reported as a WARN log message, and otherwise ignored.
It is not an error condition...
- if a second playCommand is received after an animation has become ready but before it started playing. In this case, the new priority etc. overwrites the previous values.