Version 4 (modified by masc01, 9 years ago) (diff)


State information defined in the SEMAINE system

The SEMAINE system needs to maintain various kinds of state information: the context, such aswhich character is currently active, and whether or not there is a user present; the agent's mental state; the agent's interpretation of the user's behavioural and emotional state; and the state of the dialogue.

In a Message-Oriented Middleware (MOM) it is not straightforward to preserve a centrally held state in such a way that several components can access it, because a MOM does not provide a shared memory. One option for implementing distributed access to state information would have been a specialised information repository component; however, this would have required each component to send and receive a message every time that it wants to access a certain information.

The solution implemented in the SEMAINE API is of a different kind. A specialised class State­Receiver is keeping track of state-related messages; it parses incoming messages and saves the information in a local information repository. That means, every component has local access to its own copy of the latest state information. Since the local copies are updated through the same state-related messages, they are updated in synchrony. Looking up a certain piece of information is thus as easy as accessing a local variable.

An important challenge in this kind of setup is to make the encoding-decoding link between the representation of information in XML-based messages, and a unique “short name” by which components can access the information. For example, the dialogue state information that the agent does not have the turn at the moment is encoded as follows:

<semaine:dialog-state xmlns:semaine=””>
	<semaine:agent believesHasTurn=”false”/>

The component, on the other hand, should be able to access this information independently of the representation format, e.g. through a short name such as agentHasTurn.

If the link between message format and short name were hard-coded in the program code, it would not allow users to flexibly reuse the state information mechanism in novel domains and applications, which would be incompatible with the SEMAINE API's ambition to be a reusable framework. Therefore, we have developed and implemented a mechanism that allows developers to represent this relationship in a configuration file, using namespace-aware XPath expressions. XPath is a formalism for navigating through XML documents to access information. In the above example, the information whether or not the agent belives it has the turn can be accessed by going to the element <dialog-state> in the namespace, then to the child element <agent> in the same namespace, and finally by accessing the value of the attribute believesHasTurn. In XPath, this can be encoded as:


where the namespace prefix semaine is bound to the namespace By associating this XPath expression with a short name userHasTurn, it is possible to provide the relation in a configuration file.

XPath in its general form is a very powerful framework which is intended only for accessing information in existing XML documents. It is not originally intended to be used for the generation of documents. However, by using only the subset of navigating to elements and accessing attribute values or textual content, we can reuse the XPath expressions also for the generation of XML documents and thus for encoding the information.

As a result, we have a fully configurable mechanism for encoding and decoding state information. The current content of the configuration file is at trunk/java/config/stateinfo.config. It can be seen that both the relation between short names and XPath expressions and the namespace prefixes can be given in the configuration file, providing full flexibility for future extensions.

The following tables aim to give a complete account of the meaning of state information currently defined.

These are the “current best guess” information we have on anything we know anything about. Short name is what can be used in StateReceiver.getCurrentBestGuess() to obtain the latest value of that variable. XPath is the XPath expression used for extracting the information from messages. Possible values is a data type and, as applicable, a value range or a list of values – e.g., “float: [0,1[” or “string: a lot, a little, not really” or “boolean”. Definition is the meaning of the information, in human-readable terms.

The following namespace prefixes are used in the XPath expressions.

PrefixNamespace URI

User State

Topic: state.user.behaviour

Short nameXPathPossible valuesDefinition
headGesture/semaine:user-state/bml:bml/bml:head/@typestring: NOD, SHAKE, TILT-LEFT, TILT-RIGHT, APPROACH, RETRACTUser is currently performing the given head gesture (note that TILT-LEFT and TILT-RIGHT is left and right from the user's perspective.
headGestureStarted/semaine:user-state/bml:bml/bml:head/@startfloat: [0, very large number]time when the user started with the gesture, since the system last got ready, in seconds
headGestureStopped/semaine:user-state/bml:bml/bml:head/@endfloat: [0, very large number]time when the user stopped the gesture, since the system last got ready, in seconds
facialExpression/semaine:user-state/bml:bml/bml:face/@shapestring: SMILE, FROWN, RAISED_EYEBROWS (placeholder for a better solution!!)simplified categorisation of facial expressions
facialActionUnits/semaine:user-state/bml:bml/bml:face/@austring: space-separated list of action unit identifiersaction units detected in the face
facialExpressionStarted/semaine:user-state/bml:bml/bml:face[@shape]/@startfloat: [0, very large number]time when the user started with the gesture, since the system last got ready, in seconds
facialExpressionStopped/semaine:user-state/bml:bml/bml:face[@shape]/@endfloat: [0, very large number]time when the user stopped the gesture, since the system last got ready, in seconds
facialActionUnitsStarted/semaine:user-state/bml:bml/bml:face[@au]/@startfloat: [0, very large number]time when the user started with the gesture, since the system last got ready, in seconds
facialActionUnitsStopped/semaine:user-state/bml:bml/bml:face[@au]/@endfloat: [0, very large number]time when the user stopped the gesture, since the system last got ready, in seconds
pitchDirection/semaine:user-state/semaine:pitch/@directionstring: rise, fall, rise-fall, fall-rise, high, mid, lowabstraction of the pitch contour, if the pitch values exceed a certain threshold (tbd)
speaking/semaine:user-state/semaine:speaking/@statusboolean: true, falsewhether the user is currently speaking
vocalization/semaine:user-state/semaine:vocalization/@namestring: (laughter), (sigh)a non-verbal vocalisation produced by the user
facePresent/semaine:user-state/semaine:face-present/@statusboolean: true, falsewhether a face is currently present
interest/semaine:user-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='interest']/@valuefloat: [0, 1]the degree to which the user seems to be bored (<0.5), or interested (>0.5)
valence/semaine:user-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='valence']/@valuefloat: [0, 1]negative (<0.5) or positive (>0.5)
arousal/semaine:user-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='arousal']/@valuefloat: [0, 1]below average arousal (<0.5) or higher than average arousal (>0.5)
potency/semaine:user-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='potency']/@valuefloat: [0, 1]out of control (<0.5) or in control (>0.5)
unpredictability/semaine:user-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='unpredictability']/@valuefloat: [0, 1]predictable situation (<0.5) or unpredictable situation (>0.5)
intensity/semaine:user-state/emotion:emotion/emotion:intensity/@valuefloat: [0, 1]how intense the emotion is, globally
emotion-quadrant/semaine:user-state/emotion:emotion[@category-set='']/emotion:category/@namestring: positiveActive, negativeActive, positivePassive, negativePassive, neutral
userUtterance/semaine:user-state/semaine:userutterance/@utterancestringA list (seperated with spaces) of all the detected words
userUtteranceStartTime/semaine:user-state/semaine:userutterance/@starttimefloat: [0, inf]The starting time of the detected words
userUtteranceFeatures/semaine:user-state/semaine:userutterance/@featuresstringA list (separated with spaces) of all the detected linguistic features that are extracted from the detected words
gender/semaine:user-state/semaine:gender/@namestring: male, female, unknownThe user's gender.

Agent State

Topic: state.agent

Short nameXPathPossible valuesDefinition
needToSpeakSuggestion: /semaine:agent-state/semaine:needtospeak/@valueint: [-100, 100]A value that describes how eager the agent wants to speak. Values below 0 indicate the agent wants to be silent, values above 0 indicate that the agent wants to speak. A higher or a lower value increases the strength of this.
turnTakingIntentionSuggestion: /semaine:agent-state/semaine:turntakingintention/@valueString: startSpeaking, stopSpeakingDescribes the turn taking intention of the agent, if the agent wants to start speaking or if it wants to stop.
agentUtterance/semaine:agent-state/semaine:agentutterance/@utteranceStringThe text currently spoken by the agent, be it as a speaker or as a listener. empty or empty string if agent is currently not speaking.
agentUtteranceStartTime/semaine:agent-state/semaine:agentutterance/@starttimelongthe system time at which the agentUtterance started.
agentHead/semaine:agent-state/semaine:agent-head/@behaviourThe behaviour currently shown by the agent, be it as a speaker or as a listener. empty or empty string if agent is currently not showing any behaviour on this modality.
agentHeadStartTime/semaine:agent-state/semaine:agent-head/@starttimelongthe system time at which the behaviour started.
agentFace/semaine:agent-state/semaine:agent-face/@behaviourThe behaviour currently shown by the agent, be it as a speaker or as a listener. empty or empty string if agent is currently not showing any behaviour on this modality.
agentFaceStartTime/semaine:agent-state/semaine:agent-face/@starttimelongthe system time at which the behaviour started.
agentGaze/semaine:agent-state/semaine:agent-gaze/@behaviourThe behaviour currently shown by the agent, be it as a speaker or as a listener. empty or empty string if agent is currently not showing any behaviour on this modality.
agentGazeStartTime/semaine:agent-state/semaine:agent-gaze/@starttimelongthe system time at which the behaviour started.
agentGesture/semaine:agent-state/semaine:agent-gesture/@behaviourThe behaviour currently shown by the agent, be it as a speaker or as a listener. empty or empty string if agent is currently not showing any behaviour on this modality.
agentGestureStartTime/semaine:agent-state/semaine:agent-gesture/@starttimelongthe system time at which the behaviour started.
agentTorso/semaine:agent-state/semaine:agent-torso/@behaviourThe behaviour currently shown by the agent, be it as a speaker or as a listener. empty or empty string if agent is currently not showing any behaviour on this modality.
agentTorsoStartTime/semaine:agent-state/semaine:agent-torso/@starttimelongthe system time at which the behaviour started.
agreement/semaine:agent-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='agreement']/@valuefloat: [0, 1]bipolar scale (neutral point: 0.5) disagreement-agreement
acceptance/semaine:agent-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='acceptance']/@valuefloat: [0, 1]bipolar scale (neutral point: 0.5) refusal-acceptance
belief/semaine:agent-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='belief']/@valuefloat: [0, 1]bipolar scale (neutral point: 0.5) disbelief-belief
liking/semaine:agent-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='liking']/@valuefloat: [0, 1]bipolar scale (neutral point: 0.5) disliking-liking
understanding/semaine:agent-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='understanding']/@valuefloat: [0, 1]bipolar scale (neutral point: 0.5) no-understanding - understanding
interest/semaine:agent-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='interest']/@valuefloat: [0, 1]bipolar scale (neutral point: 0.5) no-interest - interest
anger/semaine:agent-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='anger']/@valuefloat: [0, 1]unipolar scale (0 = not present, 1 = intensely present)
sadness/semaine:agent-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='sadness']/@valuefloat: [0, 1]unipolar scale (0 = not present, 1 = intensely present)
amusement/semaine:agent-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='amusement']/@valuefloat: [0, 1]unipolar scale (0 = not present, 1 = intensely present)
happiness/semaine:agent-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='happiness']/@valuefloat: [0, 1]unipolar scale (0 = not present, 1 = intensely present)
contempt/semaine:agent-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='contempt']/@valuefloat: [0, 1]unipolar scale (0 = not present, 1 = intensely present)
anticipation/semaine:agent-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='anticipation']/@valuefloat: [0, 1]bipolar scale (neutral point: 0.5) low-to-high anticipation
solidarity/semaine:agent-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='solidarity']/@valuefloat: [0, 1]bipolar scale (neutral point: 0.5) low-to-high solidarity
antagonism/semaine:agent-state/emotion:emotion[@dimension-set='']/emotion:dimension[@name='antagonism']/@valuefloat: [0, 1]bipolar scale (neutral point: 0.5) low-to-high antagonism
emotion-quadrantSuggestion: /semaine:agent-state/emotion:emotion/emotion:category/@namestring: positiveActive, negativeActive, positivePassive, negativePassive, neutralThe emotional quadrant of the agent
stance to user
stance to topic
emotionally-concordant-with-useris this necessary? Can't this be concluded from the user-state:emotion-quadrant and the agent-state:emotion-quadrant?

Dialog State

Topic: state.dialog

Short nameXPathPossible valuesDefinition
userTurnState/semaine:dialog-state/semaine:user/@believesHasTurnboolean: true, false
agentTurnState/semaine:dialog-state/semaine:agent/@believesHasTurnString: speaking, listening, expectingAnswer
convState/semaine:dialog-state/semaine:agent/@convStateString: listening, asking

Context State

Topic: state.context

Short nameXPathPossible valuesDefinition
userPresent/semaine:situational-context/semaine:user/@statusstring: present, absentwhether there is currently a user ready to interact with the system
character/semaine:situational-context/semaine:character/@namestring: Poppy, Prudence, Obadiah, Spike, Moderatorthe name of the character who is currently the active character in the system.
nextCharacter/semaine:situational-context/semaine:character/@nextstring: Poppy, Prudence, Obadiah, Spikeonly used with character='Moderator' -- the name of the next character that the moderator should introduce
dialogContext/semaine:situational-context/semaine:dialog-context/@namestring: AnnounceNextCharacter, Introduction, Questionsonly used with character='Moderator' -- the type of dialog the moderator should perform

3.4 System State

Topic: state.system

Short nameXPathPossible valuesDefinition
cameraPresent/semaine:setup/semaine:camera/@statusstring: present, absent (default: absent)whether a camera is present in the system
cameraXResolution/semaine:setup/semaine:camera/@xresinteger: [1,10000]x resolution of camera images
cameraYResolution/semaine:setup/semaine:camera/@yresinteger: [1,10000]y resolution of camera images
cameraFrameRate/semaine:setup/semaine:camera/@frameratefloat: [10, 1000]number of camera frames per second
cameraNumChannels/semaine:setup/semaine:camera/@numChannelsinteger: [1,3]color: 3 channels, grayscale: 1 channel
microphonePresent/semaine:setup/semaine:microphone/@statusstring: present, absent (default: absent)whether audio input is present in the system
microphoneFrameRate/semaine:setup/semaine:microphone/@frameratefloat: [10, 1000]number of audio frames per second
ecaPresent/semaine:setup/semaine:eca/@statusstring: present, absent (default: absent)whether the Greta ECA player is present

Every component is expected to send their respective information when the system has just become ready.