Multi-modal expression markup

The text passed to the Skill say skill (and, more generally, any text that the robot speaks out, including chatbot responses) can contain markup actions. Markup actions are embedded in the sentence, and let you synchronise the speech with other robot capabilities: facial expressions, gestures, LEDs, gaze, pauses…

For instance:

<set expression(happy)> <start motion(wave)> Hello! <wait motion timeout=1> <set expression(neutral)>

makes the robot say “Hello!” with a happy face while waving, then wait until the waving motion is finished (or at most 1 second), and finally return to a neutral expression.

We call such a string a multi-modal expression.

The reference implementation of the markup language is the SocialMinds dialogue_manager. The default values given below are the ones of this implementation.

Syntax

A markup action is enclosed in angle brackets. Its full form is:

<verb name(arguments) timeout=seconds>

The arguments and the timeout are optional. The minimal form is therefore <verb name>.

Everything outside of angle brackets is regular text, spoken by the TTS engine. Consecutive pieces of text are spoken as one utterance; speech is blocking, meaning that the markup actions following a piece of text are only executed once that text has been spoken.

Verbs

The verb must be one of:

Verb	Meaning
`set`	Start and forget the action. Useful when you do not need to know if or when the action completes.
`start`	Start the action, without waiting for it to complete. The action is tracked, and can later be waited for (`wait`) or cancelled (`stop`).
`wait`	Wait for a previously started action to complete (the most recent one with the same name).
`stop`	Stop an ongoing action (the most recent one with the same name).
`do`	Equivalent to `start` immediately followed by `wait`: blocks until the action is completed.

wait and stop only take the action name (and optionally a timeout), not its arguments: <wait motion>, <stop look_at>.

Actions that are started (start, not set) and neither waited for nor stopped explicitly are implicitly waited for at the end of the expression.

Note

Actions implemented as a ROS topic (for instance, expression) cannot be tracked: start and do behave as set for those.

Arguments

Arguments are written between parentheses, like a function call. They can be positional or named (key=value). Positional arguments must come before named ones:

<set expression(happy)>
<set expression(name=happy)>
<do leds(blue, groups=[ear_leds], effect=blink)>

Supported values are:

numbers: 1, -0.5, 1e-3, 0x1F
strings, either quoted ("hello world", 'hello') or as a bare word (happy)
booleans and null: true, false, null
arrays: [ear_leds, back_leds]

Timeout

The optional timeout=<seconds> specifies the maximum time to wait for the action to complete. If omitted, an implementation-defined default is used (10 seconds in dialogue_manager).

<do motion(bow) timeout=3>
<wait motion timeout=1>

In addition, the complete multi-modal expression has a global timeout (60 seconds by default in dialogue_manager), after which all ongoing actions are cancelled.

Built-in actions

Built-in actions resemble regular markup actions, but do not take a verb.

The only built-in action currently is:

<pause(seconds)>: stay silent for the given duration.

For instance:

<start motion(wave)> Hello! <pause(2)> Anything new going on? <wait motion>

makes the robot stay silent for 2 seconds after saying “Hello!”.

Available actions

The available markup actions are defined by the component executing the multi-modal expression (typically, the dialogue manager), and map to standard skills. The default ones are:

Markup action	Arguments	Underlying skill
`expression(name="neutral")`	`name`: the expression to display (eg `happy`, `sad`, `amazed`…)	Skill set_expression
`motion(name)`	`name`: a pre-recorded motion (see Skill list_motions)	Skill replay_motion
`leds(color, groups, effect="solid_color", duration, alpha=1.0, secondary_color, secondary_alpha=1.0, cycle=1.0, partition=1.0)`	see Skill do_led_effect	Skill do_led_effect
`look_at(x=1.0, y, z, frame="base_link", policy="glance")`	target point, in the given `frame`	Skill look_at

Default values are shown in the signatures above. When unsure of the argument order, prefer named arguments: <set look_at(x=1.0, y=0.5, z=1.2)>.

Note

Implementations can disable specific markup actions (for instance, dialogue_manager exposes a disabled_markup_actions parameter; motion is disabled by default). Disabled actions are silently skipped.

Variables

Both the spoken text and the argument values can refer to variables, using the syntax @name|default (or @name.field|default). If the variable is not provided by the executing component, the default value is used:

Hello @user.name|"there"! <set expression(@mood|neutral)>

In the spoken text, the default value must be a quoted string.

Error handling

If a multi-modal expression cannot be parsed (for instance, because of an unterminated <), the whole string is spoken as-is, without executing any action.

Examples

Change expression while speaking:

<set expression(happy)> I'm the famous emotion mirroring robot! My holiday was great <set expression(amazed)>, thank you for asking! <set expression(neutral)>

Look somewhere, then speak once the head movement is completed:

<do look_at(x=1.0, y=-0.5, z=1.0, policy="glance")> Over there is the kitchen.

Blink the LEDs while talking, and stop them afterwards:

<start leds(blue, groups=[ear_leds], effect=blink)> Let me think about it... <stop leds> I've got it!