Multimodal Architecture and Interfaces

1 Conformance Requirements

An implementation is conformant with the MMI Architecture if it consists of one or more software constituents that are conformant with the MMI Life-Cycle Event specification.

A constituent is conformant with the MMI Life-Cycle Event specification if it supports the Life-Cycle Event interface between the Interaction Manager and the Modality Component defined in 6 Interface between the Interaction Manager and the Modality Components . To support the Life-Cycle Event interface, a constituent must be able to handle all Life-Cycle events defined in 6.2 Standard Life Cycle Events either as an Interaction Manager or as a Modality Component or as both.

Transport and format of Life-Cycle Event messages may be implemented in any manner, as long as their contents conform to the standard Life-Cycle Event definitions given in 6.2 Standard Life Cycle Events . Any implementation that uses XML format to represent the life-cycle events must comply with the normative MMI XML schemas contained in C Event Schemas .

The key words MUST , MUST NOT , REQUIRED , SHALL , SHALL NOT , SHOULD , SHOULD NOT , RECOMMENDED , MAY , and OPTIONAL in this specification are to be interpreted as described in [IETF RFC 2119] .

The terms BASE URI and RELATIVE URI are used in this specification as they are defined in [IETF RFC 2396] .

Any section that is not marked as 'informative' is normative.

2 Summary

This section is informative.

This document describes a loosely coupled architecture for multimodal user interfaces, which allows for co-resident and distributed implementations, and focuses on the role of markup and scripting, and the use of well defined interfaces between its constituents.

2 3 Overview

This section is informative.

This document describes the architecture of the Multimodal Interaction (MMI) framework [MMIF] and the interfaces between its constituents. The MMI Working Group is aware that multimodal interfaces are an area of active research and that commercial implementations are only beginning to emerge. Therefore we do not view our goal as standardizing a hypothetical existing common practice, but rather providing a platform to facilitate innovation and technical development. Thus the aim of this design is to provide a general and flexible framework providing interoperability among modality-specific components from different vendors - for example, speech recognition from one vendor and handwriting recognition from another. This framework places very few restrictions on the individual components or on their interactions with each other, but instead focuses on providing a general means for allowing them to communicate with each other, plus basic infrastructure for application control and platform services.

Our framework is motivated by several basic design goals:

Encapsulation. The architecture should make no assumptions about the internal implementation of components, which will be treated as black boxes.
Distribution. The architecture should support both distributed and co-hosted implementations.
Extensibility. The architecture should facilitate the integration of new modality components. For example, given an existing implementation with voice and graphics components, it should be possible to add a new component (for example, a biometric security component) without modifying the existing components.
Recursiveness. The architecture should allow for nesting, so that an instance of the framework consisting of several components can be packaged up to appear as a single component to a higher-level instance of the architecture.
Modularity. The architecture should provide for the separation of data, control, and presentation.

Even though multimodal interfaces are not yet common, the software industry as a whole has considerable experience with architectures that can accomplish these goals. Since the 1980s, for example, distributed message-based systems have been common. They have been used for a wide range of tasks, including in particular high-end telephony systems. In this paradigm, the overall system is divided up into individual components which communicate by sending messages over the network. Since the messages are the only means of communication, the internals of components are hidden and the system may be deployed in a variety of topologies, either distributed or co-located. One specific instance of this type of system is the DARPA Hub Architecture, also known as the Galaxy Communicator Software Infrastructure [Galaxy] . This is a distributed, message-based, hub-and-spoke infrastructure designed for constructing spoken dialogue systems. It was developed in the late 1990's and early 2000's under funding from DARPA. This infrastructure includes a program called the Hub, together with servers which provide functions such as speech recognition, natural language processing, and dialogue management. The servers communicate with the Hub and with each other using key-value structures called frames.

Another recent architecture that is relevant to our concerns is the model-view-controller (MVC) paradigm. This is a well known design pattern for user interfaces in object oriented programming languages, and has been widely used with languages such as Java, Smalltalk, C, and C++. The design pattern proposes three main parts: a Data Model that represents the underlying logical structure of the data and associated integrity constraints, one or more Views which correspond to the objects that the user directly interacts with, and a Controller which sits between the data model and the views. The separation between data and user interface provides considerable flexibility in how the data is presented and how the user interacts with that data. While the MVC paradigm has been traditionally applied to graphical user interfaces, it lends itself to the broader context of multimodal interaction where the user is able to use a combination of visual, aural and tactile modalities.

3 4 Design versus Run-Time considerations

This section is informative.

In discussing the design of MMI systems, it is important to keep in mind the distinction between the design-time view (i.e., the markup) and the run-time view (the software that executes the markup). At the design level, we assume that multimodal applications will take the form of multiple documents from different namespaces. In many cases, the different namespaces and markup languages will correspond to different modalities, but we do not require this. A single language may cover multiple modalities and there may be multiple languages for a single modality.

At runtime, the MMI architecture features loosely coupled software constituents that may be either co-resident on a device or distributed across a network. In keeping with the loosely-coupled nature of the architecture, the constituents do not share context and communicate only by exchanging events. The nature of these constituents and the APIs between them is discussed in more detail in Sections 3-5, below. Though nothing in the MMI architecture requires that there be any particular correspondence between the design-time and run-time views, in many cases there will be a specific software component responsible for each different markup language (namespace).

3.1 4.1 Markup and The Design-Time View

At the markup level, an application consists of multiple documents. A single document may contain markup from different namespaces if the interaction of those namespaces has been ~~defined (e.g., as part of the Compound Document Formats Activity [CDF] .)~~ defined. By the principle of encapsulation, however, the internal structure of documents is invisible at the MMI level, which defines only how the different documents communicate. One document has a special status, namely the Root or Controller Document, which contains markup defining the interaction between the other documents. Such markup is called Interaction Manager markup. The other documents are called Presentation Documents, since they contain markup to interact directly with the user. The Controller Document may consist solely of Interaction Manager markup (for example a state machine defined in CCXML [CCXML] or SCXML [SCXML] ) or it may contain Interaction Manager markup combined with presentation or other markup. As an example of the latter design, consider a multimodal application in which a CCXML document provides call control functionality as well as the flow control for the various Presentation documents. Similarly, an SCXML flow control document could contain embedded presentation markup in addition to its native Interaction Management markup.

These relationships are recursive, so that any Presentation Document may serve as the Controller Document for another set of documents. This nested structure is similar to 'Russian Doll' model of Modality Components, described below in ~~3.2~~ 4.2 Software Constituents and The Run-Time View .

The different documents are loosely coupled and co-exist without interacting directly. Note in particular that there are no shared variables that could be used to pass information between them. Instead, all runtime communication is handled by events, as described below in ~~5.1 Common Event Fields~~ 6 Interface between the Interaction Manager and the Modality Components . Note, however, that this only applies to non-root documents. The IM, which loads the root document, interacts with "other components". I.e., the IM (having the root-document) interacts directly through life-cycle events with Modality Components (having different documents and/or namespaces).

Furthermore, it is important to note that the asynchronicity of the underlying communication mechanism does not impose the requirement that the markup languages present a purely asynchronous programming model to the developer. Given the principle of encapsulation, markup languages are not required to reflect directly the architecture and APIs defined here. As an example, consider an implementation containing a Modality Component providing Text-to-Speech (TTS) functionality. This Component must communicate with the Interaction Manager via asynchronous events (see ~~3.2~~ 4.2 Software Constituents and The Run-Time View ). In a typical implementation, there would likely be events to start a TTS play and to report the end of the play, etc. However, the markup and scripts that were used to author this system might well offer only a synchronous "play TTS" call, it being the job of the underlying implementation to convert that synchronous call into the appropriate sequence of asynchronous events. In fact, there is no requirement that the TTS resource be individually accessible at all. It would be quite possible for the markup to present only a single "play TTS and do speech recognition" call, which the underlying implementation would realize as a series of asynchronous events involving multiple Components.

Existing languages such as ~~XHTML~~ HTML may be used as either the Controller Documents or as Presentation Documents. Further examples of potential markup components are given in ~~4.2.7~~ 5.2.7 Examples

3.2 4.2 Software Constituents and The Run-Time View

At the core of the MMI runtime architecture is the distinction between the Interaction Manager (IM) and the Modality Components, which is similar to the distinction between the Controller Document and the Presentation Documents. The Interaction Manager interprets the Controller Document while the individual Modality Components are responsible for specific tasks, particularly handling input and output in the various modalities, such as speech, pen, video, etc.

The Interaction Manager receives all the events that the various Modality Components generate. Those events may be commands or replies to commands, and it is up to the Interaction Manager to decide what to do with them, i.e., what events to generate in response to them. In general, the MMI architecture follows a 'targetless' event model. That is, the Component that raises an event does not specify its destination. Rather, it passes it up to the Runtime Framework, which will pass it to the Interaction Manager. The IM, in turn, decides whether to forward the event to other Components, or to generate a different event, etc.

Modality Components are black boxes, required only to implement the Modality Component Interface API which is described below. This API allows the Modality Components to communicate with the IM and hence with each other, since the IM is responsible for delivering events/messages among the Components. Since the internals of a Component are hidden, it is possible for an Interaction Manager and a set of Components to present themselves as a Component to a higher-level Interaction Manager. All that is required is that the IM implement the Component API. The result is a "Russian Doll" model in which Components may be nested inside other Components to an arbitrary depth. Nesting components in this manner is one way to produce a 'complex' Modality Component, namely one that handles multiple modalities simultaneously. However, it is also possible to produce complex Modality Components without nesting, as discussed in ~~4.2.3~~ 5.2.3 The Modality Components .

In addition to the Interaction Manager and the modality components, there is a Runtime Framework that provides infrastructure support, in particular a transport layer which delivers events among the components.

Because we are using the term 'Component' to refer to a specific set of entities in our architecture, we will use the term 'Constituent' as a cover term for all the elements in our architecture which might normally be called 'software components'.

3.3 Differences from Compound Document Formats The W3C Compound Document Formats Activity [CDF] is also concerned with the execution of user interfaces written in multiple languages. However, the CDF group focuses on defining the interactions of specific sets of languages within a single document, which may be defined by inclusion or by reference. The MMI architecture, on the other hand, defines the interaction of arbitrary sets of languages in multiple documents. From the MMI point of view, mixed markup documents defined by CDF specifications are treated like any other documents, and may be either Controller or Presentation Documents. Finally, note that the tightly coupled languages handled by CDF will usually share data and scripting contexts, while the MMI architecture focuses on a looser coupling, without shared context. The lack of shared context makes it easier to distribute applications across a network and also places minimal constraints on the languages in the various documents. As a result, authors will have the option of building multimodal applications in a wide variety of languages for a wide variety of deployment scenarios. We believe that this flexibility is important for the further development of the industry. 3.4 4.3 Relationship to EMMA

The Extended Multimodal Annotation Language [EMMA] , is a set of specifications for multimodal systems, and provides details of an XML markup language for containing and annotating the interpretation of user input. For example, a user of a multimodal application might use both speech to express a command, and keystroke gesture to select or draw command parameters. The Speech Recognition Modality would express the user command using EMMA to indicate the input source (speech). The Pen Gesture Modality would express the command parameters using EMMA to indicate the input source (pen gestures). Both modalities may include timing information in the EMMA notation. Using the timing information, a fusion module combines the speech and pen gesture information into a single EMMA notation representing both the command and its parameters. The use of EMMA enables the separation of recognition process from the information fusion process, and thus enables reusable recognition modalities and general purpose information fusion algorithms.

4 5 Overview of Architecture

Here is a list of the Constituents of the MMI architecture. They are discussed in more detail below.

the Interaction Manager, which coordinates the different modalities. It is the Controller in the MVC paradigm.
the Data Component, which ~~vides~~ provides the common data model and represents the Model in the MVC paradigm.
the Modality Components, which provide modality-specific interaction capabilities. They are the Views in the MVC paradigm.
the Runtime Framework, which provides the basic infrastructure and enables communication among the other Constituents.

4.1 5.1 Run-Time Architecture Diagram

4.2 5.2 The Constituents

This section presents the responsibilities of the various constituents of the MMI architecture.

4.2.1 5.2.1 The Interaction Manager

~~The~~ All life-cycle events that the Modality Components generate MUST be delivered to the Interaction Manager. All life-cycle events that are delivered to Modality Components MUST be sent by the Interaction Manager.

Due to the Russian Doll model, Modality Components MAY contain their own Interaction Managers to handle their internal events. However these Interaction Managers are not visible to the top level Runtime Framework or Interaction Manager.

If the Interaction Manager ~~(IM)~~ does not contain an explicit handler for an event, it MUST respect any default behavior that has been established for the event. If there is ~~responsible~~ no default behavior, the Interaction Manager MUST ignore the event. (In effect, the Interaction Manager's default handler for ~~handling~~ all events ~~that the other Components generate.~~ is to ignore them.)

The following paragraph is informative.

Normally there will be specific markup associated with the IM instructing it how to respond to events. This markup will thus contain a lot of the most basic interaction logic of an application. Existing languages such as SMIL, CCXML, SCXML, or ECMAScript can be used for IM markup as an alternative to defining special-purpose languages aimed specifically at multimodal applications. The IM fulfills multiple functions. For example, it is responsible for synchronization of data and focus, etc., across different Modality Components as well as the higher-level application flow that is independent of Modality Components. It also maintains the high-level application data model and may handle communication with external entities and back-end systems. ~~In the future we may split~~ Logically these functions ~~apart~~ could be separated into separate constituents and ~~define different components for each of them.~~ implementations may want to introduce internal structure to the IM. However, for the ~~moment,~~ purposes of this standard, we leave ~~them~~ the various functions rolled up in a single monolithic Interaction Manager component. We note that state machine languages such as SCXML are a good choice for authoring such a multi-function component, since state machines can be composed. Thus it is possible to define a high-level state machine representing the overall application flow, with lower-level state machines nested inside it handling the the cross-modality synchronization at each phase of the higher-level flow.

Due to the Russian Doll model, Components may contain their own Interaction Managers to handle their internal events. However these Interaction Managers are not visible to the top level Runtime Framework or Interaction Manager. If the Interaction Manager does not contain an explicit handler for an event, any default behavior that has been established for the event will be respected. If there is no default behavior, the event will be ignored. (In effect, the Interaction Manager's default handler for all events is to ignore them.)

4.2.2 5.2.2 The Data Component

This section is informative.

The Data Component is responsible for storing application-level data. The Interaction Manager is a client of the Data Component and ~~must be~~ is able to access and update ~~the~~ it as part of its control flow logic, but Modality Components do not have direct access to it. Since Modality Components are black boxes, they may have their own internal Data Components and may interact directly with backend servers. However, the only way that Modality Components can share data among themselves and maintain consistency is is via the Interaction Manager. It is therefore a good application design practice to divide data into two logical classes: private data, which is of interest only to a given modality component, and public data, which is of interest to the Interaction Manager or to more than one Modality Component. Private data may be managed as the Modality Component sees fit, but all modification of public data, including submission to back end servers, should be entrusted to the Interaction Manager.

~~For the initial version of this specification, we do~~ This specification does not define an interface between the Data Component and the Interaction Manager. This amounts to treating the Data Component as part of the Interaction Manager. (Note that this means that the data access language will be whatever one the IM provides.) The Data Component is shown with a dotted outline in the diagram above, however, because it is ~~only~~ logically ~~distinct. However, at some point in the future, we may define the interface between the Data Component and the Interaction Manager~~ distinct and ~~require support for~~ could be placed in a ~~specific data access language, independent of the Interaction Manager.~~ separate component.

4.2.3 5.2.3 The Modality Components

This section is informative.

Modality Components, as their name would indicate, are responsible for controlling the various input and output modalities on the device. They are therefore responsible for handling all interaction with the user(s). Their only responsibility is to implement the interface defined in 5 6 Interface between the Interaction Manager and the Modality Components . Any further definition of their responsibilities ~~must~~ will be highly domain- and application-specific. In particular we do not define a set of standard modalities or the events that they should generate or handle. Platform providers are allowed to define new Modality Components and are allowed to place into a single Component functionality that might logically seem to belong to two or more different modalities. Thus a platform could provide a handwriting-and-speech Modality Component that would accept simultaneous voice and pen input. Such combined Components permit a much tighter coupling between the two modalities than the loose interface defined here. Furthermore, modality components may be used to perform general processing functions not directly associated with any specific interface modality, for example, dialog flow control or natural language processing.

In most cases, there will be specific markup in the application corresponding to a given modality, specifying how the interaction with the user should be carried out. However, we do not require this and specifically allow for a markup-free modality component whose behavior is hard-coded into its software.

4.2.4 5.2.4 The Runtime Framework

The Runtime Framework is a cover term for all the infrastructure services that are necessary for successful execution of a multimodal application. This includes starting the components, handling communication, and logging, etc. For the most part, this version of the specification leaves these functions to be defined in a platform-specific way, but we do specifically define a Transport Layer which handles communications between the components.

4.2.4.1 5.2.4.1 The Event Transport Layer

The Event Transport Layer is responsible for delivering events among the IM and the Modality Components. Clearly, there are multiple transport mechanisms (protocols) that can be used to implement a Transport Layer and different mechanisms may be used to communicate with different modality components. Thus the Event Transport Layer consists of one or more transport mechanisms linking the IM to the various Modality Components.

We place the following requirements on all transport mechanisms:

Events ~~must~~ MUST be delivered reliably. In particular, the event delivery mechanism ~~must~~ MUST report an error if an event can not be delivered, for example if the destination endpoint is unavailable.
Events ~~must~~ MUST be delivered to the destination in the order in which the source generated them. There is no guarantee on the delivery order of events generated by different sources. For example, if Modality Component M1 generates events E1 and E2 in that order, while Modality Component M2 generates E3 and then E4, we require that E1 be delivered to the Runtime Framework before E2 and that E3 be delivered before E4, but there is no guarantee on the ordering of E1 or E2 versus E3 or E4.

For a sample definition of a Transport Layer relying on HTTP, see E F HTTP transport of MMI lifecycle events . ~~In the current draft, this~~ This definition is provided as an example ~~only, but in future drafts we may require support for this and possibly other Transport Layer definitions.~~ only.

4.2.4.1.1 5.2.4.1.1 Event and Information Security

This section is informative.

Events will often carry sensitive information, such as bank account numbers or health care information. In addition events must also be reliable to both sides of transaction: for example, if an event carries an assent to a financial transaction, both sides of the transaction must be able to rely on that assent.

We do not currently specify delivery mechanisms or internal security safeguards to be used by the Modality Components and the Interaction Manager. However, we believe that any secure system will have to meet the following requirements at a minimum:

The following two optional requirements can be met by using the W3's XML-Signature Syntax and Processing specification [XMLSig] .

Authentication. The event delivery mechanism should be able to ensure that the identity of components in an interaction are known.
Integrity. The event delivery mechanism should be able to ensure that the contents of events have not been altered in transit.
The remaining optional requirements for event delivery and information security can be met by following other industry-standard procedures.
Authorization. A component should provide a method to ensure only authorized components can connect to it.
Privacy. The event delivery mechanism should provide a method to keep the message contents secure from any unauthorized access while in transit.
Non-repudiation. The event delivery mechanism, in conjunction with the components, may provide a method to ensure that if a message is sent from one constituent to another, the originating constituent cannot repudiate the message that it sent and that the receiving constituent cannot repudiate that the message was received.

Multiple protocols may be necessary to implement these requirements. For example, TCP/IP and HTTP provide reliable event delivery, but additional protocols such as TLS or HTTPS could be required to meet security requirements.

4.2.5 5.2.5 System and OS Security

This section is informative.

This architecture does not and will not specify the internal security requirements of a Modality Component or Runtime Framework.

4.2.6 5.2.6 Media stream handling

Media streams ~~are typically~~ do not typically flow through the Interaction Manager. This specification does not specify how media connections are established, as the main focus of this specification is the flow of control data. However, all control data logically sent between modality components MUST flow through the Interaction Manager.

4.2.7 5.2.7 Examples

This section is informative.

For the sake of concreteness, here are some examples of components that could be implemented using existing languages. Note that we are mixing the design-time and run-time views here, since it is the implementation of the language (the browser) that serves as the run-time component.

CCXML [CCXML] could be used as both the Controller Document and the Interaction Manager language, with the CCXML interpreter serving as the Runtime Framework and Interaction Manager.
SCXML [SCXML] could be used as the Controller Document and Interaction Manager language
In an integrated multimodal browser, the markup language that provided the document root tag would define the Controller Document while the associated scripting language could serve as the Interaction Manager.
~~XHTML [XHTML]~~ HTML [HTML] could be used as the markup for a Modality Component.
VoiceXML [VoiceXML] could be used as the markup for a Modality Component.
SVG [SVG] could be used as the markup for a Modality Component.
SMIL [SMIL] could be used as the markup for a Modality Component.

5 6 Interface between the Interaction Manager and the Modality Components

The most important interface in this architecture is the one between the Modality Components and the Interaction Manager. Modality Components communicate with the IM via asynchronous events. ~~Components must~~ Constituents MUST be able to ~~raise~~ send events and to handle events that are delivered to them asynchronously. It is not required that ~~components~~ Constituents use these events internally since the implementation of a given ~~Component~~ Constituent is black box to the rest of the system. In general, it is expected that ~~Components~~ Constituents will ~~raise~~ send events both automatically (i.e., as part of their implementation) and under mark-up control.

The majority of the events defined here come in request/response pairs. That is, one party (either the IM or an MC) sends a request and the other returns a response. (The ~~exception is~~ exceptions are the ~~ExtensionNotification event,~~ ExtensionNotification, StatusRequest and StatusResponse events, which can be sent by either party.) In each case it is specified which party sends the request and which party returns the response. If the wrong party sends a request or response, or if the request or response is sent under the wrong conditions (e.g. response without a previous request) the behavior of the receiving party ~~MUST ignore it.~~ is undefined. In the descriptions below, we say that the originating party "MAY" send the request, because it is up to the internal logic of the originating party to decide if it wants to invoke the behavior that the request would trigger. On the other hand, we say that the receiving party "MUST" send the response, because it is mandatory to send the response if and when the request is received.

5.1 6.1 Common Event Fields

The concept of 'context' is basic to these events described below. A context represents a single extended interaction with zero or more users across one ~~(or possibly more) users.~~ or more modality components. In a simple unimodal case, a context can be as simple as a phone call or SSL session. Multimodal cases are more complex, however, since the various modalities may not be all used at the same time. For example, in a voice-plus-web interaction, e.g., web sharing with an associated VoIP call, it would be possible to terminate the web sharing and continue the voice call, or to drop the voice call and continue via web chat. In these cases, a single context persists across various modality configurations. In general, ~~we intend for~~ the 'context' to SHOULD cover the longest period of interaction over which it would make sense for components to store ~~state or~~ information.

For examples of the concrete XML syntax for all these events, see A B Examples of Life-Cycle Events

The following common fields are shared by multiple life-cycle events:

5.1.1 6.1.1 Context

A URI that is MUST be unique ~~across~~ for the ~~system and~~ lifetime of the system. It is used to identify this interaction. All events relating to a given interaction ~~will~~ MUST use the same context URI. Events containing a different context URI ~~will~~ MUST be interpreted as part of other, unrelated, interactions.

5.1.2 6.1.2 Source

A URI representing the address of the sender of the event. The recipient of the event MUST be able to send an event back to the sender by using this value as the 'target' of a message.

5.1.3 6.1.3 Target

A URI ~~representing~~ that MUST represent the address ~~of the destination of~~ to which the ~~event.~~ event will be delivered.

5.1.4 6.1.4 RequestID

A unique identifier for a Request/Response pair. Most life-cycle events come in Request/Response pairs that share a common RequestID. For ~~each~~ any such pair, ~~this id must~~ the RequestID in the Response event MUST match the RequestID in the request event. The RequestID for such a pair MUST be unique within the given context.

5.1.5 6.1.5 Status

An enumeration of ~~'success'~~ 'Success' and ~~'failure'.~~ 'Failure'. The Response event of a Request/Response pair ~~will~~ MUST use this field to report whether it succeeded in carrying out the request.

5.1.6 6.1.6 StatusInfo

~~An arbitrary value providing further error information in cases where the~~ The Response event of a Request/Response pair MAY use this field to provide additional status ~~is 'failure'.~~ information.

5.1.7 6.1.7 Data

~~An optional~~ Any event MAY use this field ~~containing~~ to contain arbitrary data. The format and meaning of this data is application-specific.

5.1.8 6.1.8 Confidential

~~An optional~~ Any event MAY use this field ~~indicating~~ to indicate whether the contents of this event ~~should be treated as~~ are confidential. The default value is 'false'. If the value is 'true', the Interaction Manager and Modality Component implementations MUST not log the information or make it available in any way to third parties unless explicitly instructed to do so by the author of the application.

5.2 6.2 Standard Life Cycle Events

The Multimodal Architecture defines the following basic life-cycle events which ~~must be supported by all modality components.~~ the Interaction Manager and Modality Components MUST support. These events allow the Interaction Manager to invoke modality components and receive results from them. They thus form the basic interface between the IM and the Modality components. Note that the ~~'Extension'~~ ExtensionNotification event offers extensibility since it contains arbitrary ~~XML~~ content and can be raised by either the IM or the Modality Components at any time once the context has been established. For example, an application relying on speech recognition could use the 'Extension' event to communicate recognition results or the fact that speech had started, etc.

5.2.1 6.2.1 NewContextRequest/NewContextResponse

A Modality Component MAY send a NewContextRequest to the IM to request that a new context be created. If this event is sent, the IM MUST respond with the NewContextResponse event. The NewContextResponse event MUST ONLY be sent in response to the NewContextRequest event. Note that the IM MAY create a new ~~context/interaction~~ context without a previous ~~NewContextRequest. In such a case, the IM will send~~ NewContextRequest by sending a PrepareRequest or StartRequest ~~to the modality components~~ containing a new context ~~ID.~~ ID to the Modality Components. Furthermore the IM may respond with the same context in response to newContextRequests from different (multiple) Modality Components, since the interaction can be started by different Modality Components independently.

5.2.1.1 6.2.1.1 NewContextRequest Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . A newly generated identifier used to identify this request.
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

5.2.1.2 6.2.1.2 NewContextResponse Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . This MUST match the RequestID in the NewContextRequest event.
Status See ~~5.1.5~~ 6.1.5 Status . If ~~the value is Success, the NewContextRequest~~ IM has ~~been~~ accepted the NewContextRequest, it MUST set this field to Success and MUST include a new context ~~identifier will be included. (See below). If the value is Failure, no context identifier will be included and further information will be included in the StatusInfo field.~~ identifier.
Context See ~~5.1.1~~ 6.1.1 Context . A newly created context identifier. This field MUST be empty if status is Failure.
StatusInfo See ~~5.1.6~~ 6.1.6 StatusInfo .
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

5.2.2 6.2.2 PrepareRequest/PrepareResponse

The IM MAY send a PrepareRequest to allow the Modality Components to pre-load markup and prepare to run. Modality Components are not required to take any particular action in response to this event, but they MUST return a PrepareResponse event. Modality Components that return a PrepareResponse event with Status of 'Success' SHOULD be ready to run with close to 0 delay upon receipt of the StartRequest.

~~A given component can only execute a single StartRequest at one time (see 5.2.3 StartRequest/StartResponse ). However, the~~ The Interaction Manager MAY send multiple PrepareRequest events to a Modality Component for the same ~~Context, each referencing~~ Context before sending a StartRequest. Each request MAY reference a different ContentURL or ~~containing~~ contain different in-line ~~Content, before sending a StartRequest. In this case,~~ Content. When it receives multiple PrepareRequests, the Modality Component SHOULD prepare to run any of the specified content. ~~The subsequent StartRequest event will determine which specific content the Modality Component should execute.~~

5.2.2.1 6.2.2.1 PrepareRequest Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . A newly generated identifier used to identify this request.
Context See ~~5.1.1~~ 6.1.1 Context . Note that the IM ~~may re-use~~ MAY use the same context value in ~~successive calls~~ multiple PrepareRequest events when it wishes to ~~Prepare if they are all within~~ execute multiple instances of markup in the same ~~session/call.~~ context.
ContentURL Optional URL of the content that the Modality Component ~~should~~ SHOULD prepare to execute. ~~Includes standard HTTP fetch parameters such as max-age, max-stale, fetchtimeout, etc. Incompatible with content .~~
Content Optional Inline markup ~~for the Modality Component to execute. Incompatible with contentURL . Note~~ that ~~it is legal for both contentURL and content to be empty. In such a case,~~ the Modality Component ~~will revert~~ SHOULD prepare to ~~its default hard-coded behavior, which could consist of returning an error event or of running a preconfigured or hard-coded script.~~ execute.
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

The IM MUST NOT specify both the ContentURL and Content in a single PrepareRequest. The IM MAY leave both contentURL and content empty. In such a case, the Modality Component MUST revert to its default behavior. For example, this behavior could consist of returning an error event or of running a preconfigured or hard-coded script.

5.2.2.2 6.2.2.2 PrepareResponse Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . This MUST match the RequestID in the PrepareRequest event.
Context See ~~5.1.1~~ 6.1.1 Context . This MUST match the value in the ~~Prepare~~ PrepareRequest event.
Status See ~~5.1.5~~ 6.1.5 Status .
StatusInfo See ~~5.1.6~~ 6.1.6 StatusInfo .
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

5.2.3 6.2.3 StartRequest/StartResponse

~~The IM sends a StartRequest to~~ To invoke a ~~Modality Component.~~ modality component, the IM MUST send a StartRequest. The Modality Component MUST return a StartResponse event in response. ~~If the Runtime Framework has sent a previous Prepare event, it~~ The IM MAY ~~leave~~ include a value in the ~~contentURL and content fields empty, and~~ ContentURL or Content field of this event. In this case, the Modality Component MUST use ~~the values from the Prepare event. If the IM includes new values for these fields, the values in the Start event override those in the Prepare event.~~ this value.

If ~~the Interaction Manager sends multiple StartRequests to a given Modality Component before it receives a DoneNotification, each such request overrides the earlier ones. Thus if~~ a Modality Component receives a new StartRequest while it is executing a previous one, it MUST either cease execution of the previous StartRequest and begin executing the content specified in the most recent StartRequest, or reject the new StartRequest, returning a StartResponse with status equal to ~~'failure'.~~ 'Failure'.

5.2.3.1 6.2.3.1 StartRequest Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . A newly generated identifier used to identify this request.
Context See ~~5.1.1~~ 6.1.1 Context . Note that the IM ~~may re-use~~ MAY use the same context value in ~~successive calls~~ multiple StartRequest events when it wishes to ~~Start if they are all within~~ execute multiple instances of markup in the same ~~session/call.~~ context.
ContentURL Optional URL of the content that the Modality Component ~~should~~ MUST attempt to execute. ~~Includes standard HTTP fetch parameters such as max-age, max-stale, fetchtimeout, etc. Incompatible with content .~~
Content Optional Inline markup ~~for~~ that the Modality Component MUST attempt to execute. ~~Incompatible with~~
contentURL . Note that it is legal for Source See 6.1.2 Source .
Target See 6.1.3 Target .
Data See 6.1.7 Data .
Confidential See 6.1.8 Confidential .

The IM MUST NOT specify both the ContentURL and Content in a single StartRequest. The IM MAY leave both contentURL and content ~~to be~~ empty. In such a case, the Modality Component ~~will either use~~ MUST run the ~~values provided~~ content specified in the most recent ~~PrepareRequest,~~ PrepareRequest in this context, if ~~one was sent, or~~ there is one. Otherwise it MUST revert to its default ~~hard-coded behavior, which~~ behavior. For example, this behavior could consist of returning an error event or of running a preconfigured or hard-coded script.

6.2.3.2 StartResponse Properties

RequestID. See 6.1.4 RequestID . This MUST match the RequestID in the StartRequest event.
Context See 6.1.1 Context . This MUST match the value in the Start event.
Status See 6.1.5 Status .
StatusInfo See 6.1.6 StatusInfo .
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

6.2.4 DoneNotification

If the Modality Component reaches the end of its processing, it MUST return a DoneNotification to the IM.

5.2.3.2 StartResponse 6.2.4.1 DoneNotification Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . This MUST match the RequestID in of the StartRequest event.
Context See ~~5.1.1~~ 6.1.1 Context . This MUST match the value in the Start event.
Status See ~~5.1.5~~ 6.1.5 Status .
StatusInfo See ~~5.1.6~~ 6.1.6 StatusInfo .
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

~~5.2.4 DoneNotification~~

The ~~Modality Component MAY return a DoneNotification to the IM to indicate that it has reached the end of its processing. The~~ DoneNotification event is intended to indicate the completion of the processing that has been initiated by the Interaction Manager with a StartRequest. As an example a voice modality component might use the DoneNotification event to indicate the completion of a recognition task. In this case the DoneNotification event might carry the recognition result expressed using EMMA. However, there may be tasks which do not have a specific end. For example the Interaction Manager might send a StartRequest to a graphical modality component requesting it to display certain information. Such a task does not necessarily have a specific end and thus the graphical modality component might never send a DoneNotification event to the Interaction Manager. Thus the graphical modality component would display the screen until it received another StartRequest (or some other lifecycle event) from the Interaction Manager.

5.2.4.1 DoneNotification Properties RequestID . See 5.1.4 RequestID . MUST match the RequestID of the StartRequest event. Context See 5.1.1 Context . MUST match the value in the Start event. Status See 5.1.5 Status . StatusInfo See 5.1.6 StatusInfo . Source See 5.1.2 Source . Target See 5.1.3 Target . Data See 5.1.7 Data . Confidential See 5.1.8 Confidential .

5.2.5 6.2.5 CancelRequest/CancelResponse

The IM MAY send a CancelRequest to stop processing in the Modality Component. In this case, the Modality Component MUST stop processing and then MUST return a CancelResponse.

5.2.5.1 6.2.5.1 CancelRequest Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . A newly generated identifier used to identify this request.
Context See ~~5.1.1~~ 6.1.1 Context . This MUST match the value in the ~~Start~~ StartRequest event.
Immediate Boolean value indicating whether a hard stop is requested. If this value is 'true' the Modality Component SHOULD stop processing for the specified context immediately. If this value is 'false' the Modality Component SHOULD stop processing for the specified context gracefully.
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

5.2.5.2 6.2.5.2 CancelResponse Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . This MUST match the RequestID in the CancelRequest event.
Context See ~~5.1.1~~ 6.1.1 Context . This MUST match the value in the ~~Start~~ StartRequest event.
Status See ~~5.1.5~~ 6.1.5 Status .
StatusInfo See ~~5.1.6~~ 6.1.6 StatusInfo .
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

5.2.6 6.2.6 PauseRequest/PauseResponse

The IM MAY send a PauseRequest to suspend processing by the Modality Component. Modality Components ~~may ignore this command if they are unable to pause, but they~~ MUST return a ~~PauseResponse.~~ PauseResponse once they have paused, or once they determine that they will be unable to pause.

5.2.6.1 6.2.6.1 PauseRequest Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . A newly generated identifier used to identify this request.
Context See ~~5.1.1~~ 6.1.1 Context . This MUST match the value in the Start event.
Immediate Boolean value indicating whether a hard pause is requested. If this value is 'true' the Modality Component SHOULD pause processing for the specified context immediately. If this value is 'false' the Modality Component SHOULD pause processing for the specified context gracefully.
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

5.2.6.2 6.2.6.2 PauseResponse Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . This MUST match the RequestID in the PauseRequest event.
Context See ~~5.1.1~~ 6.1.1 Context . This MUST match the value in the Start event.
Status See ~~5.1.5~~ 6.1.5 Status .
StatusInfo ~~ISee~~ See ~~5.1.6~~ 6.1.6 StatusInfo .
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

5.2.7 6.2.7 ResumeRequest/ResumeResponse

The IM MAY send the ResumeRequest to resume processing that was paused by a previous PauseRequest. The IM MUST NOT send the ResumeRequest to a context that is not paused due to a previous PauseRequest. Implementations ~~may ignore this command if they are unable~~ that have paused MUST attempt to ~~pause, but they~~ resume processing upon receipt of this event and MUST return a ~~ResumeResponse.~~ ResumeResponse afterwards. The 'Status' MUST be 'Success' if the implementation has succeeded in resuming processing and MUST be 'Failure' otherwise.

5.2.7.1 6.2.7.1 ResumeRequest Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . A newly generated identifier used to identify this request.
Context See ~~5.1.1~~ 6.1.1 Context . This MUST match the value in the Start event.
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

5.2.7.2 6.2.7.2 ResumeResponse Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . This MUST match the RequestID in the ResumeRequest event.
Context See ~~5.1.1~~ 6.1.1 Context . This MUST match the value in the Start event.
Status See ~~5.1.5~~ 6.1.5 Status .
StatusInfo See ~~5.1.6~~ 6.1.6 StatusInfo .
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

5.2.8 6.2.8 ExtensionNotification

This event MAY be generated by ~~either~~ the IM or and MAY be generated by the Modality Component. It is used to encapsulate application-specific events that are extensions to the framework defined here. For example, if an application containing a voice modality wanted that modality component to notify the Interaction Manager when speech was detected, it would cause the voice modality to generate an ~~Extension~~ ExtensionNotification event ( with a 'name' of something like 'speechDetected') at the appropriate time.

5.2.8.1 6.2.8.1 ExtensionNotification Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . A newly generated identifier used to identify this request.
Name The name of ~~this event. This is an~~ the application-specific ~~value.~~ event.
Context See ~~5.1.1~~ 6.1.1 Context . MUST match the value in the Start event.
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

5.2.9 6.2.9 ClearContextRequest/ClearContextResponse

The IM MAY send a ClearContextRequest to indicate that ~~t he~~ the specified context is no longer active and that any resources associated with it may be freed. Modality Components are not required to take any particular action in response to this command, but MUST return a ClearContextResponse. Once the IM has sent a ClearContextRequest to a Modality Component, it MUST NOT send the Modality Component any more events for that context.

5.2.9.1 6.2.9.1 ClearContextRequest Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . A newly generated identifier used to identify this request.
Context See ~~5.1.1~~ 6.1.1 Context . This MUST match the value in the Start event.
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

5.2.9.2 6.2.9.2 ClearContextResponse Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . This MUST match the RequestID in the ClearContextRequest event.
Context See ~~5.1.1~~ 6.1.1 Context . This MUST match the value in the Start event.
Status See ~~5.1.5~~ 6.1.5 Status .
StatusInfo See ~~5.1.6~~ 6.1.6 StatusInfo .
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

5.2.10 6.2.10 StatusRequest/StatusResponse

The StatusRequest message and the corresponding StatusResponse are intended to provide keep-alive ~~functionality, informing~~ functionality. Either the IM ~~about the presence of the various modality components. Note that both these messages may be either linked to a specific context~~ or sent to the underlying server independent of any user interaction. In the former case, the IM is inquiring about the status of the specific interaction (i.e. context). In the latter case, it is in effect asking the underlying server whether it could start a new Context if requested to do so. The StatusRequest message is sent from the IM to a Modality Component. By waiting for an implementation dependent period of time for a StatusResponse message, the IM may determine if the Modality Component is active. If automatic updates are enabled, the Modality Component ~~SHOULD~~ MAY send ~~multiple StatusResponse messages in response to a single~~ the StatusRequest message. The recipient MUST respond with the StatusResponse message.

5.2.10.1 6.2.10.1 Status Request Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . A newly generated identifier used to identify this request.
Context See ~~5.1.1~~ 6.1.1 Context . Optional specification of the context for which the status is requested. If it is ~~not~~ present, the ~~request~~ recipient MUST respond with a StatusResponse message indicating the status of the specified context. If it is ~~directed to~~ not present, the recipient MUST send a StatusResponse message indicating the status of the underlying server, namely the software that would host a new context if one were created.
RequestAutomaticUpdate. A boolean ~~value indicating whether~~ value. It it is 'true' the ~~Modality Component should~~ recipient SHOULD send ~~ongoing~~ periodic StatusResponse messages without waiting for an additional StatusRequest ~~messages from~~ message. If it is 'false', the ~~Runtime Framework.~~ recipient SHOULD send one and only one StatusResponse message in response to this request.
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

5.2.10.2 6.2.10.2 StatusResponse Properties

RequestID. See ~~5.1.4~~ 6.1.4 RequestID . This MUST match the RequestID in the StatusRequest event.
AutomaticUpdate. A boolean ~~indicating whether~~ value. If it is 'true' the ~~Modality Component will~~ sender MUST keep sending StatusResponse messages in the future without waiting for another StatusRequest message. If it is 'false', the sender MUST wait for a subsequent StatusRequest message before sending another StatusResponse message.
Context See ~~5.1.1~~ 6.1.1 Context . An optional specification of the context for which the status is being returned. If it is present, the response MUST represent the status of the specified context. If it is not present, the response ~~represents~~ MUST represent the status of the underlying server.
Status An enumeration of 'Alive' or 'Dead'. The meaning of these values depends on whether the 'context' parameter is present. If it is, ~~the status is 'Alive' means that~~ and the specified ~~session~~ context is still active and capable of handling new life cycle ~~events. The status 'Dead' means that~~ events, the sender MUST set this field to 'Alive'. If the 'context' parameter is present and the context has terminated ~~and no further interaction with the user~~ or is ~~available using it.~~ otherwise unable to process new life cycle events, the sender MUST set the status to 'Dead'. If the 'context' parameter is not provided, the status refers to the underlying server. ~~A value of 'Alive' indicates that~~ If the ~~Modality Component~~ sender is able to ~~handle subsequent Prepare and Start messages. If status is 'Dead',~~ create new contexts, it ~~is not able to handle such requests. Thus the status of 'Dead' indicates that the modality component is going off-line. If~~ MUST set the ~~IM receives a StatusResponse message with~~ status ~~of 'Dead', it may continue~~ to ~~send StatusRequest messages, but~~ 'Alive', otherwise, it MUST set it ~~may not receive a response~~ to ~~them until the Modality Component comes back online.~~ 'Dead'.
Source See ~~5.1.2~~ 6.1.2 Source .
Target See ~~5.1.3~~ 6.1.3 Target .
Data See ~~5.1.7~~ 6.1.7 Data .
Confidential See ~~5.1.8~~ 6.1.8 Confidential .

event / state	Idle	Running	Paused
PrepareRequest	preload or update content	preload or update content	preload or update content
StartRequest	Transition: Running use new content if provided, otherwise use last available content	stop processing current content, restart as in Idle	Transition: Running stop processing current content, restart as in Idle
StartRequest	Failure: NoContent if MC requires content to run and none has been provided	stop processing current content, restart as in Idle
CancelRequest	~~Failure: NotRunning~~ Ignore	Transition: Idle	Transition: Idle
PauseRequest	Failure: ~~NotRunning~~ ignore	Transition: Paused	~~Failure: AlreadyPaused~~ Ignore
PauseRequest	Failure: ~~NotRunning~~ ignore	Failure: CantPause if MC is unable to pause	~~Failure: AlreadyPaused~~ Ignore
ResumeRequest	~~Failure: NotRunning~~ Ignore	Failure: AlreadyRunning	Transition: Running
StatusRequest	send status	send status	send status
ClearContextRequest	close session	close session	close session

4.2.4 5.2.4 The Runtime Framework

4.2.5 5.2.5 System and OS Security

5.2.1.2 6.2.1.2 NewContextResponse Properties

5.2.2 6.2.2 PrepareRequest/PrepareResponse

5.2.2.2 6.2.2.2 PrepareResponse Properties

5.2.5 6.2.5 CancelRequest/CancelResponse

5.2.5.2 6.2.5.2 CancelResponse Properties

5.2.6 6.2.6 PauseRequest/PauseResponse

5.2.6.2 6.2.6.2 PauseResponse Properties

5.2.7 6.2.7 ResumeRequest/ResumeResponse

5.2.7.2 6.2.7.2 ResumeResponse Properties

5.2.8 6.2.8 ExtensionNotification

5.2.9 6.2.9 ClearContextRequest/ClearContextResponse

5.2.9.2 6.2.9.2 ClearContextResponse Properties

5.2.10 6.2.10 StatusRequest/StatusResponse

5.2.10.2 6.2.10.2 StatusResponse Properties

5.3 A Modality Component States

E.1 F.1 Lifecycle event transport from modality components to Interaction Manager