Content Awareness Class

The experience of having a conversation with a digital person can be greatly enhanced by making the digital person aware of content on the screen, allowing them to gesture to that content when appropriate in the conversation.

For custom UIs, adding Content Awareness markup to your UI will provide a more engaging experience for end users by allowing the digital person to be aware of and react to elements on the screen.

For Content Awareness to work with a custom UI, the following requirements must be met:

The DDNA Studio project must have Content Awareness enabled
The custom UI must be using smwebsdk v14.1.0 or higher
The custom UI must identify elements:
- The video element must use the `data-sm-video` attribute
- Content elements must use the `data-sm-content="id"` attribute

Enabling Content Awareness

Content Awareness can be enabled on a per-project basis from DDNA Studio.

Enabling the feature in DDNA Studio will allow content awareness features to be used in any web UI using smwebsdk version 14 or higher.

You can check that your custom UI has content awareness activated by running your UI and accessing the hasContentAwareness() property on the Scene after the connection has been established:

let enabled = scene.hasContentAwareness();

// will log "true" if content awareness is enabled for the UI 
console.log(enabled);

Enabling Content Awareness for a project will not make any visible difference to the Digital Person's behavior until the web content in the custom UI has been marked up with Content Awareness HTML attributes.

Adding Content Awareness HTML Attributes

The Content Awareness API requires certain HTML elements to be identified so that their metadata can be automatically gathered by the smwebsdk library. HTML elements should be marked up using data-* attributes.

Elements which must be identified are the video element where the digital person is displayed, and any content elements that the digital person should react to, for example any content cards or a conversation transcript.

Identifying HTML elements in this way allows smwebsdk to track those elements and share metadata about their size and position with the server.

Identifying the video element

Add the data-sm-video attribute to the HTMLVideoElement being used to display your digital person.

Before:

<video></video>

After:

<video data-sm-video></video>

If more than one element has the data-sm-video attribute, only the first element will be identified and used for Content Awareness.

If the video element has been marked up but has a zero width and height, the smwebsdk will issue a warning and will block UpdateContentAwareness messages from being sent.

Identifying content elements

Marking up HTML elements with the data-sm-content="contentId" attribute will cause the digital person to glance at this content when it appears on the screen.

The contentId of an element must be unique among the elements currently present in the UI. Duplicates will be ignored.

For elements that represent content cards displayed using the @showcards(contentId) command from the conversational corpus, the data-sm-content value should match the card ID used in the corpus.

Before:

<img src="..." alt="...">

After:

<img data-sm-content="contentId" src="..." alt="...">

Alternatively, the contentId can be left empty and may be set at a later time. Content elements that have their data-sm-content attribute set to null or an empty string will be ignored.

Leaving the value empty or undefined can be useful if:

You do not know the id at the time
The content is not ready to be shown
The content is not visible

If a content element has been marked up but has no width or height, i.e. the bounding box is (0, 0, 0, 0) then the element will be ignored.

Configuration options

The Content Awareness functionality can be configured by passing config settings to the Scene constructor.

contentAwarenessDebounceTime: number (default: 300)

Content Awareness messages are debounced so that they are not sent more frequently than every 300ms to avoid unnecessarily high traffic on the websocket. This frequency can be changed if needed by passing a contentAwarenessDebounceTime in milliseconds.

let scene = Scene({ contentAwarenessDebounceTime })

Triggers for automatic content measurement

Elements will be remeasured when:

the web browser is resized
the device is rotated
an element with the data-sm-video attribute is added/removed from the DOM
an element with the data-sm-content attribute is added/removed from the DOM
an existing element has the data-sm-video attribute added/removed from it
an existing element has the data-sm-content attribute added/removed from it
an element's data-sm-content attribute has its value changed

Measuring content manually

For any instances where the automatic content measurement does not meet the requirements of the UI, the measurement functionality can be called manually to measure all tagged content elements and send a content awareness update to the server.

scene.sendContent()

Note that this function is unthrottled and may result in excessive websocket traffic if called too often.

Measuring on scroll events

When measuring on scroll events, make sure to only request a re-measure after the scrolling is complete so that the collecting and sending of metadata does not happen during scroll. This is important because each re-measure request may result in a change of camera framing.

Debouncing a scroll event so that it only sends after the scroll position has remained unchanged for some set duration can be achieved like so:

var timer = null;
var debounceTimeMs = 150;

window.addEventListener('scroll', function() {
  if(timer !== null) {
    clearTimeout(timer);
  }

  timer = setTimeout(function() {
    scene.sendContent();
  }, debounceTimeMs);
});

Measuring on move/drag

When measuring the position of elements as they move due to animation or dragging, make sure to debounce the event handler so that the collecting and sending of metadata does not happen too frequently. Triggering a re-measure may result in a change of camera framing.

If appropriate, the measurement of content elements could alternatively be deferred to occur on completion of the drag or animation.

draggableElement.addEventListener('dragend', function(e) {
  scene.sendContent();
});

Disconnect

All observers and event listeners will be removed automatically when the scene is disconnected at the end of a session.

scene.disconnect()

Content Awareness Metadata

The Content Awareness API captures metadata about elements that have been identified by the developer. This consists of each elements’ bounding box. For content elements, the ID provided is also sent.

Debugging

Use the developer tools to inspect the raw websocket message being sent to the server. Filter for messages with the name updateContentAwareness.