Table of Contents

Speech and Avatar Flow

Last updated:

Avatar speech is the path from generated text to speech synthesis, queued audio playback, message-panel text, and lip sync in the Avatar Engine.

Note

Text-to-speech, often shortened to TTS, means converting written text into spoken audio.

Output modes

Admins can choose where assistant speech plays:

  • Browser only plays speech in the web browser.
  • Browser and avatar plays speech in the browser and queues the same clips to the avatar.
  • Avatar only queues speech to the avatar without browser audio.

Non-admin users resolve to browser-only output even when the saved admin setting includes the avatar.

Queue model

The current engine speech path is a clip queue:

  • speech/queue accepts an absolute audio URL plus the matching message text.
  • speech/queue-file accepts an audio file already available on the engine machine.
  • speech/stop stops current playback and clears queued speech.
  • speech/volume changes avatar playback volume.

Queued clips play in order. The engine shows each clip's Message text while that clip plays, which keeps subtitles and lip sync aligned with audio.

Oscova no longer uses a byte-stream speech route for the avatar path. It synthesizes a complete WAV clip, exposes it as a /tts/ URL, and queues that URL.

Chat read-aloud and voice mode

For chat read-aloud and voice mode, Oscova breaks the assistant reply into sentence-sized text chunks. Each chunk is synthesized into a WAV clip and exposed as a /tts/ URL. Oscova then posts that absolute URL to api/avatarbridge/speech/queue, and the host bridge forwards it to /avatarbridge/speech/queue on the engine.

When output is Browser and avatar, browser playback is the timing source. Oscova waits until the browser starts a clip, then queues the same clip to the avatar.

When output is Avatar only, Oscova queues clips directly and uses synthesized clip durations to estimate when voice mode can resume listening.

The first avatar-routed clip in an assistant reply triggers one readiness attempt for that reply. If readiness fails, Oscova keeps browser speech working when browser output is enabled and skips avatar queueing for that reply. If a later avatar queue post fails, Oscova marks the runtime unready and stops posting more avatar clips until the next reply.

Voice mode timing

Voice mode has to know when it can listen again:

  1. The user speaks.
  2. Oscova transcribes the audio.
  3. The agent replies.
  4. Oscova queues synthesized speech.
  5. Voice mode waits until the reply is complete, playback is complete, and no speech requests are pending.
  6. Voice mode resumes listening.

In browser playback modes, the browser reports when playback finishes. In avatar-only mode, Oscova estimates completion from the clip's sample count, sample rate, channel count, or duration metadata.

Agent tool speech

The avatar_speak tool synthesizes one complete clip with the configured Avatar Voice. It applies saved avatar settings first, builds an absolute /tts/ URL, and queues that URL through the bridge.

If saved settings cannot be applied, the tool reports that the avatar could not be prepared for speech.

Unlike chat read-aloud, avatar_speak uses the shared avatar voice saved on the Avatar settings screen. It does not use the current user's browser speech voice.

Message text without audio

Message text is also available without speech:

  • message/text shows text in the engine message panel.
  • message/clear hides the message panel.
  • The Avatar settings Send Preview and Clear buttons use these message endpoints.

This is useful when tuning message font size, color, opacity, position, size, and display duration.

Preconditions

Avatar speech requires:

  • Speech synthesis to produce a clip with a reachable /tts/ URL.
  • A bridge URL that points at a running engine.
  • Saved avatar settings that can be applied, or an already-ready runtime.
  • An admin output mode that includes the avatar.

Non-admin users resolve to browser-only speech output, even if avatar output modes exist in settings.

Stop behavior

When speech synthesis is turned off or voice mode stops after avatar speech has started, Oscova sends speech/stop through the host bridge. The engine is expected to stop current audio and lip sync and clear queued speech.

Keep ForcedStop set to false for the recommended graceful stop. Set it to true to interrupt immediately.