Documentation

LyraType Tide animates a virtual character's expressions and upper body from speech, in real time. LyraType Moon keeps the character listening between turns and lets it decide when to speak. Each works on its own. Together, the moment that starts the character's speech also cues its face.

Tide sits between a voice and an avatar. It listens to the conversation and moves the avatar continuously, whether the character is speaking, listening or waiting.

Input

Speech audio, streamed as it is produced

Output

The avatar's expression and upper-body motion

Mouth

Left to your own lip sync, driven by the same audio

Avatar

Your existing Live2D model

Tide drives the avatar through the software you already use, so a model that works with face tracking works with Tide.

Moon runs alongside your speech recognition and speech synthesis. It keeps listening between turns, so it answers quickly and can also speak without being asked.

Input

What the character hears, as text, as it arrives

Output

The words to speak

Works with

Your speech synthesis for the voice, and Tide for the face

During the research preview Tide drives avatars through VTube Studio's public plugin API.

1. 1In VTube Studio, open the settings and turn on the plugin API (it listens on port 8001 by default).
2. 2Start the Tide client. VTube Studio asks once whether the plugin may connect; allow it.
3. 3Tide then drives the model. On models that have body parameters the upper body moves too.
4. 4Keep your own lip sync running: it still drives the mouth, and Tide does not take it over.

LyraType Studio drives your own avatar in VTube Studio with our voice and motion, from lyratype.com/studio. Open it in Chrome or Edge on the computer that runs VTube Studio; there is nothing to install.

1. 1Open lyratype.com/studio in Chrome or Edge on the computer that runs VTube Studio.
2. 2Paste the studio token from your LyraType email. It starts with lt_.
3. 3In VTube Studio, open Settings, find VTube Studio Plugins and switch on Start API (allow plugins). The port is 8001 unless you changed it.
4. 4When the browser asks whether lyratype.com may reach apps on this computer, choose Allow. If you blocked it, open Site settings from the icon left of the address and set Apps on device to Allow.
5. 5VTube Studio shows a pop-up for the plugin "LyraType Studio". Press Allow, and only right after your own click on lyratype.com/studio.
6. 6Let the model check run. If an input is missing, connect it to your model's parameter in VTube Studio's model settings, or go on without it.
7. 7Install a virtual audio cable such as VB-Audio Virtual Cable and choose CABLE Input as the TTS output. In OBS, add an Audio Input Capture of CABLE Output and turn on its audio monitoring to monitor it yourself.
8. 8Press Say hello and watch your model and OBS. If motion runs ahead of the audio in OBS, raise Motion delay in Settings.
9. 9On Character, write the persona, choose a voice and set the motion parameters. On Live, start a talk.

Write to us with a few lines about what you are building, the avatar software you use, and whether you would run the models in the cloud or on your own devices. We reply by email.

Cloud API

The main way to use Tide and Moon. Tide's real-time API is open for a free trial: write to us and we send a token by email. Tell us what you think.

Local licence

The models run on your own devices and work offline.

Custom work

Tuning and integration for a particular character or setting.

Open a WebSocket, send the character's voice as it is synthesised and the user's voice as it is heard, and every third of a second a block of ten frames comes back, each frame stamped by its index: frame f belongs at f / 30 s of session time. Play the sound and the frames against the same clock, holding the display about a tenth of a second behind it so a block is in hand before its first frame is due.

GET /v1/session

The WebSocket. One session per socket.

POST /v1/sessions

Optional admission for a backend that lets its own users in: returns a session id and a single-use ticket, valid 60 seconds, that opens one socket without the token.

GET /healthz

Load of the host: sessions, the cap, the server's time per tick (a tick is a third of a second of motion). No token.

Token

Authorization: Bearer <token>, or ?token= on the socket address (browsers cannot set headers on a WebSocket). A ticket goes as ?ticket= or as the Sec-WebSocket-Protocol value.

The first message gives the character's style settings (preset) and a seed; the reply lists the channels of every frame. The style settings are fixed for the session: different settings mean a new session. Two sessions with the same audio, settings and seed move the same way.

{"type": "start", "preset": {"smile_rest": 0.6, "head_activity": 0.5}, "seed": 0}

{"type": "started", "session": "9f2c…", "fps": 30, "commit": 10,
 "channels": ["head_yaw", "head_pitch", "head_roll", "gaze_x", "gaze_y",
              "eye_open_left", "eye_open_right", "brow_left", "brow_right", "smile",
              "chest_yaw", "chest_pitch", "chest_roll", "sway_x",
              "mouth_press", "mouth_pucker", "smile_skew"]}

smile_rest

0 mouth at rest … 1 smiling at rest

brow_rest

0 brows low … 1 brows raised

head_activity

0 still head … 1 lively head

blink_rate

0 rare blinks … 1 frequent blinks

gaze_activity

0 steady gaze … 1 wandering gaze

listen_activity

0 still while listening … 1 moving while listening

listen_nods

0 few nods while listening … 1 many nods

Any style setting left out takes the model's middle value. Session time zero is the moment of the started message; audio stamps and frame indices count from it.

Audio travels in binary messages: one byte for the voice (0 the character's own, 1 the other speaker), eight bytes little-endian float64 for the session time at which the first sample plays or was heard, then little-endian 16-bit mono samples at 16 kHz. Blocks can be any length; one every 50 to 200 ms is usual.

1. 1Forward the character's audio the moment the synthesiser delivers it, stamped with the time it will play. Nothing has to be delayed for Tide; the earlier it knows the voice, the better the face follows it.
2. 2Play as early as you like. Holding playback back is your choice: none for conversation, up to half a second for a video or a monologue, where the motion follows the voice more closely.
3. 3Send the other speaker's voice as it is heard; there is nothing to send ahead of time.
4. 4After an interruption, withdraw what was not played: the audio from that time on is void.
5. 5Tell the session who is speaking: the spans of each voice as you know them. Spans shorter than a second count as backchannels.

spans

{"type":"spans","voice":"own"|"partner","spans":[[start_s,end_s],…]} — all spans of that voice known so far; replaces the last list. An open span ends at the present.

withdraw

{"type":"withdraw","at":seconds} — own audio from that time on was not played; the own spans are cut there.

ping

{"type":"ping"} — answered with {"type":"pong","t":seconds}. A socket with no audio and no ping for 60 seconds is closed.

close

{"type":"close"} — ends the session, as closing the socket does.

One binary message a tick: a 32-bit unsigned first-frame index, a 32-bit unsigned count (10), then count × channels little-endian 32-bit floats, row by row, in the channel order of the started message. A block is drawn at the moment its first frame is due and reaches you one tick's work later, 40 to 70 ms on an idle host and up to a third of a second when the host is full.

Seventeen numbers a frame, in the tracker's units. A display maps them to its rig once; the same mapping serves every character of that rig. The rest column gives the pose a session starts from.

head_yaw, head_pitch, head_roll

Radians; left and up positive. A lively nod is about 0.1. Rest about 0.

gaze_x, gaze_y

−1 to 1 of the eye's travel. Rest 0 and 0.03.

eye_open_left, eye_open_right

0 shut to 1 wide; relaxed open 0.65; a blink dips to 0. Rest 0.65.

brow_left, brow_right

0 low to 1 raised, absolute. Rest 0.33 and 0.28.

smile

0 to 1, absolute: 0.37 a neutral mouth, 0.7 a full everyday smile. Rest 0.37.

chest_yaw, chest_pitch, chest_roll

Radians of the upper body. Rest 0.

sway_x

Sideways lean, −1 to 1. Rest 0.

mouth_press

0 to 1, lips pressed together. Rest 0.1.

mouth_pucker

0 to 1. Rest 0.1.

smile_skew

Left corner minus right, −1 to 1; a smirk is about 0.3. Rest 0.

The mouth's opening for speech is not a channel: your lip sync drives it from the sound, and the model's smile, mouth_press, mouth_pucker and smile_skew go on top.

import asyncio, json, struct
import numpy as np, soundfile, websockets

URL, TOKEN = "wss://<host>/v1/session", "<token>"

async def main():
    own, _ = soundfile.read("own.wav", dtype="int16")  # mono, 16 kHz
    headers = {"Authorization": f"Bearer {TOKEN}"}
    async with websockets.connect(URL, additional_headers=headers) as ws:
        await ws.send(json.dumps({"type": "start", "preset": {"smile_rest": 0.6}, "seed": 0}))
        channels = json.loads(await ws.recv())["channels"]
        # the character's voice, stamped to play at 0.5 s, handed over ahead of playback
        await ws.send(bytes([0]) + struct.pack("<d", 0.5) + own.tobytes())
        end = 0.5 + len(own) / 16000
        await ws.send(json.dumps({"type": "spans", "voice": "own", "spans": [[0.5, end]]}))
        while True:
            message = await ws.recv()
            if isinstance(message, bytes):
                first, count = struct.unpack_from("<II", message)
                rows = np.frombuffer(message, dtype="<f4", offset=8).reshape(count, len(channels))
                print(f"frames {first}-{first + count - 1} at {first / 30:.2f} s", dict(zip(channels, rows[0].round(3))))

asyncio.run(main())

Message size

8 MB at most. Own audio may run as far ahead of playback as you like; the service keeps what it needs.

Unread frames

A client that leaves about 30 seconds of frames unread is dropped (close code 1013).

Capacity

One host serves 16 to 20 sessions. Above that, or when the host's work passes 85 % of real time, new sessions are refused with capacity; running sessions are never slowed.

limit

Your token's own caps, sent with it: sessions open at once and minutes a day (UTC). Past either, a new session is refused with limit (a ticket request gets HTTP 429); a session whose minutes run out is closed with limit and close code 1008.

First connection

The host runs only while someone is connected. After a quiet spell the first connection waits about 10 seconds while it starts; the next ones are immediate. Call GET /healthz when your user arrives, so the start is over before the first sound.

unauthorized

No token, a wrong one, or a ticket used twice or too late. The socket is closed after the error.

bad_request

A malformed message; the socket stays open. A bad start message closes it.

internal

The session failed on the host; the socket is closed. Open a new one.

Versioning

The path is /v1. New channels at the end of the list, new message types and new fields do not change the version; a change in the meaning or order of existing ones does. Read the channel list from started and you are safe.