AniwaffleDocsHomeContact our team
Developer preview. Public API access is not open yet. Contact our team to discuss integration.

Public beta

Embed an Agent in your application

Connect a published Agent to a browser or React Native app using short-lived credentials. Includes native audio setup, lifecycle, and current SDK release status; permanent API keys stay on your backend.

Browser packages

Choose your integration

For a React app with a talking Avatar, start with @aniwaffle/react and useWaffle + AvatarStage. The published Agent supplies the character; the SDK handles audio, lip sync and motion. Use the web package for other browser frameworks. Select the headless client only when you deliberately own rendering.

@aniwaffle/react

Recommended for React: useWaffle + AvatarStage, with a diagnostics overlay enabled in development. Includes the client API.

@aniwaffle/web

Plain JavaScript and other browser frameworks: WaffleClient + mountWaffleAvatar, with a diagnostics overlay enabled in development. Includes the client API and the same renderer as React.

@aniwaffle/client

Headless connection, microphone, text, state, transcripts and diagnostics data. Your app deliberately owns its renderer and controls.

Package names

Move to the @aniwaffle scope

All public npm packages now use @aniwaffle: client, server, web, react, react-native, avatar, and cli. Use @aniwaffle/ in your package dependencies and imports, including subpaths such as @aniwaffle/server/realtime and @aniwaffle/avatar/performance-tags. Remove the previous package entries, upgrade the packages used by your application together, then regenerate your lockfile. Exported APIs such as WaffleClient, WaffleServer and useWaffle keep their names.

bash
# Example: migrate a React application and its server SDK
npm install @aniwaffle/react @aniwaffle/server

The old scope receives matching compatibility releases until December 9, 2026 at 00:00 Asia/Shanghai, then stops receiving updates. Existing published versions remain installable. The CLI migration also changes the command from waffle to aniwaffle; install it with npm install -g @aniwaffle/cli.

Native voice

React Native voice SDK

@aniwaffle/react-native provides a headless iOS/Android client and useWaffle(client) hook for foreground, bidirectional voice conversations. It shares the browser client's session and transcript protocol, with native WebRTC capture/playback and audio-session cleanup. Automatic speech turns are the default. Explicit interruption requires an application session; the backend retains control credentials and sends only the media session to the phone.

Install @aniwaffle/react-native@latest from npm. Request access to the native package README and Expo 57 example for installation, permissions and development-build configuration. The example uses local tarballs to verify the current source tree. Expo Go and this package's web rendering are unsupported. Leaving the app or unmounting the hook ends the call; real-device audio acceptance remains separate from typechecks and bundles.

Before you code

Three things you need

  1. A published Agent

    Build and publish it in the Dashboard first. Session tokens are only minted for published Agents.

  2. A workspace API key

    An Owner, Admin, or Builder creates it under Integrations in the Dashboard sidebar (direct link: /app/keys): enter a Label, select Create key, and copy the awp_… value immediately — it is shown exactly once. Full walkthrough in the API reference.

  3. The Agent’s id

    Your backend calls GET /v1/agents with the key and selects your Agent by its slug; the response’s id is the stable UUID used everywhere below. See Look up Agents.

Public beta

Install from npm

Terminal
# React applications (recommended for React)
npm install @aniwaffle/react

# Plain JavaScript / other browser frameworks
npm install @aniwaffle/web

# Deliberately headless / custom rendering
npm install @aniwaffle/client

@aniwaffle/react and @aniwaffle/web each include the client — install only the adapter that matches your app. The diagnostics APIs require client 0.5.3, React 0.5.2 or web 0.1.0. Check npm availability before upgrading; source examples can also use the repository tarballs.

Client 0.5.6 (Web 0.1.5 / React 0.5.7) and React Native 0.2.1 automatically report the installed core SDK version and runtime environment to the connected Agent. Performance shows this information per reply, including sessions created through a custom backend. Older clients show Not reported; the server cannot infer their installed version. Update the SDK dependency and rebuild the app to send version information. Native and custom playback may still lack a usable playback timestamp even with the latest SDK.

The default managed-session identity and sessionContext APIs were introduced in version 0.3.1. Use @latest for the current registry release.

Trust boundary

Mint session tokens on your backend

Permanent workspace keys belong on your backend. The browser asks your backend for one short-lived Agent token per connection; the SDK does the rest.

server.ts · your token endpoint
// Your backend. The awp_ key comes from Dashboard → Integrations
// and never leaves the server.
app.post('/api/animated-waffle/session-token', async (req, res) => {
  const user = await authenticateYourUser(req)
  if (!user) return res.status(401).end()

  const response = await fetch(
    `https://aniwaffle.com/v1/agents/${AGENT_ID}/session-tokens`,
    {
      method: 'POST',
      headers: {
        Authorization: `Bearer ${process.env.NARYA_API_KEY}`,
        'Content-Type': 'application/json',
      },
      // end_user_id: required when the Agent has memory or Calendar
      // enabled; must be a stable UUID for this user in your system.
      body: JSON.stringify({ end_user_id: user.stableUuid }),
    },
  )
  if (!response.ok) return res.status(502).json(await response.json())

  const { token } = await response.json()
  res.json({ token })
})
What Narya returnsHTTP 201 with { token, expires_at, agent_id, capabilities }. The token is valid for 15 minutes and starts exactly one Agent conversation.
Authorize your own user firstBefore minting, authenticate the user, verify they may talk to this Agent, and apply your own rate, duration, concurrency, and spend limits.
end_user_idOptional unless the Agent has conversation memory, documents, or Calendar enabled — then it must be a stable UUID for this user from your identity system. Send {} otherwise.

React

Connect and render with useWaffle

Use the React package when Aniwaffle should own Agent audio and Avatar playback.

Agent.tsx
import { AvatarStage, useWaffle } from '@aniwaffle/react'

export function Agent() {
  const agentId = '22222222-2222-4222-8222-222222222222'
  const waffle = useWaffle({
    agentId,
    sessionToken: () => fetchSessionToken(agentId),
  })

  return (
    <>
      {/* Session churn rebinds signals without removing the visible stage */}
      <AvatarStage {...waffle.avatar} style={{ width: 480, height: 480 }} />
      {waffle.avatarLoad.status === 'downloading' && (
        <progress value={waffle.avatarLoad.progress ?? undefined} />
      )}
      <button
        disabled={waffle.connectionState !== 'disconnected'}
        onClick={() => void waffle.connect()}
      >
        Start conversation
      </button>
      <button onClick={() => void waffle.disconnect()}>End</button>
    </>
  )
}

The hook returns the app-facing client state — connectionState, session, transcriptProgress, transcript, decline (the latest deliberate silence), agentSpeaking, muted/setMuted, performance (the cue the Avatar is performing right now), error, connect, disconnect, sendText, and the push-to-talk controls — plus an opaque avatar binding. Spread it onto a sized AvatarStage and the published character renders itself without exposing the remote audio track. The returned action functions keep stable identity across renders. avatarLoad reports first-load progress (downloading → verifying → parsing → ready with byte counts); repeat visits come from the browser cache after hash verification, so loading is near-instant.

Disconnect and reconnect only rebind session signals. The visible renderer is destroyed when AvatarStage unmounts.

Other browser frameworks

Mount an Avatar without React

ts
import { WaffleClient, mountWaffleAvatar } from '@aniwaffle/web'

const client = new WaffleClient({ agentId, sessionToken: () => fetchSessionToken(agentId) })
const avatarHost = document.querySelector('#avatar') // Give this element a width and height.
const avatar = mountWaffleAvatar(avatarHost, client, { onError: showError })
startButton.onclick = () => { void client.connect({ inputMode: 'automatic' }).catch(showError) }
stopButton.onclick = () => { void client.disconnect().catch(showError) }

// On page/component teardown:
function dispose() {
  avatar.destroy()
  void client.disconnect().catch(showError)
}

Mount once per visible host. Disconnect/reconnect keeps the character visible; destroy() releases the renderer and subscriptions. The renderer loads the published Avatar, supports the client's optional avatarMotion: true, and does not start another audio player. Do not pass model URLs or subscribe to /internal events. An Agent without a published Avatar remains audio-only.

Request the React example and plain JavaScript example for complete controls and a backend token exchange.

Development

Inspect a running session

Detailed content capture is separate from the floating panel's diagnostics visibility. SDK sessions default to off. In development or tests, use await client.connect({ detailedDiagnostics: true }) (also supported by React's connect) or pass detailedDiagnostics: true to server.startApplicationSession(...). A custom startSession callback receives the option and must forward it as detailed_diagnostics to the session-start API. server.sendMessage(...) accepts the same option per typed request. Dashboard Live enables capture automatically; environment names do not silently enable it.

Capture is frozen for each live session. It records assembled inputs, actual provider requests, raw model response text/chunks including audio tags, tool calls and Memory details. Payloads are bounded, credentials are redacted, and detailed content expires after 7 days. Truncated or interrupted responses are labelled; these records do not claim the entire response was played. Disabling capture leaves content-free latency, usage accounting and normal conversation history available.

client.diagnostics.server.waitDetails.memoryRecallMs and the memory_recall timeline stage measure how long the reply waited on recall. A spoken reply waits at most 800 ms, then answers from what the session already recalled about the user; recall that arrives later is not used for that reply. This duration is already included in input-to-LLM time: do not add it again. Memory retain is asynchronous ingestion submitted after the reply; its submission time is not first-audio latency. Missing recall measurements are not zero milliseconds.

For visible microphone feedback, render <WaffleMicrophoneControls waffle={waffle} /> from @aniwaffle/react. It includes an energy meter, the actual input label, a device selector that works during calls, and input warnings. For a custom UI, subscribe to client.on('inputLevel', ...) for { rms, peak } at up to 20 Hz, and client.on('microphone', ...) for device and mute state. null level means unavailable. Energy does not prove recognized speech; labels can be empty before permission. client.setInputDevice(deviceId) switches the connected input and cancels an active PTT hold. refreshInputDevices() never requests capture permission.

client.on('diagnosticEvent', ...) and client.diagnosticHistory expose connection, device, PTT, audio, playback and failure events. Compatible workers store these under the authenticated Session for internal-console review. Client reports include the actual device label and 5-second energy summaries, but no device IDs, audio, transcript text or credentials. Delivery is bounded and best effort; attempts that never reach a compatible worker remain in local history, which your application can forward through its own logging. When detailed capture is enabled, server workflow logs separately retain bounded original state/speech output and transition outcomes.

useWaffle and mountWaffleAvatar automatically show a collapsed Aniwaffle · Diagnostics button at the bottom right in development builds. Click it to inspect the floating panel; click again or press Escape while focused inside it to collapse. Production builds hide it by default. The panel follows the hook/Avatar mount lifecycle and makes no connection, microphone request, Dashboard fetch or feedback submission.

Set diagnostics: false in useWaffle options or mountWaffleAvatar options to disable the floating panel. Set diagnostics: true to explicitly show it. The default follows the consuming application's process.env.NODE_ENV === 'development', as replaced by its bundler; unbundled scripts or other build systems must use the explicit option.

For an inline panel, disable the automatic overlay and render <WaffleDiagnosticsPanel waffle={waffle} /> in React, or call mountWaffleDiagnostics(host, client) in another browser app. For your own UI, read client.diagnostics and subscribe with client.on('diagnostics', render); React also returns waffle.diagnostics. The headless client never adds UI.

The panel shows connection, microphone, playback, Avatar loading and motion state; token/session/transport/readiness durations; safe error stages/codes and suggested checks; and the server-authored link to the current Session in Studio. Studio authentication and authorization still apply. Connection phases can overlap; do not add them together.

The headline and full-width wait bar show client-perceived latency: microphone activity end to first playback, using one browser clock and including transport and playback buffering. Text submission timing is not yet recorded; text sessions never substitute a server duration for the client total. RTC uses decoded audio activity when its optional audio-level field is absent. RTC delivery and paired audio output are explicitly labelled estimates, not calibrated speech detection or acoustic/SLA measurements. Muted, interrupted, custom-playback, output-only, ambiguous or missing observations remain unavailable, never zero. Paired audio and motion share the output clock; poses are not counted as audible playback.

Updated workers also send the latest accepted input's ASR, LLM first-token, TTS first-audio and worker-output durations, computed by the same reducer as Studio. Below the client total, a separate server breakdown plots request spans on one shared server timeline: solid bars show waiting for first output, pale tails show continued generation, and aligned bars reveal overlap. The first-audio marker comes from the worker audio boundary, not from adding stages. Each span includes its own duration; later spans outside the first-audio window and omitted spans are called out. Text input skips speech recognition. Connection setup and technical IDs are collapsed by default. Workers without relative offsets fall back to duration summaries without inventing an order. Server and client rows may describe different inputs: compare the server trace and response identifiers in Studio before correlating them. Never subtract timestamps across machines or sum overlapping stages. Older workers leave server rows unavailable. Detailed motion-service/provider traces remain in Studio.

Diagnostic snapshots contain state, durations, correlation IDs and error codes, not transcripts, audio, API keys or raw error messages. Keep diagnostics disabled in customer-facing production builds; the existing content-free timing telemetry follows the SDK session lifecycle independently of panel visibility.

Framework independent

Connect with JavaScript

Use the client package when your application already owns rendering and controls.

client.ts
import { WaffleClient } from '@aniwaffle/client'

const agentId = '22222222-2222-4222-8222-222222222222'
const client = new WaffleClient({
  agentId,
  sessionToken: async () => {
    const response = await fetch('/api/animated-waffle/session-token', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ agentId }),
    })
    if (!response.ok) throw new Error('Could not start a session')
    return (await response.json()).token
  },
})

client.on('transcript', ({ id, role, text, final }) => {
  if (final) console.log(id, role, text)
})
client.on('declined', ({ reason }) => {
  settleCurrentTurn({ spoken: false, reason })
})
client.on('connectionState', (state) => console.log('connection:', state))

await client.connect({ inputMode: 'push-to-talk' })
Option / memberWhat it does
agentIdThe published Agent’s UUID from GET /v1/agents. Must be a UUID, not a slug.
sessionTokenA function (sync or async) returning a fresh token from your backend. Called on each connect. Alternatively pass startSession to start sessions through your own authenticated backend; the two options are mutually exclusive.
sessionContextDefault sessionToken starter only: optional { userName?, timeZone? }. Nonblank values are sent exactly as provided; blank values are omitted. Mutually exclusive with startSession, so token-path consumers do not recreate the raw managed-session request.
apiOriginOptional. Defaults to https://aniwaffle.com; leave it unset unless Narya gives you a different origin.
inputAudioEnabledOptional. Set false for text-triggered, output-only applications; the SDK never requests the microphone and PTT controls fail closed with input_audio_disabled. Defaults to true and stays off until the Agent is ready.
audioCaptureOptional standard microphone constraints (the audio fields of MediaTrackConstraints), applied on initial capture and reconnect. For an isolated virtual or already processed input use { echoCancellation: false }; keep it enabled for speakers. Browser support varies: inspect microphoneTrack.getSettings(). React also accepts this option.
audioPlaybackOptional, default true: the client plays the Agent’s voice through its own audio element. Set false to play the remoteAudioTrack yourself — an app routing the voice through its own element or Web Audio graph must turn this off, or the voice is heard twice.
connect({ inputMode })‘push-to-talk’ (default) or ‘automatic’ for an open microphone with voice detection. Resolves with the session.
sendText(text, { audioResponse })Sends a typed message into the conversation. audioResponse: false makes only that turn text-only.
mutedRead/write. Silences the Agent’s voice while the session, transcript, and Avatar performance carry on. No effect when audioPlayback is false.
startPushToTalk() / commitPushToTalk() / cancelPushToTalk()Hold-to-talk controls. Throw unless connected with inputMode ‘push-to-talk’.
on(‘connectionState’ | ‘session’ | ‘transcriptProgress’ | ‘transcript’ | ‘declined’ | ‘agentSpeaking’ | ‘remoteAudioTrack’ | ‘diagnostics’ | ‘error’, fn)Event subscriptions; each on() call returns its unsubscribe function. declined carries { reason } for deliberate silence. transcriptProgress is ephemeral ASR/playout caption state keyed by runtime segmentId. transcript events are completed persisted rows with the same required id as history and currently final=true.
disconnect()Ends the session and releases the microphone.

The managed SDK stays provider-neutral: 0.3 removes provider-specific overrides such as openaiModel; use the separate OpenAI Realtime page when that provider-specific path is required.

Media-vendor credentials, room credentials, and Avatar asset URLs stay inside the SDK — your code never handles them.

When it fails

Errors you will actually see

FailureCause and fix
Your token endpoint gets 409 “Publish the agent before requesting a session token.”The Agent is still a draft. Open the Agent editor and select Publish in the Dashboard.
Your token endpoint gets 400 memory_identity_required, documents_identity_required, or calendar_identity_requiredThe Agent has conversation memory, documents, or Calendar enabled but the request had no end_user_id. Send a stable UUID for the user.
Your token endpoint gets 401 unauthorizedThe awp_ key is missing, wrong, or revoked — check the Integrations page and your secret store.
The SDK throws ‘agentId must be a UUID.’A slug was passed as agentId. Resolve the slug to the id through GET /v1/agents on your backend.
connect() rejects with an expired-token errorTokens live 15 minutes and the sessionToken callback returned a cached one. Mint a fresh token on every call.
Push-to-talk methods throwThe client was connected with inputMode ‘automatic’. Reconnect with ‘push-to-talk’ or rely on voice detection.

Before launch

Production checklist

  • Publish and test the Agent in Live conversation before wiring the SDK.
  • Keep the awp_ workspace key only on your backend.
  • Bind end_user_id to a stable user in your own product.
  • Apply authorization, rate, concurrency, duration, and spend limits before minting tokens.
  • Handle disconnected and error states in the application UI.
  • Size AvatarStage and show avatarLoad progress on first load.

SDK migration

Upgrade from SDK 0.3 to 0.4

Version 0.4.0 is a breaking release of the client, server, and React packages. Character Card V3 is the complete character configuration. Replace these SDK fields before upgrading:

Previous fieldReplacement
Agent displayNamecharacterCard.data.name
Agent management descriptioncharacterCard.data.creator_notes
Agent personacharacterCard.data.description
Agent instructionscharacterCard.data.system_prompt
Session overrides.persona / overrides.systemPromptPublish fixed rules on the Agent; send changing facts as context
Authoring updatedPersona / updatedPromptupdatedCharacterCard
Authoring base revision persona / systemPromptcharacterCard
Authoring promptHashRemoved; candidate operations continue to use candidateHash

getAgent and listAgents read the latest published configuration by default. Unpublished changes do not affect these reads or new sessions. getAgent returns agent_not_published for an Agent without a publication, and listAgents omits it. Authoring uses getAgentDraft and listAgentDrafts; draftRevision is editing metadata and does not need to match activePublication.revision to start a session.

Card writes replace the complete card. Preserve its other fields when editing one part of an existing Agent:

ts
const agent = await waffle.getAgentDraft(agentId)
await waffle.updateAgent(agentId, {
  characterCard: {
    ...agent.characterCard,
    data: { ...agent.characterCard.data, system_prompt: 'Keep replies concise.' },
  },
})

Push-to-talk controls return promises so callers can await queued controls. Update custom WaffleSessionClient and UseWaffleResult implementations to match these signatures. WaffleSessionClient also exposes microphoneTrack, setInputMode, setMicrophoneEnabled, and interrupt. Existing calls that ignore the built-in client's return value still work; control failures continue to emit typed error events. Application sessions require an API and worker deployment advertising protocol 2 before the new consumer is enabled.

Backend extensions

Application tools and conversation state

The published Agent revision owns persona, policies, templates and workflow. The platform Runtime compiles one conversation prompt for Live and SDK. Given the same revision, modality, clock, context, history, memory and tool capabilities, both entry points generate the same model inputs.

Server SDK 0.8.0 uses structured context for application facts. application is optional; opening a session does not require copying the Agent prompt. Memory and authorized platform capabilities inherit the published Agent settings; an explicit memoryEnabled: false disables memory. Memory-enabled sessions require a stable authenticated endUserId. Custom application tools supplement authorized platform tools; duplicate names are rejected.

ts
const started = await api.startApplicationSession({
  agentId, chatSessionId, requestId, endUserId,
  context: { songlike: { phase: 'conversation', paused: false } },
})

The backend supplies history and JSON Schema tools through application. It joins a separate data connection to route inputs and execute its tools. Permanent workspace keys and the control grant stay on the backend. Browser session tokens and Viewer Companion credentials cannot start application sessions.

For application state callbacks, use application: { outputMode: 'speech_state', tools: [] }. Describe allowed state and transition policies in the Agent's saved decision configuration. onOutput({ id, inputId, state }) delivers completed nonempty state. Speech and the independent decision evaluator run in parallel; output has no playback synchronization guarantee. Validate state and permissions on the backend and reject stale or duplicate output. Keep immediate user stop controls on the application's independent path.

For direct routing, add inputRouting: 'direct', contextRevision: 0. Push complete fact snapshots with control.updateContext({ context, contextRevision }). Revisions must increase; duplicate and older snapshots are ignored. Direct updates require both fields and cannot append messages. maxInputCharacters optionally limits input to 1–12,000 characters. Direct mode requires tools: [] for application tools and an active control connection. The worker stamps the context revision on state output; the backend must reject outdated revisions.

The application inherits the Agent workflow, including node-specific model and character overrides. Updating context preserves the current node. Stable Agent/end-user identity recovers workflow state across sessions. Memory banks are scoped to workspace, Agent and end user. Context is JSON data and is never expanded as an authored template. It does not replace Agent instructions.

This is a breaking backend control contract: use server 0.8.0 with control protocol 3. Media remains protocol 2, so installed Songlike clients keep working with the upgraded backend; new integrations use client 0.7.0 and React Native 0.3.0. application.instructions, context-update instructions, and arbitrary legacy session rule overrides are rejected. Installed Streamer Assistant envelopes are converted into validated business context without forwarding their rule text. Move fixed rules into the published Agent and send changing facts as context. Coordinate API, worker and application backend upgrades; installed media and vtuber clients do not require an upgrade.

Previous-session history

application.historyMode accepts 'summary' or 'none'. AniWaffle preserves supplied history verbatim by default. To summarize older dialogue, set historyMode: 'summary', select a saved history_summary template that consumes {{history_summary_text}}, and include its marker in the Agent prompt or its saved conversation layout. The API only summarizes when the rendered prompt will actually contain the summary. historyMode: 'none' explicitly discards supplied history.

History items may carry at, the time the message was sent (ISO 8601 with an offset, server 0.8.1). Earlier user messages then show it in the session time zone, next to the current local time the prompt already has, so a new call does not treat an earlier conversation as just now. When summarizing and the last message is more than 30 minutes old, the whole history is handed over as a summary that says when it happened in local time, with no verbatim tail to continue.

Opted-in summaries use the resolved conversation model, bounded source input and a five-second deadline. The newest four bounded messages retain their roles and text. A summary failure preserves the original supplied history. Summary data stays in the session snapshot and is rendered through the selected template again for each turn. requestReply does not fabricate a user turn; the optional application_event template can describe it.

Completed idempotent starts reuse the prepared snapshot. New configuration applies to new sessions; existing sessions retain their frozen configuration. memoryEnabled remains a separate switch inherited from the Agent for long-term memory recall and retention. Setting both historical sources off does not disable within-call context or delete existing records.

ts
import { WaffleServer } from '@aniwaffle/server'
import { WaffleApplicationSession } from '@aniwaffle/server/realtime'

const api = new WaffleServer({ apiKey: process.env.ANIMATED_WAFFLE_API_KEY! })
const started = await api.startApplicationSession({
  agentId,
  chatSessionId: crypto.randomUUID(),
  requestId: crypto.randomUUID(),
  memoryEnabled: false,
  context: { collectionId: 'my-records' },
  application: {
    history: [],
    historyMode: 'summary',
    tools: [{
      name: 'search', description: 'Search records.',
      parameters: { type: 'object', properties: {}, additionalProperties: false },
      cancellable: true, onDuplicate: 'reject',
    }],
  },
})
const control = new WaffleApplicationSession(started.control, {
  async handleRequest(request, { signal, update }) {
    switch (request.type) {
      case 'input': return { action: 'reply', text: request.text }
      case 'tool':
        await update({ status: 'Searching' })
        return searchAuthorizedRecords({ signal })
      case 'transcript': await saveTranscript(request); return null
      case 'prepare_reply': return { context: await loadCurrentFacts(request.id) }
    }
  },
  onError: reportError,
})
await control.connect()
// Keep control alive with the application's session owner.
return Response.json(started.session)

Install the optional Node transport with npm install @aniwaffle/server@latest @livekit/rtc-node@1.1.0. Importing the server SDK's main entry does not load the native transport. The backend must authenticate its caller and authorize all tools before creating a session. The control token expires after five minutes for joining, is room-scoped, and cannot publish or subscribe to audio/video. It can send data to its worker. Only started.session is returned to the browser:

ts
const client = new WaffleClient({
  agentId,
  application: true,
  startSession: ({ region, requestId, sessionId, inputMode, signal }) =>
    fetch('/api/voice/session', {
      method: 'POST', signal,
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ region, requestId, sessionId, inputMode }),
    }),
})
await client.connect()

The backend callback receives:

  • input: finalized text, id, and modality (audio or text). Return { action: 'handled' } or { action: 'reply', text, context? } before the foreground model responds.
  • tool: a declared name, arguments, and tool-call id. Validate arguments and authorize the operation. Await update(value) to report JSON progress. Its first update makes the tool non-blocking; subsequent updates and the final result use LiveKit's idle-aware delivery. Progress can trigger conversation.
  • transcript: durable message id, role, and text. Persist idempotently and acknowledge. This direct backend delivery does not rely on the browser.
  • prepare_reply: a spoken update id. Return { context } after preparing current context and any visual focus needed before speech.

control.updateContext({ context, messages }) replaces the fact snapshot and appends any supplied messages without requesting speech. Await each update to preserve application ordering. control.requestReply(id) explicitly queues a spoken update, asks the backend to prepare it when idle, and returns { id, interrupted, completion: 'worker_playout' }. This means the worker's playout completed or was interrupted; it is not a client speaker acknowledgement. Reusing an ID in the same connection does not speak twice. New user input invalidates queued updates. The application owns replay across connections.

Tools default to cancellable: false, onDuplicate: 'allow', and runReply: true. Native duplicate policy (allow, reject, replace, confirm) compares concurrent tool names, not business arguments. Use runReply: false for dispatch tools whose eventual spoken result arrives via requestReply; business idempotency still belongs to the application. control.cancelTool(toolCallId) requests explicit cancellation. After its first progress update, an async tool survives speech interruption. Tools without progress retain normal blocking-tool interruption behavior. Honor the callback signal for connection-bound work; a durable business task needs its own owner and explicit business cancellation, independent of audio and session lifetime.

Browser controls remain client.setInputMode('automatic' | 'push-to-talk'), client.setMicrophoneEnabled(enabled), client.interrupt(), and PTT controls. client.inputLevel and the inputLevel event support microphone meters without another capture stream. PTT promises settle after the existing control queue; failures also emit error.

The start request has a 256 KiB body limit, with at most 200 history messages and 64 tools. Application text streams allow 256,000 characters per envelope and 128 pending calls. Requests need a result or first tool progress within 60 seconds; acknowledged async tools then share the session lifetime. Responses use type (progress, result, error) and the same UUID requestId in the body and stream attributes. Values must be JSON-serializable; undefined results become null. Late responses are ignored after completion or cancellation. Backend commands use application.command / application.command_result; worker callbacks use application.request / application.result and application.cancel. Only the bound backend identity can send commands/results; only the current Agent can invoke callbacks. Backend control requires animated-waffle.application.control-version=3; older workers fail closed. The unchanged media capability remains animated-waffle.application.version=2, so installed media clients can connect after the backend is upgraded.

Request the runnable application-session example to check slow tools, speech interruption, cancellation, context-only updates, reply completion, and connection cleanup without depending on Canvas.

Deployment

Select a region

Set region: 'jp' to use the Japan API and Agent. Set region: 'us' for the US. Waffle Live speech requires region: 'jp'; US Waffle sessions are rejected before starting. Stereo Waffle voices do not support realtime Live sessions; choose a mono voice. The SDK chooses the endpoint; production callers do not need to know its hostname. If you omit both region and apiOrigin, the default is the United States; the SDK does not automatically choose the nearest region. Choosing a nearby region can reduce network latency, including the wait for the first audio response. Model processing and queueing still contribute to total latency. This is available in Client 0.5.10, React 0.5.11, Web 0.1.9, React Native 0.2.5, and Server 0.7.2. Upgrade the package your application imports.

Browser · Japan production
import { WaffleClient } from '@aniwaffle/client'

const client = new WaffleClient({
  agentId,
  sessionToken, // Short-lived token from your backend.
  region: 'jp',
})

await client.connect()

Alternatively, client.connect({ region: 'jp' }) selects the region for that connection and overrides the constructor default. Disconnect before changing an active session's region. React accepts the same option in useWaffle({ agentId, sessionToken, region: 'jp' }); Web and React Native use WaffleClient.

For DEV, combine apiOrigin: 'https://dev.aniwaffle.com' with region: 'jp'. The SDK selects the Japan DEV endpoint and keeps DEV credentials separate from PROD. Region selection remaps only Aniwaffle's known production and DEV origins; custom proxy, self-hosted and localhost origins stay unchanged. Omitting region preserves the existing API origin and its default Agent region.

A custom startSession callback receives the resolved region. Forward it to your backend and then to startApplicationSession; the SDK cannot reroute an application-owned HTTP request. The server SDK supports the same constructor default and per-session override.

Backend · Japan production
import { WaffleServer } from '@aniwaffle/server'

const server = new WaffleServer({
  apiKey: process.env.ANIMATED_WAFFLE_API_KEY!,
  region: 'jp',
})

const grant = await server.startApplicationSession(sessionInput)

For direct REST calls, use the selected region's API hostname and send POST /v1/agents/:agentId/sessions with "region": "jp" or "region": "us". Omit it to use the API endpoint's default. A region change starts a new session with a new idempotency key. The Web Live settings use the same selection. External ASR/LLM provider locations remain provider-specific.

Read and select conversation workflow state

A published Agent owns the conversation workflow. Applications read the current node with server.getWorkflowState({ agentId, endUserId }). The result contains revision, sequence, initialNodeId, nodeId, nodeName, and the current node's outgoing transitions. It is null when the Agent has no published workflow or its published workflow is disabled. Reading does not create a conversation.

During an application session, control.getWorkflowState() reads the same durable owner. To apply a user's explicit selection, call control.transitionWorkflow({ id, edgeId, expectedRevision }) using a transition and revision from that response. Only the authenticated backend control participant can select a transition. Reuse id when retrying the same operation. A stale revision raises workflow_conflict and requires a fresh read. A successful selection invalidates pending background decisions, interrupts the previous reply, and applies the node to subsequent input. It does not generate a reply. Call requestReply(id) separately when the product requires an opener.

onWorkflow(state) receives committed state changes from the worker. Ignore notifications with a lower sequence than the latest observed state. The HTTP reader and reconnects recover the durable state if a notification is missed. Keep device enablement, pause, output limits, and immediate stop controls in the embedding application. Conversation modes and relationship stages belong in workflow nodes, so applications should display the returned node instead of interpreting a second model-generated mode field.

These methods require @aniwaffle/server 0.7.1 or later to query or select Workflow state before the browser or mobile media participant joins. Connect the backend control participant first. Reply and context commands wait until the media connection is ready. Existing application state callbacks and context updates remain available for device requests and runtime facts.

Reproducing application inputs in Live and Tests

Live, Tests and Session Replay accept context plus explicit testInputs containing history, memory, an ISO now timestamp, and tool definitions with fixed result values. Omitted application history reuses managed conversation history; an explicit empty array starts without imported messages. Supplied test history is prepended to the test conversation; memory replaces recall. Supplying tools (including an empty array) replaces real tools for that test. Preview and run pin these inputs in the configuration hash. Old sessions do not contain all historical external state; missing data is not reconstructed automatically.

OpenAI Realtime also requests its prompt from the same server compiler on each turn; the browser handles media and carries server-produced instructions. Its explicit test inputs support history, memory, time and mock tools. Managed calendar, document, long-term memory and workflow evaluation still run on the managed transport.

Contact our team

Tell us about your character and what you’d like to build.