Wire protocol (reference)

This is the contract the SDK speaks, transcribed from the running backend. It is protocol version 1 and is tracked by this docs line (/v1/*). Both SDKs implement it identically — you do not need a backend checkout to build against it.

This is the ground truth

Any older aspirational docs (POST /v1/voice/session, session_ready, tool_result, tools_enabled, text_message, open_product) describe a non-running protocol. Do not build against them. If anything disagrees with this page, this page wins.

Two divergence traps (memorize)

1. The backend reads user_text, not text_message. It reads action_ack, not tool_call_response. The client-message handler accepts only: ping, hello, mic_state, action_ack, navigation_pending, user_text, cart_event, client_context, end.
2. UI tool calls are executed server-side and pushed to the client as a fire-and-forget action message; the client replies action_ack. The client does not compute or return a tool result for these — the server already produced the Gemini tool result.

1. Auth — session-token issuance

A tenant has one VoiceBot key = paired tenant_id + a shared secret. The same key authorizes the website widget (origin-bound) and the mobile app (bundle-id-bound). The SDK never holds the shared secret; it obtains a short-lived session token.

Billing kill-switch: tokens are minted only while provider_connections.status == "connected". Flip the status and every call fails, active sessions are revoked ≤30s.

Mode A — server-minted (recommended)

POST /api/v1/mobile/issue-token
X-VoiceBot-Tenant-Id:  <uuid>
X-VoiceBot-Timestamp:  <epoch seconds>          # ±300s window
X-VoiceBot-Nonce:      <single-use, 600s TTL>
X-VoiceBot-Signature:  <hex>
X-VoiceBot-App-Id:     <ios bundle id / android package>

{ "lang": "uk" }            # optional body

200  { "session_token": "<JWT>", "expires_at": <epoch>, "refresh_after": <epoch> }

Signature preimage — exact, do not guess:

sig = HMAC_SHA256(secret, "{METHOD}\n{path}\n{timestamp}\n{nonce}\n{sha256_hex(raw_body)}").hex()
# e.g.  "POST\n/api/v1/mobile/issue-token\n1718000000\n<nonce>\n<sha256hex(body)>"
# body_sha256 = sha256(raw_request_body_bytes).hexdigest()   (empty body → sha256(b"") hex)

Mode B — publishable key

POST /api/v1/mobile/issue-token-public
{ "api_key": "vb_pk_…", "app_id": "<bundle/package>", "attestation": "<optional>", "lang": "uk" }

200   { "session_token": "<JWT>", "expires_at": <epoch>, "refresh_after": <epoch> }
4401  key inactive
4403  app_id not in allowlist

The session token

HS256 JWT, claims { iss, aud, iat, exp(~3600s), tenant_id, shop_id, app_id, language, channel:"mobile", allowed_app_ids:[…] } — allowed_app_ids replaces the web token's allowed_origins. TTL 3600 / refresh-after 1800. Store in Keychain / EncryptedSharedPreferences; never log. The SDK re-issues before refresh_after.

Full narrative with SDK snippets: Auth & API keys.

2. WebSocket open

GET wss://<host>/api/v1/widget/ws?tenant_token=<session_token>&conversation_id=<optional>&lang=<optional>&protocol=1

The WS endpoint is shared with the web widget; the backend branches on channel == "mobile" to skip the origin check and assert app_id.

Fatal close codes — do NOT reconnect:

Code	Meaning
4400	malformed
4401	invalid/expired token (or key blocked)
4403	app-id not allowed
4503	voice provider down
1008	session limit
1011	server crash

All other drops → reconnect (backoff), capped at ≤3 attempts.

3. Handshake (server → client, immediately on open)

{ "type": "ready", "conversation_id": "…", "language": "uk", "issued_at": 1718000000 }

{
  "type": "session_state",
  "state": {
    "conversation_id": "…",
    "session_id": "…",
    "language": "uk",
    "bot_state": "…",
    "voice_enabled": true,
    "reconnected_via_resume": false
  }
}

No voice_config message

There is no voice_config on this path. Hardcode the sample rates: uplink 16000 Hz, downlink 24000 Hz.

4. Audio uplink (client → server)

Raw binary WS frames (no JSON envelope). The server reads the bytes and forwards to Gemini as audio/pcm;rate=16000.

• Format: PCM16, mono, signed little-endian, 16000 Hz.
• Frame size: 20–40 ms recommended. 20 ms = 640 bytes, 40 ms = 1280 bytes. The backend is frame-size agnostic.

5. Text turn (client → server)

{ "type": "user_text", "text": "…" }

(Not text_message.)

6. Downlink (server → client)

• Binary frames = assistant audio, PCM16 mono 24000 Hz LE. Decode Int16 → float for playback.
• { "type": "transcript", "role": "user" | "model", "text": "…", "partial": true }
• { "type": "turn_complete" }
• { "type": "barge_in" } → flush the playback buffer immediately.
• { "type": "tool_call", "tool_call_id": "…", "name": "…", "args": {…} } → informational ("the bot is doing something"); show it in the UI.
• { "type": "action", "action_id": "…", "action": "…", "params": {…}, "context": {…}, "issued_at": …, "timeout_ms": … } → the actionable UI command (navigate / open_product / add_to_cart / apply_filter / …). Run the registered handler or fire the deep link, then reply action_ack (§7).
• Also: session_state, session_limit_exceeded, error, navigation_pause_acknowledged, hello_ack, pong.

7. Client control / ack / context (client → server)

• { "type": "action_ack", "action_id": "…", "ok": true, "result": {…} } — confirm a UI action ran.
• { "type": "ping", "ts": <ms> } → server { "type": "pong", "ts": … }. Heartbeat ~every 25s; if no pong in ~30s, close 4000 and reconnect.
• { "type": "hello", "protocol_version": 1, "sdk_version": "…", "is_leader": true, "capabilities": {…} } → hello_ack.
• { "type": "mic_state", "state": "granted" | "denied" | … }.
• { "type": "client_context", "page_context": {…}, "discovered_urls": [...] } — app-side context (§10).
• { "type": "cart_event", … } — cart snapshot / changes (§10).
• { "type": "end", "reason": "user_closed" } → real termination. Send only on explicit close (X) / endSession(), never on minimize.

8. Close & reconnect (the minimize ≠ close contract)

• WS drop (network blip, app backgrounded, minimize-that-drops-socket) → the backend keeps Redis state alive 60s. Reconnect to the same conversation_id within 60s → the server resumes the Gemini session. Outside 60s → fresh session, prior context recapped from DB.
• minimize() = UI-only. Keep the WS open (preferred) or let it drop and rely on the 60s reattach. Never send end.
• endSession() = send { "type": "end", "reason": "user_closed" }, then tear down mic/playback/WS.
• Bot-initiated end = server sends a terminal session_state + closes; emit ended, tear down.

See Lifecycle for the SDK-level view.

9. Tool action handling (DOM-less mobile profile)

Server-side local tools (search_catalog, get_product, find_similar_products, compare_products, list_subcategories, list_top_categories, site-nav resolution, end_session) are unchanged — the bot speaks the results; the client does nothing.

For action messages (client-executed tools), map each to a merchant handler / deep link, then action_ack:

action	Mobile mapping
open_product / navigate / open_category / view_cart / open_checkout	deep link → merchant route; fallback native callback
add_to_cart / remove_from_cart / update_cart_qty	native callback (merchant cart API); no handler → { ok: false, error: "no_handler" }
apply_filter / filter_by_attribute / clear_filter	native callback (merchant list)
highlight / scroll_to	native callback or no-op ok: true if the list is not addressable
show_toast	native callback (snackbar)
set_delivery_method / set_payment_method / update_order_note / confirm_order	native callback if implemented, else disabled (user_must_confirm)
submit_form	server-side plugin callback unchanged, or native override
open_popup / switch_language	native callback or disabled

Unregistered tool → the SDK auto-replies { success: false, error: "no_handler" } without crashing. The bot adapts. SDK-level guide: Tool handlers & deep links.

10. Context enrichment (app → bot)

The backend gets catalog data from the merchant's e-com backend (ingest), but the app can supplement it with live client-side context. This reuses existing backend frames — no backend change:

• client_context — sent on session start and on screen changes via updateContext({…}). Carries an AppContext: current screen/route, current product id, viewed category, locale, auth state (a loggedIn boolean, not PII), and arbitrary merchant custom key/values. Zero PII.
• cart_event — sendCartEvent({…}): cart snapshot + add/remove/qty changes, so the bot knows the cart without a round-trip.
• hello.capabilities — declare { voice, text, native_audio, handlers: [registered tool names] } so the bot only offers tools the app can actually execute.

Keep all of it PII-free (zero-PII guarantee). SDK-level guide: Context enrichment.