Operations & troubleshooting

Running the sync day to day: the heartbeat, disconnecting, why signed requests don't blindly retry, and how to run it on a schedule.

Heartbeat — status()

status = client.status()
print(status.provider_status, status.entity_counts, status.last_session)
if status.freshness and not status.freshness.bot_ready:
    print("catalog is stale:", status.freshness.staleness_health)

A signed GET that bumps the connection pulse and returns the connection status, per-kind entity counts, the last sync session summary and the server's freshness block (StatusResult). Throttle to at most once every ~5 minutes — it's a heartbeat, not a poll loop.

status.freshness is a FreshnessStatus (from monoverse_voicebot import FreshnessStatus), or None when the server sends no freshness block:

• staleness_health (str) — "ok", "warn" or "critical"; "critical" if the server leaves it out. "warn" means the catalog is older than the connection's freshness window, "critical" that it passed the server's critical threshold.
• bot_ready (bool) — a successful sync exists and staleness_health is "ok".
• last_successful_sync_at (str or None, ISO-8601) and staleness_seconds (float or None) — when the last successful sync finished, and how long ago.
• freshness_window_seconds (int or None) — the connection's freshness window.
• consecutive_error_count (int) — sync errors in a row.
• indexing_pending (int) — always 0 for now: /status does not report an indexing count yet, so bot_ready does not wait for indexing.

Disconnect — unpair()

client.unpair()

Tells the backend to disconnect, then wipes the locally stored credentials — even if the remote call fails, so you can always re-pair. Run it before pairing a different tenant.

Retry semantics

This is the rule that trips people up: a signed request is never blindly retried.

Path	Signed?	Retry policy
pair	no	up to 3 attempts on 429/5xx/connection errors (no nonce spent)
snapshot upload (PUT)	no (single-use URL)	not retried
send_events / finalize	yes	up to 3 resends by default after a 429, a 5xx or a connection error, each one re-signed
init / status / unpair	yes	one attempt by default

A signed request carries a single-use nonce the backend consumes the moment it arrives — before it can return a 429/5xx. A blind retry would replay a spent nonce and be rejected as a replay. So every resend is a resign-and-resend: a fresh X-VoiceBot-Timestamp and X-VoiceBot-Nonce, the same Idempotency-Key. send_events and finalize carry a stable key — each batch's batch_id, and for finalize(sync_id, idempotency_key=None) one key per call, generated if you pass none — so the server replays a resend instead of applying it twice. That is why those two resend by default and init_snapshot does not.

The number of resends is resolved per call, and the first match wins:

1. the call's own value — client.send_events(ops, resign_retries=N); only send_events takes one, and 0 means a single attempt;
2. the constructor's value — VoiceBotClient(..., resign_retries=N) sets every signed call, send_events and finalize included;
3. the operation's default — 3 for send_events and finalize, 0 for init_snapshot, status and unpair.

client.send_events(ops, resign_retries=0)  # one attempt for this call only

A 429's Retry-After is honoured up to 60 s; that ceiling is fixed. Without Retry-After the wait backs off exponentially — RetryPolicy(base_ms=250, max_ms=5_000) by default: 250 ms, doubling, capped at 5 s — plus up to 0.1 s of jitter. A single send_events call can therefore block for minutes; see Running on a schedule.

Logic errors don't raise

A 429/5xx on a signed call surfaces as a raised TransientError once its resends are used up (it's a transport failure); raised from send_events, it also carries the results of the batches sent before the failing one — see Syncing → When a batch raises. But per-op logic errors (a missing required field, a bad slug, money as a float) come back inside EventsResult.errors and do not raise — read them and fix the data. See Syncing → Reading the result.

426 Upgrade Required

The backend requires a newer sync protocol than this package speaks. There is no fallback — update the package (once the PyPI release lands, pip install --upgrade monoverse-voicebot; until then, pull the repo and reinstall from source):

Terminal

pip install --upgrade monoverse-voicebot

Running on a schedule

You own the cadence — the proven pattern is frequent deltas plus one periodic full. A minimal cron-driven script:

sync_worker.py

import sys
from voicebot import client
from catalog import collect_changed_ops, full_catalog

def delta_run():
    ops = collect_changed_ops()  # since your last watermark
    if ops:
        client.send_events(ops)

def full_run():
    client.snapshot(full_catalog())

# Wire these to your scheduler:
#   */15 * * * *  python sync_worker.py delta
#   0 3 * * *     python sync_worker.py full
if __name__ == "__main__":
    (full_run if sys.argv[1:2] == ["full"] else delta_run)()

On Django/Celery you'd register delta_run and full_run as Celery-beat tasks instead of cron.

• High-churn store → delta every 5 minutes.
• Typical store → delta every 15 minutes (above).
• Schedule the full for a low-traffic hour so the larger upload doesn't compete with peak load.

Never sync inside a request handler

A single send_events call can block for minutes: per batch (at most 500 ops or 5 MB) up to 3 waits of 60 s on 429s, plus up to 7 s of re-checks. snapshot() ends in finalize, which resends the same way. Call both from a worker, a cron job, a Celery task or CatalogScheduler, never inside a web request handler.

Don't overlap runs

A long sync that runs past its next tick must not start a second copy. Guard with your scheduler's overlap protection (or a lock).

Long-running process? Use CatalogScheduler

If your app is a daemon (a worker, a container that stays up), the package ships a scheduler so you don't hand-code the loop — background threads, a without-overlapping guard, and on_error / on_sync callbacks that keep the daemon alive through a failed run:

worker.py

from monoverse_voicebot import CatalogScheduler

scheduler = CatalogScheduler(
    full=lambda: client.snapshot(full_catalog()),
    delta=lambda: client.send_events(collect_changed_ops()),
    run_full_on_start=True,
)
scheduler.start()  # stop() on shutdown

Defaults: delta every 15 minutes, full every 24 hours (delta_interval_s / full_interval_s to change).

The cabinet's Synchronize button doesn't reach a producer

The Synchronize button in the Monoverse cabinet works by calling into your store over HTTP — it can only reach integrations that expose an inbound sync hook (the WordPress plugin, the Laravel package). A Python producer has no inbound endpoint, so that button does not apply to this stack: your cron/Celery/CatalogScheduler cadence is the trigger. To push immediately, run your own sync script; the cabinet's sync coverage panel reflects the result either way.

Encrypted credential storage

The shared secret is stored encrypted at rest (AES-256-GCM, cryptography) and is never logged, printed, or serialized — Credentials redacts it from repr() and str().

• Pass a full-entropy 32-byte encryption_key (openssl rand -hex 32); keep it in your secret manager / env.
• credentials_file writes the AES-GCM envelope to disk; the plaintext secret never touches disk.
• Omit credentials_file for an in-memory store — the secret lives encrypted even in process memory.
• For a KMS-backed at-rest key, implement CredentialStore and pass it as credential_store.

Rotating the encryption key

If you change encryption_key, the stored envelope can no longer be decrypted and the client raises a CredentialDecryptError. Re-pair after a key rotation.

Troubleshooting

Symptom	Cause & fix
ConfigError: …must use https://	base_url/ingest URL is plaintext http://. Switch to https:// (localhost excepted).
refusing to push an empty full snapshot	Your entity source yielded zero rows. Fix the source; never force-push an empty snapshot — it would wipe the catalog.
NotPairedError	No stored credentials. Call pair() first, or check that encryption_key/credentials_file match the run that paired.
426 protocol_upgrade_required	Update the package — there is no protocol fallback.
Per-op errors in the result	Validation/backend logic error. Common causes: missing required key, money as a float, category referenced by id instead of slug.
CredentialDecryptError on load	Wrong/rotated encryption_key. Re-pair.

Next

• Ingest Protocol Reference → Errors — the full status table and the no-retry-on-signed rationale.