Watchtower: canary probe & health alerts

Pitchbar watches itself, so the operator finds out about problems before visitors do — not the other way round. Three monitors run from the scheduler and raise alerts through the same channels as the Widget Monitor: bell, browser push and (when configured) WhatsApp, throttled so an error storm pages once, not hundreds of times.

The canary probe

pitchbar:canary runs a real conversation against every published agent every five minutes — over public HTTP, through the proxy, Octane, retrieval, the LLM and the SSE stream. If a visitor would hit an error, the canary hits it first.

• The probe's conversation is playground-flagged (no billing, no analytics, no content gaps, no operator alerts) and deleted the moment the probe finishes — the only residue is one visitor row per agent.
• A single failed probe is counted but not alerted — blips self-heal. Two consecutive failures (ten minutes apart at most) raise a canary_failed alert naming the agent, the failing stage (init / stream / latency) and the detail.
• A workspace at its message quota is reported as a skip, never an outage.
• Opt-in per install: CANARY_ENABLED=true — each probe spends one real LLM turn per agent. Budgets: CANARY_TTFT_BUDGET_MS (default 25000), CANARY_TOTAL_BUDGET_MS (default 90000), CANARY_BASE_URL (defaults to the app URL).

The queue heartbeat

Every five minutes pitchbar:queue-heartbeat drops a tiny job onto each monitored queue (OPS_MONITORED_QUEUES, default analytics,default,integrations,index,crawl). The job's only work is stamping "this queue consumed something at time T". A stamp proves consumption, not dispatch — a dead worker leaves the job queued and the stamp ages.

pitchbar:health-scan checks the stamps: one older than 15 minutes raises a queue_stalled alert naming the queue. This permanently closes the silent-dead-worker class of failure, which twice went unnoticed for over a month.

The health scan

Besides heartbeats, the five-minute scan checks two more signals:

• Error traces — visitor turns that died with a server-side exception raise a turn_errors alert with a sample of the question that failed.
• Log fatals — hard PHP failures (e.g. "Maximum execution time exceeded") never produce a trace because the process dies; the scan tails laravel.log and raises a log_fatals alert for fresh ERROR entries. A cursor guarantees each entry alerts once.

Watchdog alerts are recorded inline, never via the queue — an alert about a dead queue must not depend on that queue.

Scripted QA runs

Beyond "is it up?", the Watchtower can answer "is it still giving the right answers?". Each agent carries QA scenarios (agent → QA scenarios): a scripted conversation — one or more visitor messages — plus assertions on the final answer: text it must contain, text it must not contain, whether it must ask a follow-up question, and whether product cards are forbidden.

pitchbar:qa-run executes every enabled scenario nightly against the live agent through the full public stack, on a playground-flagged conversation that is deleted afterwards. Failures raise a qa_failed alert carrying the scenario name, the broken rule and an excerpt of the offending answer, and the verdict is shown next to each scenario on its page. This turns a client's manual test script into a permanent regression suite: a knowledge edit or model change that silently breaks an answer is caught the next night, not the next time a human happens to re-test.

The widget reports its own deaths

Every way a chat turn can die client-side now phones home: stream timeouts, HTTP failures on the stream endpoint, network failures and — the subtlest one — a stream that closes without a completion event (the signature of a server-side crash mid-answer). These arrive as client_stalled entries on the Widget Monitor with the precise reason.

Crucially, an expired widget session can still deliver telemetry: the events endpoint accepts a genuinely-signed token up to seven days past expiry (signature validation unchanged — forgeries are rejected). A dying widget is exactly the one whose distress call must not be dropped at the door.

Running the scheduler

All of this rides the Laravel scheduler. Self-hosted installs must run it — either a system cron calling php artisan schedule:run every minute, or a supervised php artisan schedule:work process (the shipped ecosystem.config.cjs includes a pitchbar-scheduler PM2 app for exactly this; activate it with pm2 start ecosystem.config.cjs --only pitchbar-scheduler). Without a scheduler, none of the recurring tasks — monitors, re-crawl sweeps, digests — ever run.