Widget Monitor: errors & provider failover

The Widget Monitor at /settings/system/widget-monitor (super-admin only) is the error counterpart to the hot-path latency dashboard. Latency answers “why is the bot slow?”; the Widget Monitor answers “what's breaking?” — stream failures, LLM-provider outages, and visitor-reported freezes, all in one feed you can triage and mark resolved as you fix each one.

What a failed canary records

When the synthetic canary can't reach an agent, the entry keeps what the failing response actually said — the status, a bounded excerpt of the body, and the headers that identify which process answered (x-request-id, x-powered-by). An empty body on a 500 is itself the finding: it means the worker died before the application could render an error page, which is what a crashed or recycling process looks like from the outside. Without this the entry read only "init: HTTP 500", and every investigation started from a guess.

What it tracks

Each row is one failure or anomaly on the visitor path:

• Provider down — every configured LLM provider failed for a turn. This is the one to act on first: a real outage your visitors felt.
• Stream failed — an unhandled exception killed a reply mid-flight. Open it to read the exception class and trace the cause. The event is stamped with the provider that was actually serving the turn (e.g. cloudflare, cloudflare-fallback, openai), so you see the culprit on the failure itself instead of cross-referencing a separate Provider down row. The same provider tag is written to the widget.llm_failed / widget.stream_unhandled log lines and the conversation debugger's error trace.
• Failover — the primary provider stumbled but a backup picked the turn up. Informational: the visitor saw nothing wrong, but a rising count means the primary is unhealthy.
• Client freeze — the visitor's browser saw the stream stall (no data for 35s, or the 120s ceiling) and gave up. Reported by the widget itself, so you catch freezes even when the server looks fine.
• Tool-loop timeout / Retrieval failed — slower-path failures in tool calling or knowledge retrieval.

Filter by time window (24h / 7d / 30d), type, severity, and open-vs-resolved. The banner at the top gives a plain-English verdict (“provider outage in progress…”, “healthy with hiccups…”, “all clear…”) so you can read the state at a glance. Click Resolve on a row once you've dealt with it — or Reopen if it comes back.

Provider failover — why a single outage no longer kills every chat

The reliability backbone behind the monitor is runtime provider failover. Pitchbar can be configured with more than one LLM provider (Cloudflare Workers AI, OpenAI, OpenRouter). When two or more are configured, every visitor turn runs through a failover chain: if the primary provider is slow, returns a 5xx, is rate-limited (429), or has run out of credits, Pitchbar transparently retries the next provider — same turn, before the visitor sees an error.

• Retryable failures (timeouts, 429s, 5xx, connection errors) trigger failover — the provider is the problem, so another may succeed.
• Client errors (a 4xx “bad request”) are not retried — every provider would reject them identically, so the error surfaces immediately instead of doubling the latency.
• Streaming only fails over before the first token. Once the visitor has seen text, a mid-stream drop can't be silently restarted on another provider — it surfaces as a stream failure.

Fail-fast timeouts

Blocking provider calls (tool decisions, embeddings) are capped well below the streaming budget so a hung provider is abandoned quickly and failover reaches the next one fast, rather than waiting out a full minute. Streaming keeps a generous budget because a legitimate long answer holds the connection open; a 5-second connect cap still detects a dead host immediately.

Tuning the freeze detector

“Client freeze” events come from the widget's own stall detector. If you see them clustered, cross-reference the hot-path latency monitor — a freeze is usually a turn that genuinely took longer than the widget's patience (35s without a single byte), which points at provider responsiveness or an overloaded tool loop rather than a hard crash.

What closes itself, and what does not

An alert you cannot trust is worse than no alert, because a real incident arrives already buried. Two rules keep the Monitor honest:

• One incident, one alert. While an event is still open for the same agent and the same symptom, further failures do not raise a second one. A canary failing every five minutes for half an hour is one notification, not six. A different symptom — the stream breaking while init recovers, or a second QA case going red — is a different incident and still alerts.
• Recovery closes the incident. When the canary passes again, or a failing QA case goes green, its open events are resolved automatically with the reason auto:recovery. This happens silently: there is no "all better" notification, because proving quiet by sending more traffic defeats the purpose.

So an empty Monitor means healthy, not unwatched. Anything still open is either live or was never resolved by the recovery path — worth a look either way.

Clearing an old backlog

Installs that ran before automatic resolution existed can carry hundreds of stale open events. This closes the ones old enough to be history, and leaves recent ones alone because they may still be live:

php artisan pitchbar:resolve-stale-events --dry-run   # show what would close
php artisan pitchbar:resolve-stale-events             # close events older than 2 days
php artisan pitchbar:resolve-stale-events --days=7    # be more conservative

Events closed this way carry the reason auto:backlog-cleanup, so they stay distinguishable from incidents that genuinely recovered. An automatic close records the reason only — who resolved an event stays empty unless a person clicked it, and a person acting on an event clears the automatic reason.

What provider_failover means

One of these means your primary AI provider genuinely failed a request and the next provider in the chain answered it instead — the visitor still got a reply, but the primary is worth watching.

It does not fire for optional features that set their own short deadline. The answer-chip generator and the query rewriter each give the model a couple of seconds and quietly skip themselves if that runs out; missing that budget says nothing about the provider's health, so it is not reported as a failover. Before this distinction existed, those two produced hundreds of false outage events and buried the real ones.