Skip to main content
Pioneer has two related diagnostic paths. Local tracing and structured logs help an operator inspect a gateway. Optional Sentry and OpenTelemetry pipelines provide runtime and performance signals for configured deployments. Neither path is the source of truth for threads, tasks, or authorization. Durable state stays in gateway.db and the related stores.

​
Shared observability crate

crates/observability is shared by gateway, desktop, and mobile runtime binaries. It provides:
  • tracing setup and Sentry initialization;
  • consent-gated event capture;
  • OTLP/HTTP metrics and traces;
  • startup stage and failure measurements;
  • gateway operation and provider-readiness traces;
  • native lifecycle and readiness metrics;
  • markdown, timeline, and patch performance metrics;
  • explicit flush and shutdown behavior for short-lived clients.
The telemetry target is part of the resource metadata. Current targets are pioneer-gateway, pioneer-desktop, and pioneer-mobile. Mobile reports through the shared client FFI boundary and supplies the application version explicitly. Telemetry export is behind a runtime consent gate. Sentry events are dropped when the gate is closed. OTLP metrics and traces use the same gate, including signals already buffered by an SDK during an opt-out transition. Changing consent does not block a settings request on exporter shutdown. Disabling closes the gate immediately. Re-enabling drains old buffers before opening the gate again, so data recorded before the new opt-in is not exported as if it were new consented data. Flushes for mobile and desktop startup are scheduled away from the UI thread. The gateway configuration controls whether the pipeline is enabled, its metrics and traces endpoints, the export interval, and the export timeout. The default app configuration points to the Pioneer OTLP endpoints and enables the gateway telemetry setting. Operators can override the setting and endpoint values in their gateway configuration. The effective consent state remains the final gate.

​
Signals and labels

Gateway operation traces describe bounded stages such as request admission, provider readiness, task execution, recovery, and delivery. Performance metrics cover provider and gateway markdown processing, desktop timeline stages, code highlighting, patch operations, and database admission. Labels are kept at operational cardinality. They identify stable operation, provider, runtime, stage, outcome, or target categories. They should not contain message text, arbitrary paths, raw URLs, tokens, user prompts, or unbounded ids. Startup is measured as a stage report rather than one opaque duration. Gateway stages include configuration and secret readiness, database setup, migrations or startup repair, provider readiness, task recovery, and service readiness. Desktop and mobile record connection, authentication, bootstrap, projection, and first usable state through their respective startup reports.

​
Provider readiness

Provider readiness is supervised by the gateway. A provider can be configured but not ready because its credentials, endpoint, model catalog, or network path is unavailable. The readiness supervisor exposes a state and diagnostics that clients can use without treating a transient probe failure as a missing configuration. CLI runtime readiness follows the same principle but has runtime-specific states such as missing binary, needs auth, unsupported version, degraded, or spawn failure. See CLI runtime architecture.

​
Native lifecycle readiness

The gateway exposes three small operational endpoints: The report includes status, acceptingNewTurns, a report generation, checkedAtUnix, and component snapshots. Components cover the database, native agent manager, durable listeners, recovery coordinator, terminalization, and provider registry. Each snapshot carries its component state, generation, active count, pending count, optional oldest pending age, and an optional reason code. Critical component failures close new-turn admission. Provider-registry degradation is reported in the snapshot and changes the overall status to degraded, but it does not by itself close admission. Probes have a two-second bound. The report treats running turns older than five minutes, recovery backlog over 100 or older than five minutes, and terminal-effect backlog over 64 or older than five minutes as degradation signals. An unresolved terminal effect makes terminalization unhealthy. Native lifecycle metrics are consent-gated and use bounded enum labels: These signals describe bounded operational state. They do not replace thread/timeline/page, turn/work/page, task APIs, settings snapshots, or authorization capabilities when a client needs product state.

​
Database maintenance and repair

Startup is not the only maintenance point. The gateway runs cooperative database maintenance and post-startup work so repair does not monopolize the SQLite writer. The runtime includes:
  • fair SQLite write admission;
  • bounded database operation metrics;
  • incremental read-model repair;
  • projection stream state backfill;
  • patch-history projection repair;
  • execution-authority and authorization integrity checks;
  • compression and legacy state cleanup where needed.
Repair is designed to preserve the durable source rows and rebuild projections from them. A repaired projection is not a new user action. Clients may receive a changed notification after repair and should reload through the normal read method. The database uses a single SQLite connection by default. Write admission and maintenance limits keep long repair work from starving interactive turns. This is an operational property of the gateway, not a guarantee that every provider or filesystem operation completes within one request.

​
Failure reporting

Observability separates a failure from its recovery state. A provider timeout, a lost runtime event stream, a blocked permission request, a stale task lease, and a missing patch-history range have different user actions. Gateway code records the typed outcome and publishes a durable or live notification when the product surface needs it. For a reconnecting client, notifications are hints. Rebuild state from thread/timeline/page, turn/work/page, task/events, settings/get, capability snapshots, or the relevant domain read method. Do not use telemetry as a replacement for those APIs.

​
Code map