> ## Documentation Index > Fetch the complete documentation index at: https://docs.getpioneer.dev/llms.txt > Use this file to discover all available pages before exploring further. # Observability and runtime maintenance > How Pioneer records startup, operation, performance, and recovery diagnostics without making telemetry part of product state. Pioneer has two related diagnostic paths. Local tracing and structured logs help an operator inspect a gateway. Optional Sentry and OpenTelemetry pipelines provide runtime and performance signals for configured deployments. Neither path is the source of truth for threads, tasks, or authorization. Durable state stays in `gateway.db` and the related stores. ## Shared observability crate `crates/observability` is shared by gateway, desktop, and mobile runtime binaries. It provides: * tracing setup and Sentry initialization; * consent-gated event capture; * OTLP/HTTP metrics and traces; * startup stage and failure measurements; * gateway operation and provider-readiness traces; * native lifecycle and readiness metrics; * markdown, timeline, and patch performance metrics; * explicit flush and shutdown behavior for short-lived clients. The telemetry target is part of the resource metadata. Current targets are `pioneer-gateway`, `pioneer-desktop`, and `pioneer-mobile`. Mobile reports through the shared client FFI boundary and supplies the application version explicitly. ## Consent gate Telemetry export is behind a runtime consent gate. Sentry events are dropped when the gate is closed. OTLP metrics and traces use the same gate, including signals already buffered by an SDK during an opt-out transition. Changing consent does not block a settings request on exporter shutdown. Disabling closes the gate immediately. Re-enabling drains old buffers before opening the gate again, so data recorded before the new opt-in is not exported as if it were new consented data. Flushes for mobile and desktop startup are scheduled away from the UI thread. The gateway configuration controls whether the pipeline is enabled, its metrics and traces endpoints, the export interval, and the export timeout. The default app configuration points to the Pioneer OTLP endpoints and enables the gateway telemetry setting. Operators can override the setting and endpoint values in their gateway configuration. The effective consent state remains the final gate. ## Signals and labels Gateway operation traces describe bounded stages such as request admission, provider readiness, task execution, recovery, and delivery. Performance metrics cover provider and gateway markdown processing, desktop timeline stages, code highlighting, patch operations, and database admission. Labels are kept at operational cardinality. They identify stable operation, provider, runtime, stage, outcome, or target categories. They should not contain message text, arbitrary paths, raw URLs, tokens, user prompts, or unbounded ids. Startup is measured as a stage report rather than one opaque duration. Gateway stages include configuration and secret readiness, database setup, migrations or startup repair, provider readiness, task recovery, and service readiness. Desktop and mobile record connection, authentication, bootstrap, projection, and first usable state through their respective startup reports. ## Provider readiness Provider readiness is supervised by the gateway. A provider can be configured but not ready because its credentials, endpoint, model catalog, or network path is unavailable. The readiness supervisor exposes a state and diagnostics that clients can use without treating a transient probe failure as a missing configuration. CLI runtime readiness follows the same principle but has runtime-specific states such as missing binary, needs auth, unsupported version, degraded, or spawn failure. See [CLI runtime architecture](https://docs.getpioneer.dev/architecture/cli-runtime.md). ## Native lifecycle readiness The gateway exposes three small operational endpoints: | Endpoint | Contract | | - | - | | `/health` | Liveness check. Returns `200 OK` with `ok` and does not inspect the native runtime. | | `/ready` | Admission check. Returns `200` when the gateway accepts new turns and `503` when it does not. The body is a `GatewayNativeLifecycleReadinessReport`. | | `/health/native` | Diagnostic readiness report. Returns `200` with the same report even when the gateway is degraded or not accepting new turns. | The report includes `status`, `acceptingNewTurns`, a report `generation`, `checkedAtUnix`, and component snapshots. Components cover the database, native agent manager, durable listeners, recovery coordinator, terminalization, and provider registry. Each snapshot carries its component state, generation, active count, pending count, optional oldest pending age, and an optional reason code. Critical component failures close new-turn admission. Provider-registry degradation is reported in the snapshot and changes the overall status to `degraded`, but it does not by itself close admission. Probes have a two-second bound. The report treats running turns older than five minutes, recovery backlog over 100 or older than five minutes, and terminal-effect backlog over 64 or older than five minutes as degradation signals. An unresolved terminal effect makes terminalization unhealthy. Native lifecycle metrics are consent-gated and use bounded enum labels: | Signal | Labels or values | | - | - | | `pioneer.gateway.native.lifecycle.events` | Counter with `native.stage`, `outcome`, and `provider.class`. Stages include `turn`, `first_event`, `durable_commit`, `provider_round`, `tool_attempt`, `tool_retry`, `recovery`, `terminal_commit`, `terminalization`, and `readiness`. Outcomes include `started`, `succeeded`, `failed`, `blocked`, `interrupted`, `timed_out`, `rejected`, `exhausted`, `saturated`, `closed`, and `recovered`. Provider classes are `api`, `cli`, and `unknown`. | | `pioneer.gateway.native.lifecycle.duration` | Millisecond histogram with the same labels. | | `pioneer.gateway.native.readiness.checks` | Counter with `native.component` and `native.state`. Components are the readiness components above; states are `starting`, `healthy`, `degraded`, and `unhealthy`. | | `pioneer.gateway.native.lifecycle.depth` | Observable gauge with `native.depth.kind`: `active_turns`, `stale_running_turns`, `recovery_backlog`, `terminal_backlog`, and `unresolved_terminal_effects`. | These signals describe bounded operational state. They do not replace `thread/timeline/page`, `turn/work/page`, task APIs, settings snapshots, or authorization capabilities when a client needs product state. ## Database maintenance and repair Startup is not the only maintenance point. The gateway runs cooperative database maintenance and post-startup work so repair does not monopolize the SQLite writer. The runtime includes: * fair SQLite write admission; * bounded database operation metrics; * incremental read-model repair; * projection stream state backfill; * patch-history projection repair; * execution-authority and authorization integrity checks; * compression and legacy state cleanup where needed. Repair is designed to preserve the durable source rows and rebuild projections from them. A repaired projection is not a new user action. Clients may receive a changed notification after repair and should reload through the normal read method. The database uses a single SQLite connection by default. Write admission and maintenance limits keep long repair work from starving interactive turns. This is an operational property of the gateway, not a guarantee that every provider or filesystem operation completes within one request. ## Failure reporting Observability separates a failure from its recovery state. A provider timeout, a lost runtime event stream, a blocked permission request, a stale task lease, and a missing patch-history range have different user actions. Gateway code records the typed outcome and publishes a durable or live notification when the product surface needs it. For a reconnecting client, notifications are hints. Rebuild state from `thread/timeline/page`, `turn/work/page`, `task/events`, `settings/get`, capability snapshots, or the relevant domain read method. Do not use telemetry as a replacement for those APIs. ## Code map | Area | Code | | - | - | | Shared initialization and consent | `crates/observability/src/lib.rs` | | OTLP exporters and flush | `crates/observability/src/telemetry.rs` | | Startup reports | `crates/observability/src/startup.rs` | | Operation traces | `crates/observability/src/operations.rs` | | Performance metrics | `crates/observability/src/performance.rs` | | Patch metrics | `crates/observability/src/patch_telemetry.rs` | | Native lifecycle metrics | `crates/observability/src/metrics.rs` | | Native readiness probe | `crates/gateway/src/message/native_health.rs` | | Native terminal-effect outbox | `crates/crud/src/repositories/native_terminal_effect_outbox.rs`, `crates/entity/src/native_terminal_effect_outbox.rs` | | Gateway maintenance | `crates/gateway/src/database/maintenance`, `post_startup.rs`, `bootstrap.rs` | | Gateway wiring | `crates/gateway/src/epic5_observability.rs`, `operations.rs` | ## Related pages * [Configuration reference](https://docs.getpioneer.dev/configuration/reference.md) lists telemetry and resilience settings. * [Gateway architecture](https://docs.getpioneer.dev/architecture/gateway.md) explains startup, dispatch, and notification flow. * [Persistence layer](https://docs.getpioneer.dev/architecture/persistence.md) explains durable rows and projections. * [Patch history](https://docs.getpioneer.dev/architecture/patch-history.md) explains mutation coverage and patch metrics. * [CLI runtime architecture](https://docs.getpioneer.dev/architecture/cli-runtime.md) explains runtime diagnostics and recovery. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.