Pioneer persists normal gateway state inDocumentation Index
Fetch the complete documentation index at: https://docs.getpioneer.dev/llms.txt
Use this file to discover all available pages before exploring further.
gateway.db through SeaORM. Secret values are stored separately in keystore.db. The persistence layer is split into entity definitions, migrations, SQLite helpers, the CrudStore repository/projector facade, and a keystore boundary used by gateway and desktop secret flows.
Persistence is not just storage for finished conversations. It is part of the runtime contract. The gateway uses it to make active turns visible, recover from interrupted work, audit which prompt and skills were used, track MCP catalogs, reconcile tasks, and keep clients in sync after reconnect.
Why This Layer Exists
Pioneer has many pieces of state that change while work is still running. A simple “save final answer” table would not be enough. The UI needs timeline items as they start and complete. Recovery needs retained LLM context. Task scheduling needs queued and retryable runs. MCP needs catalog snapshots. Skills need audit records and dependency snapshots. The persistence layer gives these flows a consistent place to append durable events, project read models, and repair deterministic state after restart.Crates
| Crate | Role |
|---|---|
crates/entity | SeaORM entity modules for every table. |
crates/migration | Schema migrations. |
crates/sqlite | SQLite-specific write coordination and helpers. |
crates/crud | Repository methods, typed records, turn projector, task projector, and materialization helpers. |
crates/keystore | db-keystore facade, stable secret ids, metadata, permission hardening, and test memory store. |
CrudStore.
That boundary keeps schema details from leaking upward. If a handler wants thread history, it asks CrudStore for thread history; it should not know which joins or tables produce that view.
CrudStore
CrudStore holds a DatabaseConnection, TurnProjector, TaskProjector, and SqliteWriteCoordinator. It exposes typed methods for threads, turns, items, skills, MCP, tasks, recovery jobs, attachment upload registry, prompt manifests, and retained LLM context.
The write coordinator is important because the gateway is async and can have concurrent request handlers, background workers, MCP runtime tasks, and task scheduler work touching SQLite.
Turn Persistence
Turns use durable event payloads and projected read models.TurnEventPayload includes turn start, item start, item completion, item updates, timeout/recovery events, retry events, tool-loop budget exhaustion, turn completion, and turn failure.
TurnProjector writes or updates:
- threads
- sandbox policy
- turns
- turn input
- turn status history
- turn items
- turn item attempts
Task Persistence
Tasks have their own event stream.AppendedTaskEvent includes event id, task id, optional run/thread/turn ids, sequence, event type, payload, workspace id, root task id, parent task id, and timestamp.
The task projector maintains task read models: tasks, triggers, runs, dependencies, agent specs, deliveries, delivery attempts, write locks, and task trees.
Important Tables By Domain
| Domain | Stored data |
|---|---|
| Workspaces and threads | workspace rows, thread rows, folder placement, thread lineage, sandbox policy, summaries |
| Turns | turn rows, turn input, turn items, turn item attempts, status history, timeline events |
| Prompt/context | prompt manifests, retained turn_llm_context rows during active recovery windows |
| Providers/attachments | attachment upload registry for provider file reuse |
| Secrets | no raw values; domain rows store refs or redacted data only |
| Skills | installations, workspace policy, turn bindings, audit events, dependency snapshots, upload sessions |
| MCP | server installations, catalog snapshots, audit events, turn MCP bindings |
| Tasks | tasks, triggers, runs, events, dependencies, agent specs, deliveries, write locks |
| Recovery | recovery jobs and provider failure metadata |
Read-Model Repair
The gateway can detect and repair deterministic read-model invariant violations on startup. Examples include terminal tool payloads still marked in progress, terminal turns with running attempts, or terminal task/run rows missing completion timestamps. This repair path is not a substitute for correct projectors. It exists because a gateway can stop between durable event append and all downstream cleanup, especially while the project is evolving.Secret Storage Boundary
gateway.db and keystore.db have different responsibilities.
gateway.db stores domain state: threads, turns, tasks, skills, provider attachment metadata, MCP installations, MCP catalog snapshots, audit records, and secret refs.
keystore.db stores raw secret values: provider API keys, MCP env/header secret values, superuser JWT signing material, and desktop gateway bearer tokens. It is created in the runtime home next to gateway.db for gateway use. The desktop app also uses the same keystore crate for saved gateway bearer tokens.
Current keystore storage is unencrypted. Pioneer hardens runtime directory and SQLite file permissions, but any OS user or process that can read the runtime home can read the keystore file. See Secret Storage for operational details.
Persistence Guidelines
Add new user-visible state through protocol types first, then storage records, then projector/repository methods. For evented flows, persist the event and project the read model in one coordinated path. For runtime-only state, prefer explicit cleanup and recovery behavior over hidden global maps. Do not store raw secrets in ordinary domain tables orgateway-settings.toml. Provider keys, MCP secrets, desktop gateway bearer tokens, and JWT signing material belong in the keystore. Database records should contain redacted config or secret refs.
Related Pages
- Protocol Layer describes the public types that often drive storage changes.
- Gateway explains request handling and durable event persistence.
- Prompt And Context explains prompt manifests and retained LLM context.
- MCP Architecture, Skills Architecture, and Tasks And Subagents describe the domain-specific records stored here.