Skip to main content
Pioneer has one central runtime boundary: the gateway. Clients connect to it, while the gateway owns durable state, runtime state, model access, CLI runtime sessions, tool execution, tasks, MCP, skills, settings, keystore-backed secrets, and history. The desktop app is a native GPUI client that can start a local gateway for a single-machine setup or connect to one or more remote gateways. The mobile app is an Expo/React Native client that connects to remote gateways through the shared Rust client FFI. Other clients can be built against the JSON-RPC protocol without depending on desktop or mobile internals. The architecture is a set of clients attached to a durable assistant runtime. A gateway can run on a laptop, workstation, or remote server. Its host owns the assistant’s state and execution.

​
How to read this section

If you are new to the codebase, start here, then read Gateway, Client Architecture, Agent Loop, Hook Runtime, and Prompt And Context. Those pages explain the central path from a client message to an LLM request and back to persisted timeline events. After that, pick the subsystem you are changing:

Protocol

Read this before changing JSON-RPC methods, notification payloads, schemas, or anything a client consumes.

Clients

Read this before moving behavior between desktop, mobile, shared client core, FFI, or shell-specific code.

CLI Runtime

Read this before changing Codex-style CLI runtimes, native thread bindings, approvals, steering, or runtime recovery.

Tools

Read this before changing shell/file/web/computer-use tools, output projection, or retry behavior.

Permissions

Read this before changing turn permission modes, tool approval prompts, audit events, or task permission caps.

Tasks And Subagents

Read this before changing scheduled work, attached task tools, task trees, or subagent execution.

Agent Memory

Read this before changing durable memory, recall, memvid capsules, semantic writes, or candidate policy.

AGENTS.md

Read this before changing thread-tree instruction files, inheritance, editor behavior, protocol payloads, or prompt injection.

Persistence

Read this before changing tables, projectors, durable events, recovery state, or read models.

Remote Access

Read this before changing gateway tunnel settings, relay keys, rathole supervision, or remote-access status.

​
System shape

​
Runtime boundaries

The gateway is the only process that directly mutates Pioneer state. It accepts protocol requests, validates them, updates the in-memory managers when needed, persists the durable read model through pioneer-crud, supervises CLI runtime and remote-access state, and publishes JSON-RPC notifications to connected sessions. The agent loop runs inside the gateway. A conversation turn is not a client-side action after submission; it becomes gateway-owned work. The gateway loads history, resolves policies, starts or reuses the agent runtime for the thread, streams model/tool progress back to clients, and persists durable events as the turn changes. The protocol crate is the public boundary. It defines request and response payloads, notification payloads, method names, error envelopes, task types, tool result types, provider types, CLI runtime types, MCP types, skill types, prompt manifests, settings snapshots, and schema export helpers. Internal crates can change implementation details without changing the client contract as long as the protocol stays stable. This boundary is what lets Pioneer support many clients without making each client understand how tasks, tools, MCP, skills, CLI runtimes, and provider adapters are implemented. The desktop app should not know how a tool output is projected for recovery. The mobile app should not know how an MCP stdio process is restarted. A custom client should not know how the task scheduler computes retries. They should all see the same protocol events.

​
Why the gateway is the center

Pioneer needs to coordinate things that do not fit cleanly inside a UI process: long-running turns, tool sessions, task schedules, child agent threads, CLI agent runtime sessions, MCP server processes, provider retries, prompt manifests, remote-access status, and recovery state. Putting that work in the gateway gives the system one place to make decisions about state, policy, and durability. This also makes remote usage straightforward. If the gateway runs on a remote workstation, the assistant can use files, tools, MCP servers, provider settings, and task history from that workstation. The desktop app becomes a window into that gateway instead of the place where work happens. The tradeoff is that developers must always ask “which host does this code run on?” Provider calls, tool commands, MCP stdio commands, skill dependency checks, and task execution all happen where the gateway runs. Client-side convenience should not accidentally become client-side authority.

​
Main flows

A client calls turn/start with a thread_id, turn_id, selected mode, model, provider or execution backend, selected permission profile, user input, optional CLI runtime options, and optional turn capabilities for composer-selected skills or MCP. The gateway updates the thread manager, materializes the effective permission profile, persists the turn start and profile-selection audit event, ensures an agent runtime exists for the thread, loads conversation history, loads workspace skill policy, and dispatches the turn into AgentManager::start_turn.
The gateway builds historical chat messages from completed turns. If history is close to the configured context budget, it compresses older turns into a summary with a provider call. The agent runs turn hooks for policy, prompt context, tool materialization, and prompt sections; resolves skills and MCP capabilities; compiles the system prompt with pioneer-promt; materializes skill, MCP, and task tools; builds the current user message; appends retained tool context for recovery when needed; and sends a ChatRequest through the selected provider.
In agent mode, provider tool calls are routed through pioneer-tools. Built-in tools, MCP tools, skill tools, and task orchestration tools are presented through one router. Before side effects run, the tools runtime evaluates the turn permission profile, checks the execution security snapshot, and either allows the action, opens a user approval request, applies scoped grants, or rejects it. Tool and permission events are forwarded back to the gateway as durable item events, audit events, and progress events. Tool outputs are projected differently for the model, timeline, storage, and recovery evidence.
A turn can select a CLI agent runtime backend instead of an API provider. The gateway maps the Pioneer thread to a native runtime thread, starts or resumes the CLI session, forwards runtime events into Pioneer turn items, handles pending approvals and user-input requests, and exposes runtime-specific operations such as model listing, compaction, fork, review start, and steering when supported.
The task runtime stores tasks as durable task events and projected read models. When a task has an agent spec, the gateway task executor creates or restores a hidden child thread, starts a child turn, records thread lineage separately from run execution state, and turns child output into a reviewable result candidate. The parent can accept that candidate or request a revision in the same child thread before the task result is finalized.
Memory hooks classify the turn, recall relevant durable facts, register memory tools when policy allows, render the memory prompt contract, and run post-turn extraction after successful turns. Active recall can also plan thread episodic context searches so old conversation snippets are retrieved through a bounded indexed layer instead of by stuffing full transcripts into every prompt. Preflight or request_tools decides which registered memory tools are visible to the model. The memory service owns scopes, ranking, canonical keys, dedupe, candidate scoring, tombstones, and memvid-backed storage.
Artifacts are gateway-owned files with stable ids, versions, bindings, previews, and download support. Current-turn attachments can be sent as provider attachments. Older attachments are represented as compact refs beside retained history messages or recalled thread-context snippets. If the model needs the file, it requests the artifact tool domain and calls artifact_read for that specific artifact instead of receiving every old file automatically.
MCP servers and skills are installed per gateway and per workspace policy. At turn time, enabled and allowed items can be selected implicitly by policy or explicitly through turn capabilities from the composer. Skills contribute a compact skills prompt and optional dynamic tools; MCP contributes dynamic tools only. Disabled or non-implicit items stay installed but are not automatically exposed to the model.

​
Durability model

Pioneer uses an event-heavy read model rather than a single mutable transcript blob. Turns emit durable events such as item started, item completed, prompt manifest compiled, skill bindings resolved, MCP bindings created, provider failures, recovery events, and terminal turn state. Tasks have their own task event stream and projector. The database stores both source events and projected tables. The UI can load read models quickly while the event data remains available for audit and recovery. Projectors and repository methods are therefore part of the architecture, not incidental persistence code. The practical reason is failure handling. A turn can stream text, call a shell command, receive a partial tool result, hit a provider timeout, create child tasks, or be cancelled. If that state lived only in memory, a gateway restart would lose the evidence needed to explain the failure. Durable events give Pioneer something to reconcile. For details, read Persistence Layer after this overview. For the event producer side, read Agent Loop and Tasks And Subagents.

​
Current scope and boundaries

Some protocol and storage shapes intentionally lead client UI. Workspace creation, switching, and renaming are exposed, while workspace deletion, archival, and cross-workspace thread moves are not part of the current desktop or mobile workflow. Pioneer has turn permission profiles, execution security snapshots, tool approval requests, sandbox/resource policy, backend enforcement status, and durable permission audit events. Tools, MCP, skills, computer use, shell execution, web access, and task/subagent launches use this shared gateway-owned security model. Use separate gateways when work should run on a separate host, account, filesystem, credential set, or network.
  • Crate Map explains where each piece lives in the Rust workspace.
  • Gateway follows request handling and runtime ownership.
  • Client Architecture explains the shared Rust client core, FFI, desktop shell, and mobile shell.
  • CLI Runtime Architecture explains Codex-style runtime sessions, thread/turn bindings, requests, steering, and recovery.
  • Hook Runtime explains how lifecycle phases and typed contributions keep domain logic out of the agent loop.
  • Permission System explains turn permission profiles, security snapshots, sandbox/resource policy, approvals, audit events, and CLI runtime mapping.
  • Prompt And Context explains how history, prompt sections, skills, memory, retries, and runtime facts become model input.
  • AGENTS.md Architecture explains thread-tree instruction files, inheritance, and hook-based prompt injection.
  • Tools System, MCP Architecture, and Skills Architecture explain the three main ways the model gets capabilities beyond plain chat.
  • Remote Access Architecture explains the gateway-owned tunnel supervisor and relay status model.