MCP

Pioneer implements MCP as a gateway-owned client runtime. The desktop app configures MCP servers, but the gateway host starts stdio servers, connects to HTTP servers, stores catalogs, resolves secrets, and calls MCP tools during agent turns. MCP connects Pioneer to external tool systems. Pioneer does not load MCP servers as trusted in-process plugins. It manages them as external runtimes with transport, policy, status, catalogs, secrets, and audit state.

MCP server commands and HTTP reachability are evaluated from the gateway host. With a remote gateway, install commands and test network access on that host.

Why this layer exists

An MCP server has startup behavior, authentication, a changing catalog, errors, retry state, and secrets. Pioneer needs a runtime service to manage that state. pioneer-mcp owns protocol-facing MCP concerns such as config parsing, transports, clients, catalogs, and runtime sessions. McpService in the gateway owns gateway-facing concerns: database rows, live task supervision, notifications, tool materialization, and calls from the agent’s tool runtime.

Components

Component	Code	Responsibility
Config parser	`crates/mcp/src/config.rs`	Parses `mcpServers` JSON, validates names/transports, extracts secrets, creates install plans.
Domain model	`crates/mcp/src/domain.rs`	Installation, transport, scope, source, policy, availability, runtime state, and secret ref types.
Runtime connector	`crates/mcp/src/runtime/mod.rs`	Connects to MCP servers and exposes `McpRuntimeSession`.
Clients	`crates/mcp/src/client/*`	`rmcp`-based stdio and streamable HTTP clients.
Gateway service	`crates/gateway/src/mcp_service.rs`	Starts/stops/restarts server tasks, tracks snapshots, stores catalogs, publishes status, materializes tools.
Gateway handlers	`crates/gateway/src/message/mcp/*`	JSON-RPC methods for install, list, policy, restart, uninstall, and details.

Install config

Pioneer accepts the familiar top-level shape:

{
  "mcpServers": {
    "server_name": {
      "command": "some-command",
      "args": ["--flag"],
      "env": {
        "API_KEY": "secret-value"
      }
    }
  }
}

or:

{
  "mcpServers": {
    "server_name": {
      "url": "https://example.com/mcp",
      "headers": {
        "Authorization": "Bearer secret-value"
      }
    }
  }
}

Each server must define exactly one transport: command for stdio or url for streamable HTTP. Server names must match ^[A-Za-z0-9_-]{1,64}$. Common optional fields include required, startup_timeout_sec, tool_timeout_sec, and timeout. Policy flags such as enabled state and implicit invocation are stored as installation/policy state, not required in the basic config examples.

Secrets and redaction

Environment variables and HTTP headers can contain secrets. The config parser materializes secret refs, stores secret values through the gateway keystore, and stores redacted transport/source data in gateway.db. Runtime sessions resolve secrets through McpSecretResolver when a server task starts. The installation remains readable and auditable without copying raw secret values into catalog snapshots or list responses. When an MCP install is updated or uninstalled, the gateway deletes stale keystore refs after the database change succeeds. The CLI also exposes pioneer secrets garbage-collection for orphan MCP secret values that need an explicit cleanup pass.

Runtime lifecycle

McpService::reload_workspace loads installed workspace servers from the database and compares them with currently running tasks. Disabled servers are stopped and reported as disabled. Changed fingerprints cause a restart. Removed servers are stopped. For enabled servers, McpService starts a task that connects through RmcpRuntimeConnector, obtains the initial catalog, stores status, watches server events, refreshes catalogs, and accepts tool-call commands over an internal channel. Runtime states include not_started, disabled, starting, ready, degraded, auth_required, failed, stopping, stopped, and restarting.

Catalogs

When a server becomes live, Pioneer stores a catalog snapshot: server info, server instructions hash, tools, resources, resource templates, prompts, catalog version, and generation timestamp. Catalog changes publish mcp/server/catalog_changed and increment the gateway MCP snapshot version. The agent uses the live snapshot and stored catalog to decide which MCP tools can be materialized for a turn.

Policy and tool materialization

MCP policy is per gateway and per workspace. A server can be installed but disabled. A server can also be installed and enabled without allowing implicit tool exposure. At turn time, the agent asks AgentMcpToolProvider for availability and materialized tools. The request includes implicit exposure state plus explicit composer selections from TurnStartParams.capabilities: selected servers and selected raw tools. MCP projection is bounded deliberately. The current defaults allow up to 512 tools, 3 MiB of combined tool schemas, and 16 concurrent MCP calls per turn. The gateway, provider configuration, and CLI bridge validate and apply the same limits so a large catalog cannot bypass the execution budget at a later layer. Invalid limits are rejected before an agent session starts. McpService reloads the workspace, checks installation policy, checks runtime state, loads the catalog snapshot, and builds dynamic tool descriptors. A selected server exposes that server’s allowed catalog tools for the turn. A selected tool exposes only that raw tool. If a server-level selection exists, individual tool selections for the same server are treated as duplicates at the UI/state boundary and the server selection wins. Blocked, missing, disabled, unavailable, unsupported, or catalog-missing selections produce rejected turn capability diagnostics instead of breaking the turn. MCP tools are not represented by prompt text. They materialize into ChatRequest.tools as dynamic extension tools. If the provider does not support tool calling, explicit MCP capabilities are rejected as provider-unsupported. When the model calls a materialized MCP tool, the tool runtime sends the call back to McpService. The service forwards it to the live server task, waits for McpToolCallResult, and records audit/binding data. The model never calls an MCP server directly. The call passes through the Pioneer tool runtime and gateway MCP service, so output policy, diagnostics, audit records, turn bindings, and runtime errors remain under Pioneer control.

Local versus remote gateways

For stdio servers, command, args, cwd, and env are evaluated on the gateway host. For HTTP servers, the gateway host must be able to reach the URL. If the gateway runs remotely, MCP connectivity must be tested from that remote machine, not from the desktop client.

Tools System explains how MCP tools become model-visible dynamic tools.
Gateway explains why MCP processes are gateway-owned.
Skills Architecture explains the parallel extension system for skills.
Persistence Layer explains MCP installations, catalog snapshots, audit events, and turn bindings.
Secret Storage explains where MCP secret values live.

Overview

Core Systems

Subsystems

Contributing

Why this layer exists

Components

Install config

Secrets and redaction

Runtime lifecycle

Catalogs

Policy and tool materialization

Local versus remote gateways

​Why this layer exists

​Components

​Install config

​Secrets and redaction

​Runtime lifecycle

​Catalogs

​Policy and tool materialization

​Local versus remote gateways

​Related pages

Why this layer exists

Components

Install config

Secrets and redaction

Runtime lifecycle

Catalogs

Policy and tool materialization

Local versus remote gateways

Related pages