> ## Documentation Index > Fetch the complete documentation index at: https://docs.getpioneer.dev/llms.txt > Use this file to discover all available pages before exploring further. # Prompt And Context > How Pioneer builds conversation history, compiles system prompts, and sends model input. Prompt construction is split across two layers: The gateway decides which previous conversation messages should be included. The agent compiles the system prompt and current turn runtime context. The provider adapter then maps the common `ChatRequest` into the provider-specific API shape. This split is deliberate. Conversation history is durable thread state, so the gateway owns loading, summarizing, and truncating it. Runtime instructions are turn-scoped, so the agent owns skills, memory prompt context, permission guidance, retry instructions, task orchestration policy, and runtime facts. Provider adapters should only translate the already-built request into API-specific payloads. If you are debugging "why did the model say that?", start here, then inspect the turn's `PromptManifest`, the thread history, and the provider request shape described in [Provider System](/architecture/providers). ## Why prompt compilation is a separate crate Prompt behavior changes the product more than most internal refactors. Small wording changes can affect tool use, recovery, task delegation, and user trust. Keeping prompt compilation in `pioneer-promt` makes it testable and reviewable as its own layer instead of hiding it in the middle of the agent loop. The compiler also gives Pioneer stable/dynamic prompt separation. Stable sections can be fingerprinted and potentially cached by providers. Dynamic sections can change between rounds when skills, retry instructions, continuation hints, or runtime facts change. ## Conversation history Historical context is loaded by `MessageProcessor::load_conversation_history` in `crates/gateway/src/message/provider_handlers.rs`. The gateway loads up to 200 completed conversation entries from `CrudStore::get_thread_conversation_history`. It also loads the thread summary if one exists. Token counting uses `tiktoken-rs` with `cl100k_base` as a provider-agnostic approximation. The budget is derived from gateway thread config: ```text theme={null} history_budget = max_context_tokens - response_reserve_tokens ``` If the estimated history is under 80% of that budget, Pioneer sends the existing summary plus all loaded entries. If it reaches 80%, the gateway asks the configured summary model, or the thread model if no summary model is configured, to compress the conversation into roughly 10% of the history budget. If compression succeeds, the model receives one system message: ```text theme={null} Summary of earlier conversation: ... ``` If compression fails, the gateway falls back to truncating recent turns to fit the budget. The compression policy optimizes for continuity over raw transcript length. Once a thread becomes too large, sending every old message is less useful than preserving decisions, file paths, user corrections, and unresolved work in a compact summary. The summary is stored on the thread and reused on later turns. When a retained history message had artifacts attached, Pioneer can append a compact artifact-reference block to that same provider message. The block lists metadata only, such as artifact id, version id, display name, kind, MIME type, size, and role. It does not include file bytes and it does not create a global "all artifacts in this thread" section. That placement is intentional. If the old message says "what car is this?" and has `car.jpg`, the reference belongs right next to that message. If the same thread also has 100 unrelated screenshots, those screenshots should not appear in the prompt unless their own messages are included or recalled. ## Prompt compiler The system prompt compiler lives in `crates/promt`. The crate name is intentionally `promt` in the current workspace. `compile_prompt` receives `PromptCompileInput`: | Input | Meaning | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `workspace_root` | Bootstrap root where canonical prompt files are read. In the main agent flow this is the gateway runtime home from `home_directory`. | | `profile` | Prompt profile: full, minimal, or none. Agent turns use `AssistantFull`. | | `skills_prompt` | Dynamic skills section built after skill resolution. | | `retry_instruction` | Dynamic instruction inserted after recoverable tool failures or loop budget exhaustion. | | `include_tool_recovery_policy` | Adds stable tool failure recovery instructions. | | `include_task_orchestration_policy` | Adds task/subagent orchestration guidance when task tools are available. | | `continue_generation_hint` | Adds continuation guidance during recovery. | | `dynamic_context` | Dynamic context contributed by hook-driven domains and runtime systems. | | `extra_system` | Runtime facts such as local date/time and OS. | | `limits` | Character budgets for bootstrap files. | ## Prompt sections The compiler builds ordered sections: | Section | Stability | Source | | --------------------- | --------- | --------------------------------------------------------------------------------------------------------- | | Identity | Stable | Built-in identity base. | | Safety | Stable | Built-in safety lines. | | Soul Core | Stable | `SOUL.md` from runtime home when present. | | Identity Core | Stable | `IDENTITY.md` from runtime home when present. | | User Persona | Stable | `USER.md` if present. | | Tool Recovery Policy | Stable | Built-in policy when enabled. | | Current Permissions | Dynamic | Turn permission guidance rendered when the effective profile is restricted or narrowed. | | Task Orchestration | Dynamic | Built-in policy when task tools are materialized. | | Memory Recall | Dynamic | Memory prompt contract rendered from hook prompt context when memory policy allows it. | | Artifact References | Dynamic | Short policy section rendered only when the current compiled prompt contains artifact refs. | | Hidden Tool Domains | Dynamic | Compact `request_tools` catalog listing hidden domain names and exact tools when tool calling is enabled. | | AGENTS.md | Dynamic | Effective thread-tree AGENTS.md rendered by the AGENTS.md prompt hook when an active file exists. | | Recovery Continuation | Dynamic | Built-in continuation hint when requested. | | Skills Runtime | Dynamic | Active skills prompt for the turn. | | Retry Instruction | Dynamic | Tool retry or final-answer instruction. | | Dynamic Context | Dynamic | Optional caller-provided context. | | Extra System | Dynamic | Current runtime date/time and OS. | Stable and dynamic sections are rendered separately. The final prompt is: ```text theme={null} stable_system_text PROMT_CACHE_BOUNDARY dynamic_system_text ``` The provider receives both the split form and `full_system_text` through `CompiledPromptPayload`. Adapters can use this to support provider-side prompt caching where available. ## Bootstrap files For agent turns, bootstrap files are read from the gateway runtime home. The runtime home is the OS user home joined with `home_directory`, for example `~/.pioneer`. On gateway startup, Pioneer ensures the runtime home contains: * `SOUL.md` * `IDENTITY.md` If either file is missing, the gateway creates it with the current seed content from `pioneer-promt`. If a file already exists, Pioneer leaves it untouched, even if it has been customized or emptied. Compile-time defaults are not used anymore. The compiler reads canonical files in this order: 1. `SOUL.md` 2. `IDENTITY.md` 3. `USER.md` `SOUL.md` and `IDENTITY.md` are model-evolvable identity files. When they are inserted into the prompt, Pioneer appends this reminder after each file block: ```text theme={null} This file is yours to evolve. As you learn who you are, update it. ``` That reminder is rendered into the prompt; it is not written into the files. The file path is already shown in the block heading. `USER.md` is optional and is not created automatically. If present in runtime home, it becomes the User Persona section. Each file is budgeted with a maximum per-file character limit and a total character limit. Missing files, read errors, file truncation, and total-budget truncation become prompt diagnostics. Missing `SOUL.md` or `IDENTITY.md` files do not fall back to Rust constants during compilation. The resulting prompt manifest is persisted on the turn so developers can audit what sections were compiled without storing the entire prompt text as the primary contract. ## Skills prompt Skills are resolved before prompt compilation. `pioneer-skills` builds a compact `[Skills]` section listing active skills, their exact `read_skill` slug, and their descriptions. Composer-selected skills stay compact. They do not expand the full `SKILL.md` body into the prompt; the model must call `read_skill` with the exact slug before following specialized instructions. Full skill bodies are only eligible for path-matched skills when few skills are active and the prompt budget allows it. If the prompt budget is tight, the section includes a `read_skill` hint so the model can load exact skill instructions on demand. ## Memory prompt Memory prompt content is contributed through hooks, not hard-coded in the compiler. In agent mode, memory hooks can: * classify the turn memory policy; * recall deterministic and active memory context; * register memory tools when policy allows them; visibility is selected separately by preflight or `request_tools`; * render the `Memory Recall` section before prompt compilation. The rendered memory section tells the model how to use memory proactively, when to search, when to write, when to forget, and what not to store. Recalled facts are marked as context rather than instructions. Current user instructions override recalled memories. The post-turn extractor uses a separate prompt renderer. It is not the main assistant prompt and does not include model-facing memory tools. Its job is to return strict JSON facts for the memory service after a completed turn. ## Thread context prompt Thread episodic recall is separate from ordinary retained history. Retained history is loaded directly from recent completed conversation entries. Thread context is searched from indexed conversation snippets and injected only when recall decides a snippet is useful. When thread context appears in the prompt, it is rendered as compact context, not as a replacement for conversation history. A recalled snippet includes provenance such as thread, turn, item, chunk, source, and boundary. If that snippet came from a message with artifacts, Pioneer renders a scoped artifact block for that snippet, for example "available artifacts for this recalled thread context item." The model can then tell which artifacts belong to which recalled fragment. The prompt should never contain a loose pile of artifact refs without the message or snippet they came from. Without that source text, the model cannot know why a file matters. ## Artifact reference prompt Artifact refs in prompt context are an invitation to inspect a file, not the file itself. When compiled prompt content includes one or more artifact refs, Pioneer adds a dynamic artifact-reference policy section. That section tells the model that: * historical artifact refs are metadata only; * if it needs file content, it should request the hidden `artifact` tool domain; * after the domain is visible, it should call `artifact_read` for the specific artifact refs it needs; * it should not ask for every artifact just because refs exist; * it should answer from conversation history when the file itself is unnecessary. The section is omitted when there are no artifact refs in the compiled prompt. A model should not be told to use `artifact_read` when there is nothing available to read. ## AGENTS.md prompt AGENTS.md prompt content is also contributed through hooks. The gateway installs an internal hook package that resolves the effective active `AGENTS.md` for the current thread at `turn.pre_prompt_compile`. If no active file exists in the thread's folder ancestry, the hook contributes nothing. If a file exists, it becomes a dynamic `AGENTS.md` prompt section with hook source metadata in the prompt manifest. See [AGENTS.md Architecture](/architecture/agents-md) for the full storage, inheritance, and protocol model. ## Tool definitions are separate Tool definitions are not embedded as prompt prose. The agent passes model-visible tool definitions in `ChatRequest.tools`. The prompt can explain recovery or task orchestration policy, but the callable JSON schema for tools goes through the provider tool-calling interface. This separation prevents two common bugs. A tool does not become callable because its name appears in prompt text, and removing it from the model-visible tool list removes the capability instead of changing only an instruction sentence. The tool router enforces availability; see [Tools System](/architecture/tools). The `Hidden Tool Domains` prompt section is only a compact catalog for `request_tools`. It lists domains such as `memory`, `task`, `artifact`, and `computer_use` with their exact tool names so the model knows which domain to request if a needed hidden tool is not currently visible. It does not include JSON schemas and does not make those tools callable by itself. MCP follows the same boundary. Composer-selected MCP servers and tools do not create an MCP prompt section. They materialize as dynamic provider tools in `ChatRequest.tools` and are called through the gateway MCP service. ## Current permissions prompt For restricted or narrowed permission profiles, the agent adds a dynamic `Current Permissions` section. It summarizes the selected mode and the action categories that may require approval, such as file writes, shell commands, network access, task/subagent launches, or other external actions. This section is guidance for the model, not the enforcement mechanism. Enforcement happens in `pioneer-tools` through `PermissionEvaluationContext` before a tool action executes. The prompt section exists so the model can avoid unnecessary blocked actions and continue with allowed alternatives when the user denies a request. The default unconstrained `full_access` profile does not render this section. ## Prompt manifest Every prompt compilation emits a `PromptManifest` durable event with compiler version, profile, section ids, stable/dynamic/full fingerprints, and diagnostics. The gateway persists this metadata on the turn through `CrudStore::update_turn_prompt_manifest`. The manifest is the audit handle. It tells a developer which prompt shape was sent, whether bootstrap files were missing or truncated, and whether dynamic sections changed between provider rounds. The manifest intentionally stores metadata rather than turning the prompt into the only source of truth. The actual behavior is the combination of prompt sections, conversation messages, tool definitions, provider adapter behavior, and runtime policies. ## Common debugging path When model behavior looks wrong, check these in order: 1. Was the thread in `chat` or `agent` mode? See [Agent Loop](/architecture/agent-loop). 2. Was history compressed or truncated by the gateway? 3. Which prompt sections appear in the `PromptManifest`? 4. Were relevant skills resolved, disabled, or non-implicit? See [Skills Architecture](/architecture/skills). 5. Did memory hooks contribute recall context or memory tools? See [Memory Architecture](/architecture/memory). 6. Which tools were model-visible for that provider round? See [Tools System](/architecture/tools). 7. Which permission profile and execution security snapshot were active, and did the model receive a `Current Permissions` section? See [Permission System](/architecture/permissions). 8. Did the provider adapter map `compiled_prompt` in the expected way? See [Provider System](/architecture/providers). ## Related pages * [Gateway](/architecture/gateway) explains where history loading is triggered. * [Agent Loop](/architecture/agent-loop) explains how prompt compilation fits into provider/tool rounds. * [Permission System](/architecture/permissions) explains the effective profile and security snapshot that produce current permission guidance. * [Hook Runtime](/architecture/hooks) explains how hook contributions become prompt context and prompt sections. * [Memory Architecture](/architecture/memory) explains the memory prompt contract and post-turn extractor prompt. * [Thread Episodic Context](/architecture/thread-episodic-context) explains recalled conversation snippets and artifact refs. * [AGENTS.md Architecture](/architecture/agents-md) explains the thread-tree instruction hook. * [Skills Architecture](/architecture/skills) explains the skills prompt. * [Tasks And Subagents](/architecture/tasks) explains why task orchestration can add prompt policy.