CapDAG documentation

Host runtime

The host-side runtime that spawns cartridge processes, routes frames to and from them, and monitors their health.

In this section

The host-side manager that spawns cartridge processes, routes frames, and monitors health.

CartridgeHostRuntime

CartridgeHostRuntime sits between a RelaySlave and one or more cartridge processes. It manages their full lifecycle: spawning, handshake, frame routing, health monitoring, and death handling.

graph LR
    RS["RelaySlave"] <--> PH["CartridgeHostRuntime"]
    PH <-->|stdin/stdout| PA["Cartridge A"]
    PH <-->|stdin/stdout| PB["Cartridge B"]
    PH <-->|stdin/stdout| PC["Cartridge C"]

The struct tracks each cartridge as a ManagedCartridge with its process handle, I/O channels, manifest, capabilities, health status, and pending heartbeats.

pub struct CartridgeHostRuntime {
    cartridges: Vec<ManagedCartridge>,
    cap_table: Vec<(String, usize)>,           // cap_urn → cartridge index
    outgoing_rids: HashMap<MessageId, usize>,   // peer requests: RID → cartridge index
    incoming_rxids: HashMap<(MessageId, MessageId), usize>, // relay requests: (XID,RID) → cartridge
    outgoing_max_seq: HashMap<FlowKey, u64>,    // max seq per flow (for ERR frames)
    capabilities: Vec<u8>,                      // aggregate manifest JSON
    event_tx: mpsc::UnboundedSender<CartridgeEvent>,
    event_rx: Option<mpsc::UnboundedReceiver<CartridgeEvent>>,
}

Source: capdag-rs/src/bifaci/host_runtime.rs.

Cartridge registration

Two ways to connect cartridges:

register_cartridge(path, known_caps)

Registers a cartridge binary for on-demand spawning. The cartridge is not started until a REQ arrives for one of its capabilities.

host.register_cartridge(Path::new("/path/to/ggufcartridge"), &[
    "cap:in=...;out=...;generate;model=gguf".to_string(),
    "cap:in=...;out=...;describe;model=gguf".to_string(),
]);

The known_caps parameter enables provisional routing before the HELLO handshake reveals the cartridge’s actual manifest. After spawn + HELLO, the real caps from the manifest replace the provisional ones.

attach_cartridge(read, write)

Attaches a pre-connected cartridge (already running). Performs HELLO handshake and identity verification immediately. Used for testing and pre-spawned processes.

let cartridge_idx = host.attach_cartridge(reader, writer).await?;

On HELLO failure, returns an error. The failure is permanent: the binary is considered broken and will not be retried.

Source: host_runtime.rs (register_cartridge, attach_cartridge).

Cartridge spawning

sequenceDiagram
    participant R as Relay
    participant PH as CartridgeHostRuntime
    participant P as Cartridge (new)

    R->>PH: REQ(cap)
    Note over PH: Cartridge not running — spawn on demand
    PH->>P: spawn process (no args)

    rect rgb(240, 248, 255)
        Note over PH,P: Handshake
        PH->>P: Hello (limits)
        P->>PH: Hello (limits + manifest)
        PH->>P: REQ(CAP_IDENTITY = cap:effect=none, nonce)
        P->>PH: nonce echo
    end

    Note over PH: Update cap table from manifest
    PH->>P: Forward original REQ(cap)

When a REQ arrives for a cap handled by a registered-but-not-running cartridge:

  1. Spawn: The host starts the cartridge binary as a child process with no arguments (triggering cartridge CBOR mode). The child inherits nothing but its three stdio descriptors. Rust gets this for free (every descriptor Rust opens is close-on-exec; kill_on_drop reaps the child with the host); the Swift host spawns with POSIX_SPAWN_CLOEXEC_DEFAULT and marks its relay listener and accepted engine socket FD_CLOEXEC. This is a contract: a cartridge that inherits the host’s descriptor table holds the engine-facing relay socket open after the host dies, so the engine never sees EOF, keeps the dead host’s slot attached, and refuses every reconnect; it also holds the write end of a sibling cartridge’s stdin, so that sibling never sees EOF either.
  2. Handshake: HELLO exchange and limit negotiation (see Handshake).
  3. Identity verification: REQ for CAP_IDENTITY (cap:effect=none) with nonce echo.
  4. Reader/Writer tasks: A reader task reads frames from the cartridge’s stdout; a writer task writes frames to the cartridge’s stdin. A third task drains the cartridge’s stderr line by line for the process’s whole life, into the host’s log and a bounded per-cartridge tail. This is a contract, not a convenience: a stderr pipe nobody reads fills its 64 KiB buffer and then blocks the cartridge’s next write inside whatever it was doing (a capture loop included), /dev/null hides every complaint a living cartridge makes, and a read-only-at-death (readToEnd) blocks for as long as any holder of the pipe’s write end lives. Every mirror drains continuously.
  5. Update cap table: The cartridge’s manifest replaces the provisional known_caps.
  6. Forward REQ: The original REQ is forwarded to the now-running cartridge.

If HELLO fails, the cartridge is marked with hello_failed = true and will not be spawned again. The REQ that triggered the spawn receives an ERR frame.

Source: host_runtime.rs.

Frame routing: engine → cartridge

flowchart TD
    F["Frame from relay"] --> FT{frame_type?}
    FT -->|REQ| CAP["Lookup cap_urn<br/>in cap_table"]
    CAP --> RUN{Cartridge running?}
    RUN -->|No| SP["Spawn on demand"]
    SP --> FWD
    RUN -->|Yes| FWD["Forward to cartridge<br/>Record (XID,RID) in<br/>incoming_rxids"]

    FT -->|"STREAM_START<br/>CHUNK<br/>STREAM_END<br/>END<br/>ERR"| CONT["Lookup (XID,RID)<br/>in incoming_rxids"]
    CONT --> FWD2["Forward to<br/>handling cartridge"]

Frames arriving from the relay (engine side) are routed to cartridges:

REQ: The cap URN (key 10) is looked up in cap_table to find which cartridge handles it. If the cartridge is registered but not running, it is spawned on demand. The (XID, RID) pair is recorded in incoming_rxids mapping to the destination cartridge index.

Continuation frames (STREAM_START, CHUNK, STREAM_END, END, ERR): Routed by (XID, RID) lookup in incoming_rxids. The frame is forwarded to the cartridge that is handling the original request.

An unroutable continuation (or LOG) is counted and classified for what it actually is, never silent (L6/L8). The host keeps a bounded ring (recent_released_rids, 64 entries, deduplicated) of RIDs whose routing entries were released by an OBSERVED terminal: a completed request, a completed peer response, or a cartridge death (which synthesizes the ERR terminal itself). A frame whose RID is in the ring crossed its request’s terminal in flight: a BENIGN post-terminal straggler, counted per frame type in the straggler counters and indicated as benign (NOT a drop; nothing went wrong). A RID the host never routed counts as a no_route DROP. GC evictions are not recorded in the ring: an evicted entry never saw its terminal, so its frames are real routing loss and stay no_route beside routing_gc_evicted_total.

CloseStream: Routed by (XID, RID) to the cartridge handling the request and forwarded verbatim; this is the tap-off (12.2 §CloseStream). Never cascaded (the tap is on the feed-bearing request alone; its peers drain naturally), never a kill, and no routing state changes: the request ends on its own, later, with END.

Cancel: Routed by (XID, RID) like a continuation, carrying its attribution in meta (12.2 §Cancel). A cooperative cancel (force_kill = false) is forwarded to the handling cartridge verbatim and cascaded, under the same reason, to every peer call the request spawned (incoming_to_peer_rids). A force_kill = true Cancel sets ShutdownReason::Cancelled(reason) and kills the process; the death handler then emits the reason’s terminal code and class (CANCELLED/user, ABORTED_COLLATERAL/the originating failure’s class, ABORTED/the host’s) for every request the process held; a kill under a collateral or host abort never reads as a user cancel.

Policy: The host’s new policy (12.2 §Policy). The host replaces the one it keeps and sends it to every running cartridge; a cartridge spawned later is sent the current one right after its handshake, before the REQ it was spawned for. A REQ from the relay that arrives before the host has any policy is answered with a PROTOCOL_VIOLATION ERR and routed nowhere. A host built without a relay is given its policy with set_policy by whoever builds it, and sends it on the same way.

All other frame types from the relay are protocol errors, since the relay should never send Hello, Heartbeat, or RelayNotify/RelayState to a host.

Source: host_runtime.rs (run loop, set_policy).

Frame routing: cartridge → engine

Frames from cartridges are forwarded to the relay (engine side):

HELLO: Fatal error during run. HELLO is consumed during handshake and must never appear afterward.

HEARTBEAT: Handled locally. The host sends back a Heartbeat with the same ID. Never forwarded to the relay.

REQ (peer invoke): The cartridge is calling another cap. The RID is recorded in outgoing_rids mapping to the source cartridge. The frame is forwarded to the relay, which will assign an XID at the RelaySwitch.

RelayNotify / RelayState / Policy: Fatal error. Cartridges must never send relay control frames, and a policy is the host’s to give, never a cartridge’s.

Everything else (STREAM_START, CHUNK, STREAM_END, END, ERR, LOG): Forwarded to the relay. These carry the XID from the original request, which the relay uses for routing.

Source: host_runtime.rs.

Self-loop routing

An edge case arises when a cartridge peer-invokes a cap that another cartridge on the same host handles. Both incoming_rxids and outgoing_rids contain entries for the same request IDs, creating ambiguity about which direction a continuation frame belongs to.

The host resolves this by frame type discrimination:

  • Request body frames (STREAM_START, CHUNK, STREAM_END, END from the relay): These are the arguments of the incoming request. Routed via incoming_rxids to the handling cartridge.
  • Peer response frames (STREAM_START, CHUNK, STREAM_END, END from the handling cartridge): These are the response to the peer call. After the incoming request’s END is received, subsequent frames for the same (XID, RID) are peer response frames routed back to the calling cartridge via outgoing_rids.

Source: host_runtime.rs.

Heartbeat health monitoring

The host probes each running cartridge every 30 seconds:

  1. Send a HEARTBEAT frame with a fresh UUID.
  2. Record the UUID and timestamp in pending_heartbeats.
  3. When the cartridge responds with a HEARTBEAT carrying the same UUID, remove the entry.
  4. If a heartbeat goes unanswered for 10 seconds, mark the cartridge as unhealthy.

Constants:

  • HEARTBEAT_INTERVAL: 30 seconds between probes.
  • HEARTBEAT_TIMEOUT: 10 seconds max wait for a response.

A cartridge whose probe is unanswered at the next tick is killed with ShutdownReason::HeartbeatTimeout: every pending request gets ERR CARTRIDGE_UNHEALTHY (class environment), and the death is written to the host log. There is no recovery of the wedged process; the next REQ for its caps respawns it.

The host asks whether the killed process has gone; it does not wait to be told. Every kill is made from inside the host’s one event loop, and Child::kill().await waits for the process’s exit to be reported — on unix, by SIGCHLD. A report that does not come parks the loop: no frame is routed, no heartbeat sent and nothing failed, for every cartridge the host has. That is what a wedged cartridge did to the engine — the frozen process was killed and stayed a zombie under the engine, the request it had been serving was never failed, and the runs after it never started; once the loop came back six minutes later, when another child exited. A killed process cannot refuse to die and the kernel says at once whether it has, so kill_and_reap sends the signal and polls try_wait (REAP_POLL, 20 ms) until the exit is collected. A process that has still not exited after REAP_LIMIT (10 s) is logged as such and the loop goes on. Why the report is sometimes not delivered in the engine has not been established; it was not reproduced outside it.

The reply is written by the cartridge runtime’s own loop, so a handler doing blocking work does not by itself starve it, but a handler that runs inference inline on the async runtime does. Model loads and forward passes therefore run off the runtime (run_with_keepalive, spawn_blocking, a dedicated worker). See Handler patterns.

Source: host_runtime.rs (HEARTBEAT_INTERVAL, HEARTBEAT_TIMEOUT).

Cartridge death handling

Three scenarios:

Ordered shutdown (ordered_shutdown = true): The host sets this flag before killing a cartridge process intentionally (e.g., on host shutdown). The death handler sees the flag and cleans up routing entries without sending ERR frames.

Unexpected death with pending requests: The cartridge process exited while requests were in flight. The host sends an ERR frame for each pending request ID (both from incoming_rxids and outgoing_rids). The outgoing_max_seq map provides the correct seq number for each ERR frame (max_seen + 1). The last_death_message (including the stderr tail the drain task collected) is included in the error message.

Idle death: The cartridge died while no requests were pending. Routing entries are cleaned up. The next REQ for this cartridge’s caps will trigger a respawn.

Every death that produces ERR frames is also written to the host log at error level, with the cartridge, the code and the message (the OS exit status or signal, the heartbeat timeout, the OOM kill) in every mirror. The ERR frames reach only the requests that were pending; the log line is what a later reader of the saved logs has.

Roster retirement (ShutdownReason::RosterRetired): a SyncRoster says this install is no longer wanted. The cartridge leaves the cap table and the inventory immediately, so nothing new routes to it, but the PROCESS is not killed while it still has in-flight requests: it drains, and is reaped on the host’s periodic tick once its last request terminates or RETIRE_DRAIN_TIMEOUT (10 min) expires. If the same identity returns to the desired roster mid-drain, the retirement is cancelled and the live process is reused. Pending work killed by a retirement gets ERR CARTRIDGE_RETIRED with class environment; it is not CANCELLED, which would report a deployment change as a user cancellation.

Source: host_runtime.rs.

Capability advertisement

When the set of available capabilities changes (cartridge added, died, or respawned), the host rebuilds its aggregate manifest and sends a RelayNotify frame to the relay. The manifest is a JSON object with the structure:

{
  "caps": ["cap:effect=none", "cap:in=...;out=...;generate", ...],
  "installed_cartridges": []
}

caps is a flat list of all capability URN strings from all healthy cartridges. installed_cartridges carries metadata about the running cartridge processes (currently always empty in the relay notify; populated by the CartridgeHostRuntime internally).

The relay’s master receives this and updates its manifest and limits. The RelaySwitch re-reads the master’s capabilities and updates its routing table. This propagation means the engine’s view of available capabilities stays in sync with the actual set of running cartridges.

The inventory is cheap to rebuild. rebuild_capabilities runs on every change and on every periodic republish, under the host lock, on the host loop. The installed-cartridge record it advertises carries the install directory’s content hash, and that hash is computed once: at registration in the reference (new_registered_dir), once per process lifetime in the Swift host (ManagedCartridge.directoryHash, cleared on spawn), never per rebuild. Hashing a model cartridge reads about a gigabyte, so hashing on every republish would keep the host loop from handling death and frame events.

Source: host_runtime.rs (rebuild_capabilities).

Swift equivalent

The CartridgeHost class in capdag-objc/Sources/Bifaci/CartridgeHost.swift provides the same functionality for Swift-based hosts. The frame routing semantics are identical; the differences are in process management (posix_spawn with POSIX_SPAWN_CLOEXEC_DEFAULT, explicit pipes, explicit kill) and async patterns (threads and dispatch instead of tokio).

Counted drops and protocol stats

The host never drops a frame silently (L8). An unroutable continuation (no entry in either routing table: the request already completed) is a counted no_route drop with a warn log. CartridgeHostRuntime::protocol_stats() exposes the per-reason drop counters, the four routing-table sizes, and the GC totals; the snapshot ships to the RelaySwitch inside every RelayNotify stats republish (host_protocol_stats on the capabilities payload). Credit frames route through the host like data continuations, toward whichever cartridge is sending the credited stream (L11).