Frame protocol

The binary wire format: frame types, CBOR encoding, field semantics.

Wire format

Every CapDAG message is a length-prefixed CBOR frame:

┌──────────────────────────────────────────┐
│  4 bytes: u32 big-endian payload length  │
├──────────────────────────────────────────┤
│  N bytes: CBOR-encoded map (integer keys)│
└──────────────────────────────────────────┘

The 4-byte prefix gives the exact byte count of the CBOR payload that follows. The receiver reads exactly that many bytes, then decodes the CBOR map.

There are two size limits:

• Negotiated max_frame (default 3.5 MB / 3,670,016 bytes): Set during handshake. Frames larger than this are rejected by both reader and writer.
• Hard limit (16 MB / 16,777,216 bytes): An absolute ceiling to prevent memory exhaustion. Cannot be negotiated upward.

Payloads that exceed max_frame must be split into chunks (see Streaming).

Source: capdag-rs/src/bifaci/io.rs (encode_frame, decode_frame, write_frame, read_frame).

Frame structure

The CBOR payload is a map with integer keys. Every frame has three required keys (version, frame_type, id). All other keys are optional and their presence depends on the frame type.

Key reference table

Source: capdag-rs/src/bifaci/frame.rs (keys module; Frame struct).

Frame types

The following diagram shows how frame types relate within a request lifecycle:

stateDiagram-v2
    [*] --> Hello : connection setup
    Hello --> Req : connection live

    state "Request Lifecycle" as RL {
        Req --> StreamStart : open stream
        StreamStart --> Chunk : send data
        Chunk --> Chunk : more data
        Chunk --> StreamEnd : stream complete
        StreamEnd --> StreamStart : open another stream
        StreamEnd --> End : all streams done
        Req --> End : empty response

        Req --> Err : request failed
        StreamStart --> Err : request failed
        Chunk --> Err : request failed
    }

    state "Anytime" as AT {
        Log : Log (progress, info)
        Heartbeat : Heartbeat (health probe)
    }

Hello (0)

Sent once by each side during connection setup. The host sends Hello first (no manifest); the cartridge responds with Hello including a JSON-encoded manifest in meta["manifest"]. The cartridge manifest has the structure:

{
  "name": "mycartridge",
  "version": "1.0.0",
  "description": "...",
  "cap_groups": [
    {
      "name": "default",
      "caps": [{"urn": "cap:effect=none", "title": "Identity", ...}, ...],
      "adapter_urns": []
    }
  ]
}

The meta map also carries limit proposals: max_frame, max_chunk, and max_reorder_buffer. Both sides negotiate limits by taking the minimum of the two proposals. A cartridge’s HELLO additionally carries the mandatory pools entry: JSON bytes encoding its full concurrency-pool state map (one singleton pool per registered cap, every manifest-declared shared pool, and the reserved all pool), each with declared/configured/optional available/active/queued; see 13.1). A missing or malformed map is a protocol violation, not an invitation to infer a default. The heartbeat is the capacity CONFIG channel: a probe may carry desired_capacities (JSON bytes: pool name → operator configured value), and every heartbeat reply carries the mandatory refreshed pools map. An unknown pool name refuses the whole desired batch with an UNKNOWN_POOL ERR in place of the reply. See Handshake for the full handshake sequence.

Hello uses id = Uint(0) and is not associated with any request.

Source: frame.rs (Frame::hello, Frame::hello_with_manifest).

Req (1)

Initiates a capability invocation. The cap field (key 10) carries the cap URN to invoke. The request may include a payload and content_type for inline data, though most arguments arrive as separate streams (STREAM_START/CHUNK/STREAM_END).

The routing_id (key 13) is assigned by the RelaySwitch at routing boundaries. Cartridges sending peer invocations do not set routing_id; it is added by the infrastructure.

The cap URN is validated at construction time: Frame::req() panics if the URN is malformed, because an invalid URN is a programming error, not a runtime condition.

Source: frame.rs (Frame::req).

Chunk (3)

Carries a piece of streaming data within a named stream. Required fields:

• stream_id (key 11): Which stream this chunk belongs to.
• chunk_index (key 14): Zero-based position of this chunk within the stream. Monotonically increasing.
• checksum (key 16): FNV-1a 64-bit hash of the payload bytes. Mandatory for corruption detection.

Optional fields set by the chunking layer:

• offset (key 8): Byte offset of this chunk’s data within the total stream.
• len (key 7): Total byte count. Set on the first chunk only (chunk_index = 0).
• eof (key 9): True on the last chunk.
• content_type (key 4): Set on the first chunk only.

Source: frame.rs (Frame::chunk, Frame::chunk_with_offset).

End (4)

Terminal frame that signals all streams for a request are complete. No flow frame may follow an END for the same request ID (L4); runtimes drop and count post-terminal frames, never write them. May carry an optional final payload.

The eof field is set to true. A successful END carries exit_code = 0 in meta and MAY carry terminal metadata: progress (f64) and message. The terminal progress is the authoritative final progress event (L5). It rides in the terminal frame itself and therefore cannot race it. A successful END without an explicit progress reads as 1.0; a failed END never synthesizes one.

Source: frame.rs (Frame::end).

Log (5)

Carries source diagnostics and progress updates. The meta map (key 5) contains:

• level (non-empty text): One of "info", "warn", "error", "progress", or a source-defined level.
• message (non-empty text): Human-readable message.

• attribution_class (text, mandatory for non-progress LOG): One of input

  resource

  environment

  internal.

• arg_urn (non-empty text, optional for non-progress LOG): The one argument the source attributes the diagnostic to.
• progress (float, optional): A value between 0.0 and 1.0. Present only when level is "progress".

Progress frames are LOG frames with level = "progress". They are constructed only through the progress API and do not carry diagnostic attribution. A non-progress LOG must carry a valid attribution_class; missing or unknown values are protocol violations. arg_urn is present only when the emit source knows that the message concerns one argument. Receivers never infer either field.

LOG frames can appear at any point during a request, interleaved with CHUNK frames. They do not affect the data stream.

Source: frame.rs (Frame::log, Frame::progress, log_level, log_message, log_progress, attribution_class, attribution_arg_urn).

Err (6)

Terminal frame that signals failure. Like END, no further frames may follow for this request. The meta map carries:

• code (non-empty text): Machine-readable error code.

• attribution_class (text): Whose domain the failure belongs to, declared at the definition or emit site. One of input

  resource

  environment

  internal.

• message (non-empty text): Human-readable error description.
• arg_urn (non-empty text, optional): The media URN of the one argument the source attributes the failure to.

All three required fields are strict. A missing or unknown attribution_class is a protocol violation; receivers do not manufacture internal. When the source cannot attribute the failure to one argument, arg_urn is absent, never empty. See Error handling.

ERR and END are mutually exclusive terminals. A request ends with exactly one of them.

Source: frame.rs (Frame::err, error_code, attribution_class, attribution_arg_urn, error_message).

Heartbeat (7)

Health monitoring frame. Either side can send a Heartbeat; the receiver must respond with a Heartbeat carrying the same id. The CartridgeHostRuntime sends heartbeats every 30 seconds and expects a response within 10 seconds. A cartridge that fails to respond is marked unhealthy.

Heartbeat frames bypass sequence numbering because they are not flow frames and do not participate in ordering.

Source: frame.rs (Frame::heartbeat); see also host_runtime.rs (HEARTBEAT_INTERVAL, HEARTBEAT_TIMEOUT).

StreamStart (8)

Announces a new named stream within a request. Carries:

• stream_id (key 11): A UUID uniquely identifying this stream within the request.

• media_urn (key 12): The media URN (see Media URNs) that identifies the data type of the stream.

• output (key 21): on a response stream that is not the cap’s main output, the media URN of the output this stream is. The stream of the main output carries no output, and neither does a request stream.

A request carries one STREAM_START per argument it is given, each labelled with the URN the cap’s definition delivers that argument under (CapArg::stream_urn()), so an op finds each argument by a URN it declares. A request stream that carries output is refused by the runtime that reads it: it is not an argument.

A response carries at most one STREAM_START per declared output. The one that names no output is the cap’s main output, the output the cap URN’s out= states: whoever asked for the cap reads its answer without holding the cap’s definition, which is what a cartridge calling a peer and a command line printing a result both do. Every other stream names the output it is. media_urn is always the type of the data; output says which of the cap’s other outputs a stream is, which the type alone cannot when two outputs carry values of one type. A stream that names the main output, a stream that names an output the cap does not declare, and a second stream for one output are each a contract violation. An optional output that does not fire has no stream. See Input and output.

Source: frame.rs (Frame::stream_start).

StreamEnd (9)

Marks the end of a specific stream. After this frame, any CHUNK for the same stream_id is a protocol error. Carries:

• stream_id (key 11): The stream being ended.
• chunk_count (key 15): The total number of CHUNK frames sent in this stream, by the sender’s count. The receiver can verify it received all chunks. A stream announced unbounded makes no length promise and its STREAM_END omits chunk_count (L16).

Source: frame.rs (Frame::stream_end).

RelayNotify (10)

Sent by a RelaySlave to its paired RelayMaster to advertise the capabilities available on that host. The meta map carries:

• manifest (bytes): JSON-encoded object with the structure:

  {
    "caps": ["cap:effect=none", "cap:in=...;out=...;generate", ...],
    "installed_cartridges": []
  }
  

  caps is a flat list of capability URN strings from all healthy cartridges on the host. installed_cartridges carries cartridge process metadata. CAP_IDENTITY (cap:effect=none) must be present; the RelaySwitch rejects the connection otherwise.

• max_frame, max_chunk, max_reorder_buffer: Protocol limits for this relay connection.

RelayNotify frames are intercepted by the relay and never pass through to the other side. The master stores the manifest and limits and makes them available to the RelaySwitch for routing decisions.

Source: frame.rs (Frame::relay_notify); see also Relay topology.

Credit (13)

Grants per-stream flow-control credit to the sender of a stream (see Flow control). Carries the target request’s RID in key 2, an optional stream_id (key 11; absent means the request’s sole stream), and the grant count in credit (key 19, CHUNK units). Intermediaries forward Credit end-to-end like Cancel; they never originate or absorb it (L11). A CREDIT frame without a credit count is rejected at decode.

Cancel (12)

Cancels one in-flight request, identified by its RID in key 2 (and its XID in key 13 when routed). A Cancel always acts: the receiving runtime cancels the request cooperatively (closes its input, releases its credit waiters, cancels its peer calls, and answers with a terminal ERR once the handler has exited), or, with force_kill (key 18), the host kills the cartridge process.

A Cancel is attributed exactly like an ERR frame, through meta (key 5), in the failure taxonomy’s vocabulary (see Error handling). Every entry is optional, and an unattributed Cancel still cancels:

CancelReason (frame.rs) is the typed form: user(force_kill), collateral(class, detail), host(class, detail, force_kill), unattributed(force_kill); Frame::cancel_reason() reads it back from any Cancel, never failing: an unknown class token decodes as “no class”, a missing meta as unattributed. Cascades carry the reason unchanged (a peer call of an aborted request is collateral of the same failure), and a force-kill death answers every pending request in the reason’s own code and class. “Cancelled” therefore always means a human cancelled; a teardown because something else failed says so and is attributed to that failure.

A request ends once. Its handler’s END (or ERR) and a cancel’s ERR are alternatives, never both: whichever claims the request first is its terminal. A cancel that arrives after the handler has ended the request adds nothing, and once a cancel has claimed it the handler’s own terminal is not sent. Any frame for the request after its terminal is a post-terminal straggler the writer suppresses (L4). An in-process host sends the cancel’s ERR before it closes the handler’s input, so a handler that notices the cancel first cannot end the request in its own terms.

Input cut short is not complete input. When a cancel (or the connection’s end) closes a request’s input, every stream still open, and the input package itself, fails with an abandoned error rather than ending; a peer response cut short fails the same way. A handler can never take part of its input for all of it. Accumulating helpers (accumulate_input) refuse input that did not reach its END or that carried an ERR.

Source: frame.rs (CancelReason, Frame::cancel, Frame::cancel_reason), cartridge_runtime.rs (the Cancel arm, TerminalClaim, the demuxes), in_process_host.rs, relay_switch.rs (cancel_request).

CloseStream (14)

Closes one request’s live input stream(s), the tap-off (15.2 §Runs stop). Carries the target request’s RID in key 2 and, optionally, the stream_id (key 11) of the one input stream whose feed to close; absent means every live feed the request holds. The receiving runtime closes the matching feed tap(s): capture ends, the handler drains what was already captured, and the request ends naturally with END and complete outputs. Nothing is cancelled: host-side request state is untouched, no terminal is synthesised, and the frame never cascades to peer calls (the tap is on the feed-bearing request alone; its peers drain naturally). A CloseStream for a request holding no matching open feed is logged and ignored; it never becomes an abort.

CloseStream is a non-flow control frame like Cancel and Credit (no seq, no reorder buffering), routed by RID→XID to the request’s destination, and forwarded by intermediaries without interpretation.

Source: frame.rs (Frame::close_stream), cartridge_runtime.rs (the CloseStream arm), relay_switch.rs (stop_request_feeds).

RelayState (11)

Sent by a RelayMaster to its paired RelaySlave to provide host resource information. The payload (key 6) carries an opaque blob (CBOR or JSON encoded by the engine) containing resource state such as model paths, GPU availability, or capability demands.

Like RelayNotify, RelayState frames are intercepted by the relay and never pass through.

Source: frame.rs (Frame::relay_state).

Policy (15)

What the host allows the cartridges it serves to do, as it stands now. It travels one way, from the engine toward cartridges, and is never sent by a cartridge. Today it says one thing, meta["network"]:

A Policy carries id = Uint(0), no seq, and is not reorder-buffered. A Policy without network, or with any other value, is rejected at decode.

Who sends it, and when. The engine gives the RelaySwitch its live policy. The switch sends it to each master as the first frame after that master’s RelayNotify, before it routes anything there, and to every master each time it changes. Relays pass it through. A CartridgeHostRuntime keeps the latest one it was given and sends it to each cartridge it has connected, before the first request it routes to that cartridge, and to every cartridge each time it changes. A host that runs without a switch (the command line, an in-process host) is given its policy by whoever builds it. A host is never without one when it routes a request: one that has not been given a policy when a request reaches it refuses that request as a protocol violation.

What a cartridge does with it. The runtime keeps the latest Policy and a handler reads it live. The request’s own value is not a snapshot. Code that reaches the network asks first, before each call, so a change to offline stops a running download or quote stream at its next call. A call the policy does not allow is not made: it fails with code NETWORK_POLICY, class user, and a message naming the policy. A REQ that reaches a cartridge before any Policy is a protocol violation, except the host’s identity check (CAP_IDENTITY), which is part of the handshake and can precede it: a host connects its cartridges before it has a policy to give them. A cartridge run from a command line, with no host, is under full, because the person running it decides for themselves.

A cartridge that never reaches the network has nothing to ask. It still receives Policy frames and keeps the latest one.

Source: frame.rs (Frame::policy, HostPolicy), cartridge_runtime.rs (the Policy arm, Request::network), host_runtime.rs (set_policy), relay_switch.rs (set_policy).

MessageId

Frame IDs come in two variants:

• Uuid: 16 random bytes (v4 UUID). Used for request IDs and stream IDs, and anything else that needs to be globally unique.
• Uint: A 64-bit integer. Used for control frames (Hello uses id = 0, Heartbeat can use any integer).

UUIDs are encoded as CBOR byte strings (major type 2, 16 bytes). Integers are encoded as CBOR unsigned integers (major type 0).

Source: frame.rs (MessageId enum).

Flow frames vs control frames

Frames are classified as flow or non-flow. The distinction matters for sequence numbering and reordering:

• Flow frames: Req, Chunk, End, Log, Err, StreamStart, StreamEnd. These participate in sequencing: the SeqAssigner assigns monotonically increasing seq values per flow, and the ReorderBuffer at relay boundaries reorders them if they arrive out of sequence.
• Non-flow frames: Hello, Heartbeat, RelayNotify, RelayState, Cancel, Credit, CloseStream, Policy. These bypass sequence assignment entirely (their seq stays at 0) and are not buffered by the reorder buffer. Credit in particular must never queue behind the data it is flow-controlling.

The is_flow_frame() method on Frame returns this classification, and FrameType::is_terminal() says which flow frames end a flow (End, Err).

Both are the proved model’s decisions (formal/CapDAG/Bifaci/Flow.lean, FrameType.isFlow / isTerminal), generated into every mirror: no runtime keeps its own list.

Source: frame.rs (FrameType::is_flow, FrameType::is_terminal, Frame::is_flow_frame).

Checksum

Every CHUNK frame carries a checksum in key 16: an FNV-1a 64-bit hash of the payload bytes. The sender computes it via Frame::compute_checksum():

const FNV_OFFSET_BASIS: u64 = 0xcbf29ce484222325;
const FNV_PRIME: u64 = 0x100000001b3;

let mut hash = FNV_OFFSET_BASIS;
for &byte in data {
    hash ^= u64::from(byte);
    hash = hash.wrapping_mul(FNV_PRIME);
}

The receiver verifies the checksum after decoding. A mismatch is a protocol error, and the frame is rejected.

FNV-1a is fast and simple, not cryptographically strong. It detects accidental corruption (bit flips, truncation) but is not designed to resist intentional tampering.

Source: frame.rs (compute_checksum).