Relay switch

Cap-aware routing multiplexer that sits between the engine and relay masters.

RelaySwitch

The RelaySwitch is the central routing component of the CapDAG network. It connects to one or more RelaySlave/RelayMaster pairs (each backed by a CartridgeHostRuntime managing one or more cartridges) and routes requests to the correct master based on cap URN matching.

All methods take &self (not &mut self). Interior mutability (RwLock, Mutex, AtomicU64, AtomicBool) allows multiple concurrent DAG executions to route frames through the switch simultaneously.

pub struct RelaySwitch {
    masters: RwLock<Vec<MasterConnection>>,
    cap_table: RwLock<Vec<(String, usize)>>,   // cap_urn → master index
    requests: RwLock<RequestTable>,            // unified per-request state (L7)
    drops: Arc<DropCounters>,                  // counted drops (L8)
    xid_counter: AtomicU64,
    live_cap_fab: RwLock<LiveCapFab>,
    cap_registry: Arc<FabricRegistry>,
    // ...
}

Unified request state. All per-request state lives in one RequestTable (capdag-rs/src/bifaci/request_state.rs): one entry per in-flight request holding its routing, origin, peer marker, cancel-cascade children, response channel, per-stream flow stats, and timestamps, plus the rid→xid secondary index. A request is registered once (duplicate registration is a protocol violation and fails hard) and terminated once (terminate(key, kind) with kind End | Err | MasterDied, or terminate_cancelled(key, reason); a Cancelled termination carries the Cancel’s attribution (terminal code, class, reason), and the summary it leaves in the terminated ring records them). After termination, zero state remains for the key, atomically. The terminal frame is forwarded using the state terminate hands back, so state release and terminal delivery cannot disagree (L6).

Source: capdag-rs/src/bifaci/relay_switch.rs.

Initialization

Creating a RelaySwitch involves two phases:

Phase 1: Verify each master:

For each Unix socket connection:

1. Split into read/write halves.
2. Read the first frame, which must be a RelayNotify carrying the host’s aggregate manifest and limits. 2a. Send the switch’s current Policy, before anything else reaches that host. The switch is built with a policy and never exists without one.
3. Run identity verification: send REQ for CAP_IDENTITY (cap:effect=none) with nonce, verify echo response.
4. During verification, additional RelayNotify frames may arrive (as CartridgeHostRuntime discovers and connects its cartridges). These update the master’s manifest.
5. Stash the reader for phase 2.

Phase 2: Spawn readers:

After all masters are verified, spawn a reader task (tokio::spawn) for each master. These tasks read frames from the socket and send them through an mpsc channel to the switch’s routing logic.

This two-phase design ensures that identity verification runs sequentially per master (no reader task interference) while reader tasks start only after the entire switch is verified.

Source: relay_switch.rs (RelaySwitch::new).

Cap routing

flowchart TD
    REQ["REQ(cap_urn)"] --> PARSE["Parse cap URN"]
    PARSE --> MATCH["Match against each<br/>master's cap set<br/>(is_dispatchable)"]
    MATCH --> RANK{Multiple matches?}
    RANK -->|Yes| SPEC["Rank by specificity"]
    RANK -->|No| SKIP
    SPEC --> SKIP{Master healthy?}
    SKIP -->|No| ERR["RelaySwitchError::NoHandler"]
    SKIP -->|Yes| XID["Assign XID<br/>(atomic increment)"]
    XID --> REC["Record (XID, RID)<br/>in routing tables"]
    REC --> FWD["Forward REQ to<br/>selected master"]

When a REQ frame arrives (from the engine or from a peer cartridge), the switch routes it:

1. Parse the cap field (key 10) as a cap URN.
2. Match against each master’s capability set using is_dispatchable(candidate, request) (see Dispatch). The candidate’s cap URN must accept the request’s input type (contravariant) and produce a compatible output type (covariant).
3. Rank if multiple masters match. Specificity ranking (see Ranking) selects the most specific candidate.
4. Skip unhealthy masters: masters marked unhealthy are excluded from routing.
5. Assign XID: Atomically increment the xid_counter and assign the new value as the request’s routing_id.
6. Record routing: Register (XID, RID) in the RequestTable with its RoutingEntry; the table also records RID → XID.
7. Forward: Send the REQ (with XID) to the selected master’s socket writer.

If no healthy master can handle the cap, the switch returns RelaySwitchError::NoHandler.

Source: relay_switch.rs.

XID assignment

XIDs (routing IDs, key 13 in frames) are unsigned integers assigned by the RelaySwitch at routing boundaries. They serve a single purpose: letting the switch route continuation frames to the correct master.

• Incoming REQ from engine: Has no XID. The switch assigns one and records the mapping.
• Continuation frames (CHUNK, STREAM_START, etc.): Carry the XID from the original REQ. The switch looks up (XID, RID) in the RequestTable to find the destination.
• Response frames from master: Carry the XID. The switch uses the entry’s origin to determine where to send them (back to the engine or to the requesting cartridge’s master).

Cartridges never see XIDs directly; they are purely infrastructure. A cartridge’s REQ has no XID; the CartridgeHostRuntime forwards it, and the switch adds one.

Source: relay_switch.rs, frame.rs (routing_id field).

Peer request routing

When Cartridge A calls a cap provided by Cartridge B:

1. Cartridge A’s REQ arrives at the switch (via CartridgeHostRuntime → RelaySlave → RelayMaster) without an XID.
2. The switch assigns an XID and routes to Cartridge B’s master using the same cap matching logic.
3. The request is registered with is_peer = true, which drives its cleanup.
4. The entry’s origin records that this request came from Cartridge A’s master index (not from the engine).
5. Response frames from Cartridge B flow back through the switch to Cartridge A’s master.

The cleanup semantics for peer requests differ from engine-initiated requests: the switch must wait for the full response (Cartridge B’s END) before cleaning up, because the response needs to reach Cartridge A.

Source: relay_switch.rs.

Request tracking

The switch keeps all per-request state in the RequestTable: one RequestState per in-flight request, keyed by (XID, RID), plus a RID → XID index for frames that carry only a RID (continuation frames).

Terminal frames (END, ERR) trigger terminate(key, kind), one atomic removal. A frame arriving with no routing state is counted and classified for what it actually is, never a protocol error and never silent (L6/L8): if its RID is in the table’s bounded recently-terminated ring (recently_terminated_rid), the frame crossed its request’s terminal in flight: a BENIGN post-terminal straggler, the ordinary teardown race of credit-based flow control, counted per frame type in the straggler counters and indicated as benign (it is NOT a drop; nothing went wrong). A RID the table never knew counts as a no_route DROP, the genuine routing-anomaly alarm. This classification (account_unrouted_frame) applies at all three unroutable sites: a duplicate terminal for a released request, a response frame with no routing state, and a continuation with no routing state.

The request lifecycle the table implements (a request is registered once, ended once, leaves no state behind, and a frame with no routing state is a straggler exactly when its request ended lately) is stated and proved in formal/CapDAG/Bifaci/Request.lean (laws in Laws/Request.lean). The decisions are generated from it into every mirror: which frame types end a request and how (terminalOf), where a frame with no route goes (dispose), what a frame does to a stream’s credit ledger (ledger), and what the writer does with a frame once its flow is over (write). Each mirror’s RequestTable is replayed against the model’s own table, step for step.

Registration seeds each request’s flow ledger with the destination master’s negotiated initial_credit (master_initial_credit), and the engine-facing send path (send_to_master) records outbound continuation/control frames in the same ledger, so per-stream credit_outstanding in the stats snapshot is the stream’s remaining window (see Flow control).

Cancel and stop

Two host-side controls act on a live request, and they are not the same thing:

• cancel_request(rid, reason): the abort path. Terminates the request as cancelled with the reason (terminal code, class and reason recorded on its summary; CANCELLED and no class for an unattributed reason), sends the Cancel frame (attribution in meta, force_kill) to the destination master, cascades to every child peer call under the same reason, and delivers the terminal ERR to the external response channel in the reason’s own words: CANCELLED/user, ABORTED_COLLATERAL with the originating failure’s class, ABORTED with the host’s. cancel_all_requests(reason) does this for every engine-origin request. The reason is optional attribution, never a precondition.
• stop_request_feeds(rid): the tap-off path (15.2 §Runs stop). Sends a CloseStream frame to the destination without touching host-side request state: the cartridge closes the request’s feed taps and the request ends naturally with END after the drain, so routing, forwarding, and every downstream cap stay live. Returns whether the request was live and the stop was sent; a request the switch does not know is not stopped, and the caller must not claim it was. The evidence that the input then ended is the request’s terminal, observable via recent_terminal_of_rid / await_request_terminal; bifaci::live_feed::stop_feed_requests builds a host’s stop verdict from exactly that evidence.

Policy

set_policy(policy) replaces the switch’s live Policy and sends it to every master, where each host passes it to its cartridges. Setting the policy it already holds sends nothing. The switch is constructed with its first policy (RelaySwitch::new(sockets, policy)), so there is never a moment a host is connected and has not been told. The engine calls set_policy whenever the operator changes the network setting.

Source: relay_switch.rs.

Dynamic master addition

add_master() connects a new master at runtime:

1. Accept a new Unix socket connection.
2. Read RelayNotify, send the current Policy, perform identity verification.
3. Spawn a reader task.
4. Merge the new master’s capabilities into the aggregate cap table.
5. Rebuild the LiveCapFab for path finding.

This enables hot-plugging of new hosts without restarting the switch.

Slots, ids, and attachment

Every master fills a slot with a stable, caller-chosen id ("external-cartridges", "cartridge-host", …). The slot index is the routing key for the request table and the cap table, so it never changes: when the host behind a slot dies and reconnects, add_master with the same id reattaches the new socket in place at the existing index.

Whether a slot can be (re)attached is decided by its socket, never by its health flag: a slot is attached while its reader task is alive, and detached once the reader has exited (EOF, I/O error) or handle_master_death has declared the connection dead, which also retires the reader. A slot that is merely unhealthy (identity probe pending or failed) with its socket open is still attached, and reattaching over it would abort a live reader and strand the host on the other end.

• add_master(id, socket) on an attached slot is a protocol error (“already attached to a healthy / attached-but-unhealthy slot”); the same master must not be attached twice. On a detached slot it reattaches.
• slot_accepts_attach(id) answers the same question without consuming a socket. A host that (re)connects on every control-plane event (the engine’s cartridge channel re-establishing, or a client reload) must ask it before dialling the relay socket: a socket dialled into an attached slot is a duplicate connection the other side has to refuse, and add_master can only discover that after reading the socket.

The host side of the same rule: a relay host accepts one engine socket per session and refuses (closes) a second while the first is open. A duplicate dial is never a session turnover and never a reason to take the host, and every cartridge it runs, down.

Source: relay_switch.rs (add_master, slot_accepts_attach, handle_master_death_with_reason).

Master health

Each master tracks its health status:

pub struct MasterHealthStatus {
    pub index: usize,
    pub healthy: bool,
    pub cap_count: usize,
    pub connected_seconds: u64,
    pub last_error: Option<String>,
}

A master becomes unhealthy when:

• Its reader task encounters an I/O error (socket closed, CBOR decode failure).
• Identity verification fails during initialization, or a deferred runtime identity probe is pending or failed.

Health is a routing property. Whether the slot’s socket is still attached is a separate one (see Slots, ids, and attachment above): an unhealthy master whose socket is open is still attached and cannot be reattached over; a dead connection detaches the slot whatever its health was.

Unhealthy masters are excluded from cap routing, and their capabilities are not considered when matching REQ frames. The healthy flag is an AtomicBool on MasterConnection, checked during every routing decision.

Source: relay_switch.rs (MasterHealthStatus, MasterConnection::healthy).

Capability aggregation

capabilities() returns the union of all healthy masters’ capabilities as an aggregate JSON manifest. This is the engine’s view of what the entire cartridge network can do.

The switch also exposes:

• get_reachable_targets(): queries the LiveCapFab to find which output media URNs are reachable from a given input.
• find_paths_to_exact_target(): finds machines that transform an input into a specific output.

These methods are used by the planner (see Planner) to find execution paths through the capability graph.

Source: relay_switch.rs.

Protocol stats

RelaySwitch::protocol_stats() returns a serializable snapshot: every live request’s phase (created/streaming), age, idle time, per-stream frame/byte/ chunk/credit counters, and children; a bounded ring of recently-terminated request summaries (kind, with the cancel’s terminal code, class and reason when the kind is cancelled; lifetime, flow totals); lifetime registration and per-kind termination totals; and the per-reason drop counters. The host runtime exposes the same shape via protocol_stats() (drop counters + routing table sizes + GC totals), shipped to the switch inside every RelayNotify stats republish; cartridges report their drop totals on every heartbeat.

The switch RETAINS each master’s advertised host stats (hosts map on the snapshot, keyed by master id), and the host ingests each cartridge’s heartbeat drops_total into the inventory record’s runtime_stats.protocol_drops_total (absent until the first reading). Nothing on the L8 surface is parsed-and-discarded: every reading a party produces is observable end-to-end (engine gRPC GetProtocolStats, the run’s flow_states, and the cartridge inventory).

Swift equivalent

The RelaySwitch class in capdag-objc/Sources/Bifaci/RelaySwitch.swift provides the same routing behavior for Swift-based engine hosts. The routing semantics (cap matching, XID assignment, peer tracking) are identical.