Models

Bring your own model

Paste a HuggingFace model id and FabricLoom downloads the model. When the download completes, every installed cartridge that can use the model picks it up, whether it does embeddings, transcription, vision, or generation. You choose the runtime (Apple’s MLX, llama.cpp, or Candle), the size, and the quantization.

A 0.5B model that fits in a few hundred megabytes works, and so does a 70B model that needs the whole GPU. The surface is the same, including the right-click and the Transmute menu. Only the speed and the quality of the answer change.

This page covers the user-facing side of model management: model specs, supported backends, model-family detection, where models live on disk, structured generation, and what LlmGenerationRequest lets you tune.

For the cartridge-developer side (declaring an LLM cap, building a LlmGenerationRequest, parsing LlmStreamMessages), see the Cartridge SDK reference.

Model specs

A model is referred to by a model spec string. The most common form is HuggingFace:

hf:OWNER/REPO
hf:OWNER/REPO?include=*PATTERN*

Examples:

hf:meta-llama/Llama-3.2-1B-Instruct
hf:MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF?include=*Q8_0*
hf:mlx-community/Llama-3.2-1B-Instruct-4bit

The include= filter is matched against filenames in the repo and is useful for picking a specific quantization out of a multi-file GGUF release. Without a filter, the cartridge picks a default, usually the largest available quant that fits the runtime’s preferences.

Cartridges also send the model spec string in LlmGenerationRequest.model_spec, and the dispatch rule routes on it. When two cartridges both serve cap:llm-inference;..., they are ranked by how specifically they match the spec. A Mistral-aware cartridge that names the family in its tags beats a generic one for Mistral models.

Backends

Three runtimes are first-class today:

A cartridge advertises which backend it implements. The SDK provides a backend_for_model_spec(model_spec) helper that classifies a spec into one of these, and cartridges that wrap a single backend use it to advertise only the model specs they can handle.

The standard cap URN constants in the cartridge SDK correspond to these backends:

• CAP_LLM_INFERENCE_GGUF
• CAP_LLM_INFERENCE_MLX
• CAP_LLM_INFERENCE_CANDLE
• CAP_LLM_INFERENCE_CONSTRAINED: constrained generation (JSON Schema, regex, grammar, tool-call), routed to whichever backend supports the constraint

When you ask a generic LLM cap, dispatch picks the backend from the model spec. When you call a backend-specific cap directly, you have chosen the runtime yourself.

Model families

Different LLM families use different prompt formats. Mistral uses [INST] ... [/INST], Llama 3 uses header-based turns (<|start_header_id|>user<|end_header_id|>...), Phi uses <|user|> ... <|end|>, and Qwen uses ChatML-style <|im_start|>user ... <|im_end|>. A model that receives the wrong turn markers does not fail visibly. It generates plausible but hallucinated multi-turn conversations.

MachineFabric handles this automatically. When a generation cap is dispatched, the host inspects the model spec string, detects the family from the path (mistral, llama, phi, qwen keywords), and picks the matching adapter. An adapter is a small piece of code that knows the family’s prompt format, its conversation-boundary stop sequences, and how to parse its response.

The recognised families are:

A new family is added by adding family detection in the host, defining the adapter, and adding the corresponding media URN entries. Cartridges that wrap a model do not need to know any of this, because the host selects the adapter before the cartridge sees the request.

Generation parameters

A generation request is a typed payload (LlmGenerationRequest, carried as media:fmt=json;llm-generation-request;record). The fields a cartridge can read:

Defaults are applied by LlmGenerationRequest::with_defaults() in the SDK, so callers set only what they want to override.

Streaming responses

Generation responses arrive as a stream of LlmStreamMessage values, one per line of media:fmt=ndjson;llm-text-stream:

Cartridges that consume the stream can render tokens live, count throughput, and pass tool calls back to whatever orchestrates the conversation. FabricLoom’s interface uses the same stream: the live progress shown during a generation comes from the Token and Status messages.

Structured generation

For structured outputs the request can carry a constraint:

Constrained generation is dispatched through CAP_LLM_INFERENCE_CONSTRAINED. The runtime enforces the constraint at sampling time, so the model cannot generate output that violates it. Structured outputs (JSON, tool calls, classifications) therefore come back well-formed without a parse-and-retry loop.

Standard URN helpers in the SDK pre-build the cap URNs for common structured patterns:

• generate_json_urn(lang_code): generate a JSON record matching a schema
• make_decision_urn(lang_code): pick one of a set of options
• make_multiple_decisions_urn(lang_code): pick a subset of a set of options
• llm_summarization_urn(lang_code): summarize text
• llm_codegeneration_urn(lang_code): generate code

The full URN catalogue is in capdag.standard.caps.

Embeddings and vision

Embeddings and image description follow the same pattern of typed cap URNs and typed payloads:

Embeddings are routed by model spec the same way generation is: paste a HuggingFace embeddings model id and any cartridge that can run it picks it up.

Where models live

Downloaded models are cached on disk so they are not fetched again. The directory layout is a host implementation detail and may change. FabricLoom tracks each downloaded model with a cap:model-status cap and shows installation, removal, and disk-usage views in its interface. The cartridge SDK exposes the model lifecycle through these standard URN builders:

• model_download_urn(): download a model by spec
• model_list_urn(): list installed models
• model_status_urn(): query a model’s installation status
• model_contents_urn(): list the files in an installed model
• model_availability_urn(): check whether a spec resolves to a downloadable model
• model_path_urn(): get the on-disk path of an installed model

Cartridges that need a model file should resolve it through model_path_urn() instead of hardcoding paths. The host resolves the path only after the model is fully downloaded, so cartridges do not have to handle partial downloads.

Mixing local and remote

Cartridges currently run in heavily sandboxed processes with no default network access. The protocol does not preclude a cartridge that proxies to a cloud service. A cartridge that declares network entitlements and is reviewed for inclusion can route requests to a remote model using the same media URNs as every other cartridge. Local and remote capabilities can then be mixed in one pipeline, and the dispatch rule, the planner graph, and live progress behave the same way. Only the place where inference runs differs.

The default today is local. Network proxying is opt-in.