> For the complete documentation index, see [llms.txt](https://docs.algenta.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.algenta.ai/guides/run-qwen3-family-inference.md).

# Model-agnostic LLM inference

Run any supported checkpoint on Apple Silicon through Algenta's model-agnostic SDK path — supported shapes, long context, and the quantized load cache. Currently verified on Qwen3 models.

`algenta.inference.decode()` runs any supported checkpoint through the model-agnostic Metal GPU kernel path: one call, prompt token ids in, generated token ids out, on Apple Silicon. It uses the same kernels as the [local LLM decode guide](/guides/run-qwen3-decode.md) and accepts any model whose architecture the kernels support. Verified end to end on Qwen3-0.6B-Base and Qwen3-1.7B with identical greedy output (first token matches an independent `transformers` CPU reference on both).

## Requirements

* Apple Silicon (M-series) Mac with Metal. There is no CPU fallback and no support for other GPU vendors.
* macOS 26 or newer and Python 3.12 or newer. `pip install algenta` installs the native runtime with Metal on macOS 26 and newer; on macOS 13 to 15 it installs the CPU-only runtime, which cannot run this Metal path.
* A local Hugging Face checkpoint snapshot (`config.json` + one or more `model*.safetensors`, single- or multi-shard), downloaded separately — for example `Qwen/Qwen3-0.6B-Base` or `Qwen/Qwen3-1.7B` (both Apache-2.0, ungated).
* `numpy`, to convert the checkpoint's published BF16 weights: `pip install "algenta[benchmark]"`.
* No external tokenizer needed. The SDK implements the checkpoint's own `tokenizer.json` (Qwen3-family byte-level BPE, byte-for-byte validated against `transformers`) — pass `prompt_text` straight to `decode()`. Token ids are also accepted.
* A Metal-GPU build of the runtime worker. The GPU worker binaries are produced by the repo's own build script (`scripts/build_mojo_binary.sh` with `MOJO_BUILD_METAL_GPU=1`); GPU packaging for the signed bundled runtime is on the roadmap.

## Supported models

The kernels implement the Qwen3 decoder family: grouped-query attention with RMSNorm on Q/K, tied embeddings, SiLU-gated FFN, RoPE, no attention bias. Any checkpoint matching those structural invariants and the alignment gates runs; anything else raises `UnsupportedArchitectureError` naming the exact mismatch, from `config.json` alone, before any conversion or GPU work:

| Gate           | Requirement                                                                                   |
| -------------- | --------------------------------------------------------------------------------------------- |
| GQA            | `num_attention_heads % num_key_value_heads == 0`                                              |
| Tile alignment | `hidden_size`, `intermediate_size`, and `num_attention_heads * head_dim` each divisible by 32 |
| Embeddings     | `tie_word_embeddings` true                                                                    |
| Bias           | no attention bias                                                                             |

One build serves one staging tier: the decode kernels' shared-memory staging is compiled to the model's own `stage_needed = max(hidden_size, num_attention_heads * head_dim, intermediate_size)`. Qwen3-0.6B-Base (3072) and Qwen3-1.7B (6144) are the two tiers with ready worker builds; a worker built for one tier raises a named error for a model that needs the other.

## Notes

* Chat template applies via `messages`, not raw text. Pass `messages=` (a list of chat dicts) and `decode()` formats them with the checkpoint's own `chat_template` — the SDK's dependency-free `algenta.chat_template`, byte-for-byte identical to `transformers`. A raw `prompt_text` string is tokenized as-is, with no template. Special strings in the text (like the end-of-text token) map to their own ids either way.
* Greedy by default. `decode()` with no sampling arguments is the fused argmax path. Pass `temperature > 0` to sample — see [Sampling](#sampling).
* One call, start to finish. One call prefills the prompt and decodes to `max_new_tokens` or end-of-text; there is no open session to step across calls.
* Weights load per call, with a cache. Each call loads the weights, but the first call on a converted checkpoint writes `weights_i8.cache` (the exact quantized device bytes, 8x smaller than the converted blob); later calls restore it and skip re-quantization entirely. See [The quantized load cache](#the-quantized-load-cache).

## Convert once, decode many times

`prepare_model_checkpoint()` reads the checkpoint's own `config.json` for the shape, handles single- or multi-shard safetensors, pre-transposes the GEMM weights the kernels expect, and writes the flat `manifest.json` + `weights.blob` pair the kernel reads. `decode()` converts for you into `<checkpoint>/.algenta-cache/` on first use and reuses that pair on every later call — the prompt travels separately from the converted weights, so one conversion serves every prompt. Pass `converted_checkpoint_dir` to point at a conversion you placed elsewhere yourself.

Within one worker process the loaded weights also stay resident: the first request loads them, and repeat requests adopt them directly (measured \~18x faster than the first load); switching to a different checkpoint loads that model instead.

```python
from algenta import MojoRuntime, load_tokenizer
from algenta.inference import decode, prepare_model_checkpoint

# 1. Tokenize with the checkpoint's OWN tokenizer (no external dependency) and convert once.
prompt_ids = load_tokenizer("./qwen3-1.7b").encode("The capital of France is")
converted = prepare_model_checkpoint(
    checkpoint_dir="./qwen3-1.7b",
    prompt_token_ids=prompt_ids,
    out_dir="./qwen3-1.7b-converted",
)

# 2. Decode through the runtime (reuse the conversion on every later call).
runtime = MojoRuntime(mode="local")
result = decode(
    runtime,
    checkpoint_dir="./qwen3-1.7b",
    prompt_text="The capital of France is",   # text in -- same checkpoint tokenizer
    max_new_tokens=16,
    converted_checkpoint_dir=converted,
)
print(result.generated_text)                   # text out
print(result.token_ids[result.n_prompt:])      # ids are always available too
print(result.prefill_tokens_per_sec, result.decode_tokens_per_sec)
```

To keep working in token ids, pass `prompt_token_ids=[...]` instead of `prompt_text` — `decode()` accepts exactly one of `prompt_text`, `prompt_token_ids`, or `messages`, and token-id calls never load the tokenizer. For chat, pass `messages=[{"role": "user", "content": ...}]` and the checkpoint's own `chat_template` formats the prompt before tokenization. `algenta.tokenizer.load_tokenizer()` and `algenta.chat_template.apply_chat_template()` are also exposed directly (`encode` / `decode`, and template rendering) if you want the ids or the formatted text without calling `decode()`.

## Long context

The engine sizes its own context as `max(512, n_prompt + max_new_tokens)`, auto-raised whenever a call needs more, up to a 65,536 ceiling. Verified: a 256-token prompt + 300 new tokens (556 total) and deep soaks of 2,048 new tokens on 0.6B / 1,024 on 1.7B, every token checked against the reference.

`max_ctx` on `decode()` is a caller-side guard, not an override: a value below `n_prompt + max_new_tokens` is refused before any work happens; a larger value is accepted but cannot change a one-shot run, because nothing inside a single call can consume context beyond what the prompt + generation uses.

## Sampling

The default call is greedy (fused argmax). Pass `temperature > 0` to draw every token — the first one included — from the model's own distribution, using the conventional chain: repetition / presence / frequency penalties → temperature → top-k → min-p → top-p → seeded weighted draw.

```python
result = decode(
    runtime,
    checkpoint_dir="./qwen3-1.7b",
    prompt_text="Once upon a time",
    max_new_tokens=64,
    temperature=0.8,
    top_k=40,
    top_p=0.95,
    min_p=0.0,
    repeat_penalty=1.1,
    seed=1234,                       # same seed -> byte-identical sequence
    converted_checkpoint_dir=converted,
)
```

Properties: the same prompt + parameters + seed reproduce the identical token sequence; `top_k=1` at any temperature reproduces the greedy sequence exactly; the sampled pass materializes the full logits row host-side, so it runs slower per token than greedy.

## Tool calling

Pass `tools` alongside `messages` and `decode()` renders the Qwen3 chat template's tools preamble (the same OpenAI-shaped `[{"type": "function", "function": {...}}]` list the HTTP API's `ChatCompletionsRequest.tools` accepts) before tokenization. `tools` is only valid alongside `messages` — the template is what renders it, and `prompt_text`/`prompt_token_ids` bypass the template entirely, so passing `tools` with either raises `InferenceError("invalid_arguments")`.

```python
from algenta.chat_template import split_tool_calls_from_generated_text

result = decode(
    runtime,
    checkpoint_dir="./qwen3-1.7b",
    messages=[{"role": "user", "content": "What is the weather in Paris?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location.",
            "parameters": {
                "type": "object",
                "properties": {"location": {"type": "string"}},
                "required": ["location"],
            },
        },
    }],
    max_new_tokens=64,
    converted_checkpoint_dir=converted,
)
content, tool_calls = split_tool_calls_from_generated_text(result.generated_text)
```

`algenta.chat_template.split_tool_calls_from_generated_text()` is the inverse operation: it pulls every `<tool_call>...</tool_call>` block the model emitted back out of `generated_text` into `{"name": ..., "arguments": {...}}` records, in order, and returns the remaining text (`None` for a tool-call-only turn) alongside them. Stated precisely, so nothing here overclaims:

* The tools preamble is a **hint** the model may act on — nothing in a prompt-level preamble can *force* a call. There is no `tool_choice` parameter on `decode()`; steering is out of scope for this SDK call (the HTTP API's `/v1/chat/completions` supports `tool_choice="auto"`/`"none"` on `algenta_local` for exactly this reason, and refuses a forced choice rather than pretending to honor it).
* A checkpoint that ships no chat template cannot render a tools preamble either — `messages` without a supported `chat_template` already refuses with `unsupported_chat_template`; `tools` changes nothing about that.
* A malformed or absent `<tool_call>` block degrades to plain content instead of raising — the parser reads model output, not a contract this module controls.

## The quantized load cache

The first `decode()` on a converted checkpoint writes `weights_i8.cache` next to the manifest — the exact post-quantization device bytes (int8 weights + per-row scales, \~1 byte/param against the flat blob's 8). Every later call on that checkpoint restores the cache instead of re-reading the full blob and re-running quantization. Measured on M4 Max:

| Model           | Converted blob | Cache    | Cold call | Cached call    |
| --------------- | -------------- | -------- | --------- | -------------- |
| Qwen3-0.6B-Base | 4.77 GB        | 0.598 GB | 4.87 s    | 1.57 s (3.10x) |
| Qwen3-1.7B      | 13.76 GB       | 1.724 GB | 10.98 s   | 5.70 s (1.93x) |

Cached and cold calls produce byte-identical greedy sequences — the cache holds the same bytes the GPU kernels read. The validity gate is magic + dims + SHA-256 of `manifest.json` + exact length: any mismatch falls back to the full path and rewrites a fresh cache. A cache written for one checkpoint never passes another's gate.

## Errors

Detected mismatches raise named errors:

| Error                                    | Meaning                                                                                                                                      |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `UnsupportedArchitectureError`           | The checkpoint's architecture is outside the supported family/gates (message names every problem found).                                     |
| `InferenceError("checkpoint_not_found")` | Missing `config.json`, safetensors shard, tensor, or converted pair.                                                                         |
| `InferenceError("invalid_arguments")`    | Empty/invalid prompt ids, `max_new_tokens < 1`, or `max_ctx` below the call's need.                                                          |
| `InferenceError("metaldispatch_...")`    | The engine refused by status: no Metal device, GPU build absent, context past the ceiling, and so on — the status name is always the `code`. |

## Next steps

{% content-ref url="/pages/xaVbhLD1TVrIAp719yLQ" %}
[LLM inference overview](/benchmarks/llm-inference-overview.md)
{% endcontent-ref %}

{% content-ref url="/pages/vcqdA4B3qUqGzPpMmov0" %}
[Run local LLM decode](/guides/run-qwen3-decode.md)
{% endcontent-ref %}

{% content-ref url="/pages/Yq9Ne2itDGfgjmW08TSP" %}
[Prefill (token input) throughput](/benchmarks/llm-prefill-throughput.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.algenta.ai/guides/run-qwen3-family-inference.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.