Model-agnostic LLM inference

Run any supported checkpoint on Apple Silicon through Algenta's model-agnostic SDK path — supported shapes, long context, and the quantized load cache. Currently verified on Qwen3 models.

algenta.inference.decode() runs any supported checkpoint through the model-agnostic Metal GPU kernel path: one call, prompt token ids in, generated token ids out, on Apple Silicon. It uses the same kernels as the local LLM decode guide and accepts any model whose architecture the kernels support. Verified end to end on Qwen3-0.6B-Base and Qwen3-1.7B with identical greedy output (first token matches an independent transformers CPU reference on both).

Requirements

• Apple Silicon (M-series) Mac with Metal. There is no CPU fallback and no support for other GPU vendors.

• macOS 26 or newer and Python 3.12 or newer. pip install algenta installs the native runtime with Metal on macOS 26 and newer; on macOS 13 to 15 it installs the CPU-only runtime, which cannot run this Metal path.

• A local Hugging Face checkpoint snapshot (config.json + one or more model*.safetensors, single- or multi-shard), downloaded separately — for example Qwen/Qwen3-0.6B-Base or Qwen/Qwen3-1.7B (both Apache-2.0, ungated).

• numpy, to convert the checkpoint's published BF16 weights: pip install "algenta[benchmark]".

• No external tokenizer needed. The SDK implements the checkpoint's own tokenizer.json (Qwen3-family byte-level BPE, byte-for-byte validated against transformers) — pass prompt_text straight to decode(). Token ids are also accepted.

• A Metal-GPU build of the runtime worker. The GPU worker binaries are produced by the repo's own build script (scripts/build_mojo_binary.sh with MOJO_BUILD_METAL_GPU=1); GPU packaging for the signed bundled runtime is on the roadmap.

Supported models

The kernels implement the Qwen3 decoder family: grouped-query attention with RMSNorm on Q/K, tied embeddings, SiLU-gated FFN, RoPE, no attention bias. Any checkpoint matching those structural invariants and the alignment gates runs; anything else raises UnsupportedArchitectureError naming the exact mismatch, from config.json alone, before any conversion or GPU work:

Gate	Requirement
GQA	num_attention_heads % num_key_value_heads == 0
Tile alignment	hidden_size, intermediate_size, and num_attention_heads * head_dim each divisible by 32
Embeddings	tie_word_embeddings true
Bias	no attention bias

One build serves one staging tier: the decode kernels' shared-memory staging is compiled to the model's own stage_needed = max(hidden_size, num_attention_heads * head_dim, intermediate_size). Qwen3-0.6B-Base (3072) and Qwen3-1.7B (6144) are the two tiers with ready worker builds; a worker built for one tier raises a named error for a model that needs the other.

Notes

• Chat template applies via messages, not raw text. Pass messages= (a list of chat dicts) and decode() formats them with the checkpoint's own chat_template — the SDK's dependency-free algenta.chat_template, byte-for-byte identical to transformers. A raw prompt_text string is tokenized as-is, with no template. Special strings in the text (like the end-of-text token) map to their own ids either way.

• Greedy by default. decode() with no sampling arguments is the fused argmax path. Pass temperature > 0 to sample — see Sampling.

• One call, start to finish. One call prefills the prompt and decodes to max_new_tokens or end-of-text; there is no open session to step across calls.

• Weights load per call, with a cache. Each call loads the weights, but the first call on a converted checkpoint writes weights_i8.cache (the exact quantized device bytes, 8x smaller than the converted blob); later calls restore it and skip re-quantization entirely. See The quantized load cache.

Convert once, decode many times

prepare_model_checkpoint() reads the checkpoint's own config.json for the shape, handles single- or multi-shard safetensors, pre-transposes the GEMM weights the kernels expect, and writes the flat manifest.json + weights.blob pair the kernel reads. decode() converts for you into <checkpoint>/.algenta-cache/ on first use and reuses that pair on every later call — the prompt travels separately from the converted weights, so one conversion serves every prompt. Pass converted_checkpoint_dir to point at a conversion you placed elsewhere yourself.

Within one worker process the loaded weights also stay resident: the first request loads them, and repeat requests adopt them directly (measured ~18x faster than the first load); switching to a different checkpoint loads that model instead.

from algenta import MojoRuntime, load_tokenizer
from algenta.inference import decode, prepare_model_checkpoint

# 1. Tokenize with the checkpoint's OWN tokenizer (no external dependency) and convert once.
prompt_ids = load_tokenizer("./qwen3-1.7b").encode("The capital of France is")
converted = prepare_model_checkpoint(
    checkpoint_dir="./qwen3-1.7b",
    prompt_token_ids=prompt_ids,
    out_dir="./qwen3-1.7b-converted",
)

# 2. Decode through the runtime (reuse the conversion on every later call).
runtime = MojoRuntime(mode="local")
result = decode(
    runtime,
    checkpoint_dir="./qwen3-1.7b",
    prompt_text="The capital of France is",   # text in -- same checkpoint tokenizer
    max_new_tokens=16,
    converted_checkpoint_dir=converted,
)
print(result.generated_text)                   # text out
print(result.token_ids[result.n_prompt:])      # ids are always available too
print(result.prefill_tokens_per_sec, result.decode_tokens_per_sec)

To keep working in token ids, pass prompt_token_ids=[...] instead of prompt_text — decode() accepts exactly one of prompt_text, prompt_token_ids, or messages, and token-id calls never load the tokenizer. For chat, pass messages=[{"role": "user", "content": ...}] and the checkpoint's own chat_template formats the prompt before tokenization. algenta.tokenizer.load_tokenizer() and algenta.chat_template.apply_chat_template() are also exposed directly (encode / decode, and template rendering) if you want the ids or the formatted text without calling decode().

Long context

The engine sizes its own context as max(512, n_prompt + max_new_tokens), auto-raised whenever a call needs more, up to a 65,536 ceiling. Verified: a 256-token prompt + 300 new tokens (556 total) and deep soaks of 2,048 new tokens on 0.6B / 1,024 on 1.7B, every token checked against the reference.

max_ctx on decode() is a caller-side guard, not an override: a value below n_prompt + max_new_tokens is refused before any work happens; a larger value is accepted but cannot change a one-shot run, because nothing inside a single call can consume context beyond what the prompt + generation uses.

Sampling

The default call is greedy (fused argmax). Pass temperature > 0 to draw every token — the first one included — from the model's own distribution, using the conventional chain: repetition / presence / frequency penalties → temperature → top-k → min-p → top-p → seeded weighted draw.

result = decode(
    runtime,
    checkpoint_dir="./qwen3-1.7b",
    prompt_text="Once upon a time",
    max_new_tokens=64,
    temperature=0.8,
    top_k=40,
    top_p=0.95,
    min_p=0.0,
    repeat_penalty=1.1,
    seed=1234,                       # same seed -> byte-identical sequence
    converted_checkpoint_dir=converted,
)

Properties: the same prompt + parameters + seed reproduce the identical token sequence; top_k=1 at any temperature reproduces the greedy sequence exactly; the sampled pass materializes the full logits row host-side, so it runs slower per token than greedy.

Tool calling

Pass tools alongside messages and decode() renders the Qwen3 chat template's tools preamble (the same OpenAI-shaped [{"type": "function", "function": {...}}] list the HTTP API's ChatCompletionsRequest.tools accepts) before tokenization. tools is only valid alongside messages — the template is what renders it, and prompt_text/prompt_token_ids bypass the template entirely, so passing tools with either raises InferenceError("invalid_arguments").

from algenta.chat_template import split_tool_calls_from_generated_text

result = decode(
    runtime,
    checkpoint_dir="./qwen3-1.7b",
    messages=[{"role": "user", "content": "What is the weather in Paris?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location.",
            "parameters": {
                "type": "object",
                "properties": {"location": {"type": "string"}},
                "required": ["location"],
            },
        },
    }],
    max_new_tokens=64,
    converted_checkpoint_dir=converted,
)
content, tool_calls = split_tool_calls_from_generated_text(result.generated_text)

algenta.chat_template.split_tool_calls_from_generated_text() is the inverse operation: it pulls every <tool_call>...</tool_call> block the model emitted back out of generated_text into {"name": ..., "arguments": {...}} records, in order, and returns the remaining text (None for a tool-call-only turn) alongside them. Stated precisely, so nothing here overclaims:

• The tools preamble is a hint the model may act on — nothing in a prompt-level preamble can force a call. There is no tool_choice parameter on decode(); steering is out of scope for this SDK call (the HTTP API's /v1/chat/completions supports tool_choice="auto"/"none" on algenta_local for exactly this reason, and refuses a forced choice rather than pretending to honor it).

• A checkpoint that ships no chat template cannot render a tools preamble either — messages without a supported chat_template already refuses with unsupported_chat_template; tools changes nothing about that.

• A malformed or absent <tool_call> block degrades to plain content instead of raising — the parser reads model output, not a contract this module controls.

The quantized load cache

The first decode() on a converted checkpoint writes weights_i8.cache next to the manifest — the exact post-quantization device bytes (int8 weights + per-row scales, ~1 byte/param against the flat blob's 8). Every later call on that checkpoint restores the cache instead of re-reading the full blob and re-running quantization. Measured on M4 Max:

Model	Converted blob	Cache	Cold call	Cached call
Qwen3-0.6B-Base	4.77 GB	0.598 GB	4.87 s	1.57 s (3.10x)
Qwen3-1.7B	13.76 GB	1.724 GB	10.98 s	5.70 s (1.93x)

Cached and cold calls produce byte-identical greedy sequences — the cache holds the same bytes the GPU kernels read. The validity gate is magic + dims + SHA-256 of manifest.json + exact length: any mismatch falls back to the full path and rewrites a fresh cache. A cache written for one checkpoint never passes another's gate.

Errors

Detected mismatches raise named errors:

Error	Meaning
UnsupportedArchitectureError	The checkpoint's architecture is outside the supported family/gates (message names every problem found).
InferenceError("checkpoint_not_found")	Missing config.json, safetensors shard, tensor, or converted pair.
InferenceError("invalid_arguments")	Empty/invalid prompt ids, max_new_tokens < 1, or max_ctx below the call's need.
InferenceError("metaldispatch_...")	The engine refused by status: no Metal device, GPU build absent, context past the ceiling, and so on — the status name is always the code.

Next steps

LLM inference overviewRun local LLM decodePrefill (token input) throughput