Run local LLM decode

Install Algenta, get a supported checkpoint, and run greedy decode on your own Apple Silicon GPU — requirements, limitations, and a complete working example. Currently tested on Qwen3 models.

This guide installs Algenta, downloads the Qwen3-0.6B-Base checkpoint, tokenizes a prompt, and runs a greedy decode through the compiled int8 Metal GPU kernel path on an Apple Silicon machine. Everything runs inside the runtime that ships with the algenta package; there is nothing separate to start or manage.

Scope: Qwen3-0.6B-Base on Apple Silicon with Metal. Read Requirements and Limitations before starting.

Requirements

• Apple Silicon (M-series) Mac with Metal. There is no CPU fallback and no support for other GPU vendors. The LLM serving path runs on Metal only. NVIDIA CUDA is run-verified for the portable GPU kernel layer on T4, L4 and A100 hardware, and a CUDA-native LLM session is benchmarked on A100/L4, but the shipped serving path here is Metal-only; AMD ROCm is not verified. See GPU support.
• macOS 26 or newer and Python 3.12 or newer. pip install algenta installs the native runtime with Metal on macOS 26 and newer; on macOS 13 to 15 it installs the CPU-only runtime, which cannot run this Metal path.
• The checkpoint this kernel path was compiled for: Qwen/Qwen3-0.6B-Base (Apache-2.0, ungated — no Hugging Face account or token needed). No other Qwen3 size and no other model family.
• numpy, to convert the checkpoint's published BF16 weights. Install it with the benchmark extra: pip install "algenta[benchmark]".
• A tokenizer for the checkpoint. This capability takes token ids in and token ids out, not raw text — the example below uses transformers' AutoTokenizer against the same checkpoint directory.

Limitations

• One model, one exact shape. Qwen3-0.6B-Base's exact architecture only (hidden_size=1024, 28 layers, 16 query heads, 8 key/value heads, head_dim=128, vocab_size=151936, tied embeddings). A checkpoint of any other shape raises UnsupportedCheckpointArchitectureError naming every mismatched field before any conversion or GPU work starts. Other Qwen3 sizes (for example Qwen3-1.7B) run through the model-agnostic path — see Model-agnostic LLM inference.
• One hardware family. Apple Silicon with Metal. A host without a Metal accelerator raises a named error. CUDA runs the portable kernel layer and has a benchmarked native LLM session, but not this shipped session path; ROCm is not verified — GPU support.
• Greedy decode only. Every generated token is the argmax at that step. There is no sampling, no temperature, no top-p/top-k, and no beam search.
• One call, start to finish. A single call prefills the prompt and decodes to max_new_tokens or an end-of-text token. A step-by-step session is available through the Hub kernel; see Kernels on Hugging Face.
• Every call converts the checkpoint first. The checkpoint's BF16 weights are converted into the kernel's own flat format on every call by default, which takes time before decoding starts.
• Token ids in, not raw text. Tokenize with the checkpoint's own tokenizer — either algenta.load_tokenizer() (built into the SDK, no dependencies) or transformers' AutoTokenizer; the example below shows the latter.

Install Algenta

1

Install the package

The algenta package installs the matching compiled native runtime automatically — there is no separate Mojo/Metal toolchain to install.

python3 -m venv .venv
source .venv/bin/activate
pip install "algenta[benchmark]"

2

Verify the runtime is healthy

algenta runtime doctor

Expect a report that ends in READY, with the bundled native runtime verified and no fallback. Its runtime wheel line names the installed wheel; this guide needs the macosx_26_0_arm64 one.

Get the checkpoint

Algenta does not download the checkpoint; get a copy of Qwen/Qwen3-0.6B-Base onto disk first:

pip install huggingface_hub
python3 -c "
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id='Qwen/Qwen3-0.6B-Base',
    local_dir='./qwen3-0.6b-base',
    allow_patterns=['*.json', '*.safetensors', 'tokenizer*', 'vocab*', 'merges*'],
)
"

Run a decode

1

Tokenize your prompt

qwen3_i8_allsimd7_decode() takes a list of integer token ids, not a string. Tokenize with the checkpoint's own tokenizer:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("./qwen3-0.6b-base")
prompt_token_ids = tokenizer("The capital of France is", return_tensors=None)["input_ids"]

2

Call the decode function

qwen3_i8_allsimd7_decode() takes a MojoRuntime, the checkpoint directory, your prompt token ids, and how many tokens to generate. It converts the checkpoint automatically the first time you call it.

from algenta import MojoRuntime, qwen3_i8_allsimd7_decode

runtime = MojoRuntime(mode="local")

result = qwen3_i8_allsimd7_decode(
    runtime,
    checkpoint_dir="./qwen3-0.6b-base",
    prompt_token_ids=prompt_token_ids,
    max_new_tokens=16,
)

3

Read the result

result is a Qwen3DecodeResult: token_ids is your prompt followed by every generated token, in order.

print(result.token_ids)
print(result.decode_tokens_per_sec)

generated_only = result.token_ids[result.n_prompt :]
print(tokenizer.decode(generated_only))

Expected result: result.token_ids is a non-empty list of integers starting with your prompt's own token ids, followed by up to max_new_tokens generated ids (fewer if the model produced an end-of-text token first). Decoding generated_only with the tokenizer reads as ordinary continuation text, and result.decode_tokens_per_sec is a positive number.

Troubleshooting

Detected mismatches raise named errors:

What you did	What you get
Pointed checkpoint_dir at a checkpoint that is not Qwen3-0.6B-Base's exact shape	UnsupportedCheckpointArchitectureError, naming every mismatched field, before any conversion or GPU work starts.
Pointed checkpoint_dir at a path with no config.json / model.safetensors	Qwen3DecodeError with code checkpoint_not_found.
Ran on a host, or a runtime build, without working Metal GPU support	Qwen3DecodeError with code metaldispatch_shader_toolchain_absent.

Every one of these is a Qwen3DecodeError (or its subclass UnsupportedCheckpointArchitectureError), and every instance carries a stable .code string:

from algenta import Qwen3DecodeError

try:
    result = qwen3_i8_allsimd7_decode(
        runtime,
        checkpoint_dir="./qwen3-0.6b-base",
        prompt_token_ids=prompt_token_ids,
        max_new_tokens=16,
    )
except Qwen3DecodeError as error:
    print(error.code, str(error))

Next steps

Model-agnostic LLM inferenceMetal LLM decodeDecode throughputInstall