LLM tool-calling uplift

gpt-5-mini answering 16 real questions against a 33,125-row CSV, with and without Algenta's tool-calling access to its own compiled engine — 16 of 16 correct with tools, 1 of 16 without.

This page measures gpt-5-mini answering numeric questions about a dataset it has not seen, with and without tool access to Algenta's compiled engine over the real data: 16 of 16 correct with tools, 1 of 16 without.

Results

16 objective, numerically-scored questions against a real 33,125-row CSV of purchase-quote line items, asked twice each: once bare (schema visible, no data, no tools) and once with Algenta's tool-calling surface available. Every question has a fixed ground-truth value and a tolerance, computed independently of Algenta before either arm ran (see Method).

	bare	tools
Correct	1 / 16	16 / 16
Real API cost	$0.0132	$0.0531
API calls	16	75
Prompt / completion tokens	6,245 / 5,834	327,161 / 14,250

Total real, usage-priced cost for both arms across all 16 questions: $0.066323.

id	question	ground truth	bare	tools
q01	90th percentile of UNIT_PRICE	1905.48	1000 ✗	1905.48 ✓
q02	25th percentile of EXT_PRICE	1275.25	100.0 ✗	1275.25 ✓
q03	median EXT_PRICE	6052.1	450 ✗	6052.1 ✓
q04	mean UNIT_PRICE	779.182035	250 ✗	779.1820350636932 ✓
q05	sample std dev (ddof=1) of UNIT_PRICE	2331.963152	1250 ✗	2331.9631517502244 ✓
q06	min/max UNIT_PRICE	0.0 / 215000.0	0 / 250000 ✗	0.0 / 215000.0 ✓
q07	count of STATUS == 'LOST'	14673	3300 ✗	14673 ✓
q08	sum EXT_PRICE where CUST_TYPE == 'OEM'	169562999.02	9940000 ✗	169562999.02000016 ✓
q09	average QUANTITY where STATUS == 'ORDER'	261.723926	12.5 ✗	261.7239263803681 ✓
q10	sum of QUANTITY	3436556.0	492900 ✗	3436556.0 ✓
q11	distinct CUSTOMER_NO values	2008	3300 ✗	2008 ✓
q12	distinct non-null STATUS values	12	12 ✓	12 ✓
q13	fraction with AUTO_QUOTED == 'Y'	0.03963	0.28 ✗	0.0396304578949 ✓
q14	CUST_TYPE with highest average UNIT_PRICE	MILITARY, 12515.95	MILITARY, 12500 ✗	MILITARY, 12515.949999999999 ✓
q15	STATUS with highest average EXT_PRICE	QTE, 60485.014002	ORDER, 10000 ✗	QTE, 60485.0140023753 ✓
q16	Pearson r(UNIT_PRICE, QUANTITY)	-0.071331	-0.12 ✗	-0.07133063138937178 ✓

Tools-arm values match ground truth to six or more significant digits. The bare arm's only correct answer, q12, is a low-cardinality count; every arithmetic, percentile, sum, group-by and correlation question misses, several by more than an order of magnitude.

Quick start

The two tools that carry almost all of the uplift — search the capability catalog, then execute one kernel — are two calls on the public SDK, with no separate server and no API key beyond your own model provider's:

import json
from algenta import MojoRuntime
from openai import OpenAI

runtime = MojoRuntime(mode="local")
catalog = runtime.capabilities()
client = OpenAI()

tools = [
    {"type": "function", "function": {
        "name": "search_capabilities",
        "description": "Find a compiled Algenta kernel by function name (substring match).",
        "parameters": {"type": "object", "properties": {
            "function": {"type": "string"}}, "required": ["function"]},
    }},
    {"type": "function", "function": {
        "name": "median",
        "description": "The median of a list of real numbers, via a compiled Algenta kernel.",
        "parameters": {"type": "object", "properties": {
            "values": {"type": "array", "items": {"type": "number"}}},
            "required": ["values"]},
    }},
]

def dispatch(name: str, args: dict):
    if name == "search_capabilities":
        return [f"{m.module}.{m.function}" for m in catalog.search(function=args["function"])]
    if name == "median":
        # The kernel's own signature is `median(v: List[Float64], n: Int)` -- n is
        # computed HERE, never left for the model to supply. A wrong element count
        # is the single most dangerous kind of bad tool-call argument this SDK can
        # receive: the kernel trusts the count it is given, and a count that does
        # not match the real list length fires an uncatchable bounds assertion that
        # kills the whole worker process, not a catchable exception.
        values = args["values"]
        return runtime.execute("stats", "median", [values, len(values)])
    return {"error": f"unknown tool {name}"}

messages = [{"role": "user", "content": "What's the median of [10, 12, 23, 23, 16, 23, 21, 16]?"}]
resp = client.chat.completions.create(model="gpt-5-mini", messages=messages, tools=tools)
msg = resp.choices[0].message
messages.append(msg.model_dump(exclude_none=True))
for call in msg.tool_calls or []:
    result = dispatch(call.function.name, json.loads(call.function.arguments))
    messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)})

final = client.chat.completions.create(model="gpt-5-mini", messages=messages, tools=tools)
print(final.choices[0].message.content)

This is the same shape the benchmark below uses, trimmed to one kernel: search finds it, one tool call runs it, and the model reports the returned number. The element count n is computed in dispatch, not supplied by the model: most kernels in this catalog take a value list plus a separate count. The benchmark's full six-tool version — execute generalized to any kernel the catalog can find, plus explain, get_column/get_columns, group_by_aggregate for dataset-scale questions, result truncation, and the general-purpose version of the same count guard applied automatically across every declared count relation in the catalog (_apply_count_relations) — is the reusable module at benchmarks/algenta_llm_tools/tools.py; see Reproduce.

Method

• Dataset. test_files/Processed_Quotes.csv, 33,125 real purchase-quote line items, reduced to the 7 columns this benchmark reads and anonymized (customer identifiers remapped to synthetic ids; every other identifying field dropped) before this page's questions or tools touch it. Loaded through Algenta's own local file connector — Runtime(mode="local").connect(csv_path) — the same CSV parser and connector path any pip install algenta caller uses. No cloud call and no sidecar process are involved; the connector and the kernels both run in-process.
• Model. gpt-5-mini, called live via the OpenAI API, both arms, all 16 questions, real usage-based cost taken from each response's own token counts, including OpenAI's own prompt caching (a stable, growing conversation prefix qualifies automatically; this run's tools arm hit cache on 77.7% of its prompt tokens).
• Bare arm. The question plus the dataset's schema (column names and types, no row data, no computed statistics) in the prompt. No tools. One API call per question.
• Tools arm. The same question, with six tools exposed via OpenAI function calling; every one of them dispatches into the compiled Mojo engine or reads the real CSV directly:

  • list_capabilities — search the capability catalog for a candidate kernel.
  • explain — get a kernel's exact signature and declared properties.
  • get_column / get_columns — read one or more real columns from the connected dataset, data access only (a small preview for sanity-checking; the full values stay reachable by id).
  • execute — dispatch a named kernel (for example stats.compute_std_from_values, stats.quantile, stat_tests.pearson_r_strict) against real, connected data and return its output.
  • group_by_aggregate — group real rows by a column and aggregate another, in one kernel dispatch.
• Scoring. Every ground-truth value was computed independently with pandas directly against the CSV before either arm ran, and cross-checked for several questions with a second, unrelated CSV parser (Python's csv.DictReader) to rule out a shared parsing bug. Each question carries a fixed tolerance (exact match for counts and category labels, absolute tolerance for floating-point statistics) applied identically to both arms.

Scope

• One dataset, one model, one 16-question set: results cover this CSV, gpt-5-mini, and these 16 numeric questions, run live.
• Numeric, single-answer questions only: all 16 questions score a single number, pair of numbers, or category label against a fixed ground truth. Open-ended or multi-step reasoning questions are out of scope for this page.
• Cost reflects this workload, not a ceiling: $0.07 for 16 questions across both arms on this model's pricing; heavier questions (multi-category group-bys) cost more in tool calls than simple lookups, which is visible in the per-question API-call counts in the source results.
• Model behavior varies run to run: earlier runs of this same benchmark measured 15/16 and occasional turn-budget exhaustion on individual questions before two gaps were closed — a kernel call that could receive a hand-guessed input, and no direct guidance toward the correct min/max path. Both are documented, with full before/after numbers, in benchmarks/algenta_llm_tools/eval/README.md.

Reproduce

The benchmark's own harness doubles as usage documentation for the same six tools:

python3 -m benchmarks.algenta_llm_tools.eval.run_benchmark

The question set, fixed ground truth and scoring rules are in benchmarks/algenta_llm_tools/eval/quotes_question_set.json; how each ground-truth value was derived (including the independent cross-check and the dataset's anonymization) is in benchmarks/algenta_llm_tools/eval/README.md. The tool implementations themselves — list_capabilities, execute, explain, get_column, get_columns, group_by_aggregate — are in benchmarks/algenta_llm_tools/tools.py, built entirely on the public Algenta Python SDK (algenta.MojoRuntime, algenta.Runtime), the same surface any caller has access to.

Related

• Benchmarks — methodology and what is not published.
• Dataframe group-by — Algenta's own group-by engine measured directly, without an LLM in the loop.
• Declared capabilities — the catalog list_capabilities and explain search.
• Route & execute capabilities — calling the same kernels directly, without a model choosing the calls.