> For the complete documentation index, see [llms.txt](https://docs.algenta.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.algenta.ai/benchmarks/llm-tool-calling-uplift.md). # LLM tool-calling uplift gpt-5-mini answering 16 real questions against a 33,125-row CSV, with and without Algenta's tool-calling access to its own compiled engine — 16 of 16 correct with tools, 1 of 16 without. This page measures gpt-5-mini answering numeric questions about a dataset it has not seen, with and without tool access to Algenta's compiled engine over the real data: 16 of 16 correct with tools, 1 of 16 without. ## Results 16 objective, numerically-scored questions against a real 33,125-row CSV of purchase-quote line items, asked twice each: once bare (schema visible, no data, no tools) and once with Algenta's tool-calling surface available. Every question has a fixed ground-truth value and a tolerance, computed independently of Algenta before either arm ran (see [Method](#method)). | | bare | tools | | -------------------------- | ------------- | ---------------- | | Correct | 1 / 16 | 16 / 16 | | Real API cost | $0.0132 | $0.0531 | | API calls | 16 | 75 | | Prompt / completion tokens | 6,245 / 5,834 | 327,161 / 14,250 | Total real, usage-priced cost for both arms across all 16 questions: $0.066323. | id | question | ground truth | bare | tools | | --- | ------------------------------------------- | ------------------ | ----------------- | ------------------------------ | | q01 | 90th percentile of UNIT\_PRICE | 1905.48 | 1000 ✗ | 1905.48 ✓ | | q02 | 25th percentile of EXT\_PRICE | 1275.25 | 100.0 ✗ | 1275.25 ✓ | | q03 | median EXT\_PRICE | 6052.1 | 450 ✗ | 6052.1 ✓ | | q04 | mean UNIT\_PRICE | 779.182035 | 250 ✗ | 779.1820350636932 ✓ | | q05 | sample std dev (ddof=1) of UNIT\_PRICE | 2331.963152 | 1250 ✗ | 2331.9631517502244 ✓ | | q06 | min/max UNIT\_PRICE | 0.0 / 215000.0 | 0 / 250000 ✗ | 0.0 / 215000.0 ✓ | | q07 | count of STATUS == 'LOST' | 14673 | 3300 ✗ | 14673 ✓ | | q08 | sum EXT\_PRICE where CUST\_TYPE == 'OEM' | 169562999.02 | 9940000 ✗ | 169562999.02000016 ✓ | | q09 | average QUANTITY where STATUS == 'ORDER' | 261.723926 | 12.5 ✗ | 261.7239263803681 ✓ | | q10 | sum of QUANTITY | 3436556.0 | 492900 ✗ | 3436556.0 ✓ | | q11 | distinct CUSTOMER\_NO values | 2008 | 3300 ✗ | 2008 ✓ | | q12 | distinct non-null STATUS values | 12 | 12 ✓ | 12 ✓ | | q13 | fraction with AUTO\_QUOTED == 'Y' | 0.03963 | 0.28 ✗ | 0.0396304578949 ✓ | | q14 | CUST\_TYPE with highest average UNIT\_PRICE | MILITARY, 12515.95 | MILITARY, 12500 ✗ | MILITARY, 12515.949999999999 ✓ | | q15 | STATUS with highest average EXT\_PRICE | QTE, 60485.014002 | ORDER, 10000 ✗ | QTE, 60485.0140023753 ✓ | | q16 | Pearson r(UNIT\_PRICE, QUANTITY) | -0.071331 | -0.12 ✗ | -0.07133063138937178 ✓ | Tools-arm values match ground truth to six or more significant digits. The bare arm's only correct answer, q12, is a low-cardinality count; every arithmetic, percentile, sum, group-by and correlation question misses, several by more than an order of magnitude. ## Quick start The two tools that carry almost all of the uplift — search the capability catalog, then execute one kernel — are two calls on the public SDK, with no separate server and no API key beyond your own model provider's: ```python import json from algenta import MojoRuntime from openai import OpenAI runtime = MojoRuntime(mode="local") catalog = runtime.capabilities() client = OpenAI() tools = [ {"type": "function", "function": { "name": "search_capabilities", "description": "Find a compiled Algenta kernel by function name (substring match).", "parameters": {"type": "object", "properties": { "function": {"type": "string"}}, "required": ["function"]}, }}, {"type": "function", "function": { "name": "median", "description": "The median of a list of real numbers, via a compiled Algenta kernel.", "parameters": {"type": "object", "properties": { "values": {"type": "array", "items": {"type": "number"}}}, "required": ["values"]}, }}, ] def dispatch(name: str, args: dict): if name == "search_capabilities": return [f"{m.module}.{m.function}" for m in catalog.search(function=args["function"])] if name == "median": # The kernel's own signature is `median(v: List[Float64], n: Int)` -- n is # computed HERE, never left for the model to supply. A wrong element count # is the single most dangerous kind of bad tool-call argument this SDK can # receive: the kernel trusts the count it is given, and a count that does # not match the real list length fires an uncatchable bounds assertion that # kills the whole worker process, not a catchable exception. values = args["values"] return runtime.execute("stats", "median", [values, len(values)]) return {"error": f"unknown tool {name}"} messages = [{"role": "user", "content": "What's the median of [10, 12, 23, 23, 16, 23, 21, 16]?"}] resp = client.chat.completions.create(model="gpt-5-mini", messages=messages, tools=tools) msg = resp.choices[0].message messages.append(msg.model_dump(exclude_none=True)) for call in msg.tool_calls or []: result = dispatch(call.function.name, json.loads(call.function.arguments)) messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)}) final = client.chat.completions.create(model="gpt-5-mini", messages=messages, tools=tools) print(final.choices[0].message.content) ``` This is the same shape the benchmark below uses, trimmed to one kernel: search finds it, one tool call runs it, and the model reports the returned number. The element count `n` is computed in `dispatch`, not supplied by the model: most kernels in this catalog take a value list plus a separate count. The benchmark's full six-tool version — `execute` generalized to any kernel the catalog can find, plus `explain`, `get_column`/`get_columns`, `group_by_aggregate` for dataset-scale questions, result truncation, and the general-purpose version of the same count guard applied automatically across every declared count relation in the catalog (`_apply_count_relations`) — is the reusable module at `benchmarks/algenta_llm_tools/tools.py`; see [Reproduce](#reproduce). ## Method * Dataset. `test_files/Processed_Quotes.csv`, 33,125 real purchase-quote line items, reduced to the 7 columns this benchmark reads and anonymized (customer identifiers remapped to synthetic ids; every other identifying field dropped) before this page's questions or tools touch it. Loaded through Algenta's own local file connector — `Runtime(mode="local").connect(csv_path)` — the same CSV parser and connector path any `pip install algenta` caller uses. No cloud call and no sidecar process are involved; the connector and the kernels both run in-process. * Model. `gpt-5-mini`, called live via the OpenAI API, both arms, all 16 questions, real usage-based cost taken from each response's own token counts, including OpenAI's own prompt caching (a stable, growing conversation prefix qualifies automatically; this run's tools arm hit cache on 77.7% of its prompt tokens). * Bare arm. The question plus the dataset's schema (column names and types, no row data, no computed statistics) in the prompt. No tools. One API call per question. * Tools arm. The same question, with six tools exposed via OpenAI function calling; every one of them dispatches into the compiled Mojo engine or reads the real CSV directly: * `list_capabilities` — search the capability catalog for a candidate kernel. * `explain` — get a kernel's exact signature and declared properties. * `get_column` / `get_columns` — read one or more real columns from the connected dataset, data access only (a small preview for sanity-checking; the full values stay reachable by id). * `execute` — dispatch a named kernel (for example `stats.compute_std_from_values`, `stats.quantile`, `stat_tests.pearson_r_strict`) against real, connected data and return its output. * `group_by_aggregate` — group real rows by a column and aggregate another, in one kernel dispatch. * Scoring. Every ground-truth value was computed independently with pandas directly against the CSV before either arm ran, and cross-checked for several questions with a second, unrelated CSV parser (Python's `csv.DictReader`) to rule out a shared parsing bug. Each question carries a fixed tolerance (exact match for counts and category labels, absolute tolerance for floating-point statistics) applied identically to both arms. ## Scope * One dataset, one model, one 16-question set: results cover this CSV, `gpt-5-mini`, and these 16 numeric questions, run live. * Numeric, single-answer questions only: all 16 questions score a single number, pair of numbers, or category label against a fixed ground truth. Open-ended or multi-step reasoning questions are out of scope for this page. * Cost reflects this workload, not a ceiling: $0.07 for 16 questions across both arms on this model's pricing; heavier questions (multi-category group-bys) cost more in tool calls than simple lookups, which is visible in the per-question API-call counts in the source results. * Model behavior varies run to run: earlier runs of this same benchmark measured 15/16 and occasional turn-budget exhaustion on individual questions before two gaps were closed — a kernel call that could receive a hand-guessed input, and no direct guidance toward the correct min/max path. Both are documented, with full before/after numbers, in `benchmarks/algenta_llm_tools/eval/README.md`. ## Reproduce The benchmark's own harness doubles as usage documentation for the same six tools: ``` python3 -m benchmarks.algenta_llm_tools.eval.run_benchmark ``` The question set, fixed ground truth and scoring rules are in `benchmarks/algenta_llm_tools/eval/quotes_question_set.json`; how each ground-truth value was derived (including the independent cross-check and the dataset's anonymization) is in `benchmarks/algenta_llm_tools/eval/README.md`. The tool implementations themselves — `list_capabilities`, `execute`, `explain`, `get_column`, `get_columns`, `group_by_aggregate` — are in `benchmarks/algenta_llm_tools/tools.py`, built entirely on the public Algenta Python SDK (`algenta.MojoRuntime`, `algenta.Runtime`), the same surface any caller has access to. ## Related * [Benchmarks](/benchmarks/overview.md) — methodology and what is not published. * [Dataframe group-by](/benchmarks/dataframe-groupby.md) — Algenta's own group-by engine measured directly, without an LLM in the loop. * [Declared capabilities](/engine-runtime/capabilities.md) — the catalog `list_capabilities` and `explain` search. * [Route & execute capabilities](/guides/capabilities.md) — calling the same kernels directly, without a model choosing the calls. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.algenta.ai/benchmarks/llm-tool-calling-uplift.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.