koboldcpp/examples/llama-eval

llama-eval

Simple evaluation tool for llama.cpp with support for multiple datasets.

For a full description, usage examples, and sample results, see:

• PR 21152

Quick start

# Single server
python3 llama-eval.py \
  --server http://localhost:8033 \
  --model my-model \
  --dataset gsm8k --n_cases 100 \
  --grader-type regex --threads 32

# Multiple servers (comma-separated URLs and thread counts)
python3 llama-eval.py \
  --server http://server1:8033,http://server2:8033 \
  --server-name server1,server2 \
  --threads 16,16 \
  --dataset aime2025 --n_cases 240 \
  --grader-type regex

Authentication

A server started with --api-key (or any OpenAI-compatible endpoint behind a token) needs one here too. --api-key is sent as Authorization: Bearer on every eval request; it falls back to $LLAMA_API_KEY, which keeps the secret out of your shell history and off the process list.

# One key for every server
python3 llama-eval.py --server http://localhost:8033 --api-key sk-... ...

# From the environment
export LLAMA_API_KEY=sk-...
python3 llama-eval.py --server http://localhost:8033 ...

# One key per server, in --server order
python3 llama-eval.py \
  --server http://server1:8033,http://server2:8033 \
  --api-key sk-one,sk-two --threads 16,16 ...

The LLM grader authenticates too. It reuses the key of the server it runs on, so --grader-type llm without --grader-server needs nothing extra; a grader on a separate endpoint takes --grader-api-key.

Endpoints on an opencode host additionally get an x-opencode-session header, generated fresh for every task. An agentic episode sends the same id on all of its turns, so opencode sees one session per task instead of one per request; other servers never see the header.

Agentic suite

--dataset agentic is a different shape from the other code suites. It runs SACB (Simple Agent Coding Benchmark), hosted at ilintar/SACB -- 60 tasks over three hand-written repositories, in Python and TypeScript. Rather than answering one prompt, the model is given a small but real repository, a deliberately narrow tool set, and a bug report, and drives its own multi-turn conversation until it declares itself finished. It is then graded by running a test suite it was never allowed to see.

python3 llama-eval.py \
  --server http://localhost:8033 --model my-model \
  --dataset agentic --threads 1 --temperature 0

The tools

tool	purpose
list_files	enumerate the tree
read_file	read, with 1-based line numbers and optional line-range narrowing
search	regular-expression search across the repository
edit_lines	replace an inclusive line range
edit_replace	replace an exact snippet, which must be unique
write_file	write a whole file
lint	run the language's linter and type checker
finish	declare the change complete

Three constraints are deliberate:

There is no shell and no way to run the tests. The only feedback channel is lint. A syntax or type error can be driven out mechanically; a logic error has to be reasoned about. A harness that lets an agent iterate against the test suite measures something closer to search than to understanding.

There are two edit tools, one line-addressed and one content-addressed. Which one a model reaches for, and whether it keeps line numbers straight after its own earlier edit, is itself a signal.

The tests never touch the working tree. They are held outside it and copied into a throwaway copy only after the agent has stopped, so they can be neither read nor edited. Every tool path is resolved and confinement-checked against the repository root, so a symlink planted inside the tree cannot reach outside it.

Scoring

A task is resolved when every test that was failing now passes and nothing that was already passing broke. Both sets are derived by running the tests twice when the corpus is built -- once against the defective tree and once against the reference fix -- never hand-written, so a task whose defect no test exercises is rejected rather than shipped.

Because many tasks contain several independent defects, the fraction of target tests passed is reported alongside, which separates "fixed two of the three faults" from "changed nothing". Process metrics are reported too: turns, tool calls, rejected calls, edits, and the peak conversation size.

Dependencies

Nothing here is installed unless you select this suite, and then only for the language you actually run. --agentic-lang python provisions ruff and mypy; --agentic-lang typescript provisions the TypeScript compiler and uses node's own test runner. Neither touches the other, and no other suite touches either.

Useful flags

flag	meaning
--agentic-lang	restrict to one language, and provision only its toolchain
--agentic-max-turns	give up after this many assistant turns (default 50)
--agentic-max-tokens	cap generation per turn (default: no cap); bounds a model that falls into a repetition loop, but also cuts reasoning off mid-thought. Truncated turns are counted as truncated_turns
--agentic-context-limit	abandon an episode once its context exceeds this many tokens
--agentic-transcripts DIR	save each episode's full message history to DIR/<task_id>.json, plus a DIR/<task_id>.html showing every reasoning block, reply, tool call and tool result; the main report links to it

Code suites (HumanEval, ClassEval)

--dataset humaneval runs the 164 OpenAI HumanEval problems; --dataset classeval runs the 100 ClassEval class-generation tasks. Unlike the other suites these are not graded by matching text: the model's reply is parsed for a ```python block, that code is combined with the task's own unit tests, and the result is executed.

ClassEval asks for a whole class rather than one function, so it is scored the way upstream does - class level (every one of the task's tests passes, which is what feeds pass@k) and method level (what fraction of all 2196 test methods passed). The second number separates "generated nothing usable" from "got most of the class right", and both are printed at the end of a run.

Give it more room to generate than HumanEval - a class with ten methods does not fit in a short budget:

python3 llama-eval.py \
  --server http://localhost:8033 --model my-model \
  --dataset classeval --threads 8 --n_predict 4096 --temperature 0

# pass@1, greedy
python3 llama-eval.py \
  --server http://localhost:8033 --model my-model \
  --dataset humaneval --threads 8 --n_predict 1024 --temperature 0

# pass@10 -- each extra multiple of 164 is another sample per problem
python3 llama-eval.py \
  --server http://localhost:8033 --model my-model \
  --dataset humaneval --n_cases 1640 --temperature 0.8 --top-p 0.95

--grader-type exec is the default for code suites, so neither it nor --dataset-source normally needs to be passed.

Execution sandbox - no Docker required

Generated code is untrusted, but these suites need far less than a container to run safely. The package set is derived from the dataset's own imports at startup, not hardcoded and not taken from the upstream requirements.txt - which for ClassEval lists torch, openai and transformers that its tests never touch. So each suite installs exactly what it needs:

suite	third-party packages
HumanEval	none - every problem imports only stdlib (typing, math, random, copy, string, collections, hashlib, re)
ClassEval	13: beautifulsoup4, gensim, lxml, netifaces, nltk, numpy, openpyxl, pandas, pillow, pypdf2, python-docx, reportlab, scipy

Two of ClassEval's are declared rather than derived, because no import statement mentions them: lxml is selected by string (BeautifulSoup(html, 'lxml'), ClassEval_44 - without it that task silently drops from 23/23 to 7/23), and scipy is a hard gensim requirement. Note also that PyPDF2 must not be modernised to pypdf: ClassEval_69's test imports PdfFileReader, a name pypdf removed.

The venv is cached under ~/.cache/llama-eval/ (override with LLAMA_EVAL_CACHE) and rebuilt only when the package set, interpreter version or provisioning steps change; force one with --rebuild-sandbox. uv is used when available, which makes even the ClassEval venv build in seconds.

Interpreter selection. A suite can prefer a different Python than the one running the tool, and declares it: ClassEval asks for 3.13, because gensim publishes no wheel past cp313 and its generated C no longer compiles on 3.14 (it still uses PyLongObject.ob_digit). That interpreter is found automatically, including via uv python; --sandbox-python overrides it.

3.14 is fully supported anyway. ClassEval uses exactly two gensim APIs (utils.decode_htmlentities, matutils.unitvec), both pure Python upstream, so on 3.14+ the suite drops gensim from the install set and provisions a small shim instead. Both paths score 100/100.

Offline provisioning. A suite that would otherwise fetch data at test time declares a provisioning step that runs once at venv build time, while the network is still up. ClassEval uses this to pre-seed exactly three nltk corpora (punkt_tab, averaged_perceptron_tagger_eng, wordnet, ~33 MB), so the tests themselves run with no network at all. Those are the current names - the dataset asks for punkt and averaged_perceptron_tagger, which nltk has since renamed, and seeding the legacy names too is not harmless: with a network available nltk considers them satisfied and re-fetches the modern ones anyway. The corpora are written inside the venv, not ~/nltk_data, because $HOME is a tmpfs under the strongest isolation tier.

Each candidate program runs in a throwaway directory as its own process group, under the strongest isolation the host offers:

tier	requires	isolates
bwrap	bubblewrap installed	read-only root, tmpfs /tmp and $HOME, no network
unshare	unshare --map-current-user	no network
unshare-root	any unshare -r	no network, but runs as uid 0 - see below
rlimit	always available	cpu/memory/file-size/process caps only

Candidate code must not run as root, which is why unshare -r is a last-resort tier that prints a warning. Root bypasses file permission bits, so a test asserting that writing to a chmod 0444 file fails instead sees it succeed - ClassEval_50 scores 14/16 under uid 0 and 16/16 under uid 1000, with nothing in the output to suggest the sandbox caused it. Install bubblewrap.

All tiers additionally cap CPU time, address space (4 GiB), file size and core dumps, and kill the whole process group on timeout (--exec-timeout, default 15 s - raise it for ClassEval, whose tests do real work). Writes to the throwaway working directory are allowed, which is all the file-handling tasks in ClassEval need.

Note that the weakest tier does not contain filesystem access. If you are scoring output from a model you do not trust, check the tier printed at startup.

Reproducibility

Several tasks draw their test inputs from the global, unseeded RNG, so a verdict is not reproducible run to run - which adds noise exactly when you are comparing two models or two quantisations. --exec-seed (default 0) seeds the RNG before each test; pass --exec-seed -1 for the stock upstream behaviour. This does not change what the model is asked to compute.

It matters more than it sounds. HumanEval/38, /50 and /53 are affected, and ClassEval_58 (duplicate mine placement) fails about 30% of the time on its own reference solution - measured here at 21/30 passes unseeded versus 30/30 seeded.

Dataset fidelity

The canonical dataset runs unmodified on current Python - all 164 reference solutions pass on CPython 3.14 with no patches, so no forked or re-hosted copy is needed. --dataset-source accepts a local .jsonl or a HuggingFace repo id if you do want to point at a modified copy.

HumanEval does contain a handful of long-standing defects where the prompt contradicts the graded test (notably HumanEval/47, /116 and /148). Those penalise a model that reads the docstring carefully, but correcting them makes the benchmark marginally easier and the resulting scores no longer comparable with published HumanEval numbers, so nothing is patched by default.

ClassEval is different: the upstream data has defects that cap the achievable score below 100% for reasons unrelated to the model, so --dataset classeval defaults to a patched fork, ilintar/ClassEval. Six edits over five tasks lift the reference solutions from 96/100 classes and 2182/2196 test methods to 100/100 and 2196/2196. Three repair tests nothing could pass:

• ClassEval_17 was a time bomb - a hard-coded 2024 date checked against datetime.now(), unpassable since January 2024.
• ClassEval_48 asserted get_hostname('0.0.0.0') == 'LAPTOP-2CS86KUM', the dataset author's own machine name.
• ClassEval_31 compared floats with assertEqual, failing on a 4-ULP numpy difference.

The other three fix only the reference solutions (NumPy 2.0 removals in ClassEval_51, a 30%-flaky mine generator in ClassEval_58) and cannot affect a model's score. No skeleton was touched, so what the model is asked to write is byte-identical to upstream. Pass --dataset-source FudanSELab/ClassEval for the unpatched original.