* llama-eval : add sandboxed execution for code suites
Adds a lightweight replacement for the Docker containers usually used to run
generated code, so a benchmark can be executed without a daemon, an image or
root.
A cached venv is provisioned per suite and reused; candidate programs run in a
throwaway working directory as their own process group, under the strongest
isolation the host offers:
bwrap read-only root, tmpfs /tmp and $HOME, no network
unshare no network, via --map-current-user
rlimit cpu/memory/file-size/process caps only
Candidate code must not run as uid 0, which is why "unshare -r" is a
last-resort tier that warns: root bypasses file permission bits, so a test
asserting that a write to a chmod 0444 file fails would instead see it succeed.
Two details worth noting for the bwrap tier. Only paths hidden by the $HOME
tmpfs are bound back, because bwrap cannot mount onto a symlink and
version-manager alias directories usually are one. And the venv's base
interpreter tree is resolved by walking the symlink chain, since a
version-managed interpreter is commonly reached through an unversioned alias
that startup resolves through.
Suites can declare provisioning steps, which run once at build time while the
network is still up, so test time never needs it.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
* llama-eval : add HumanEval and ClassEval code suites
Adds two execution-graded suites alongside the existing text-graded ones. The
model's reply is parsed for a python code block, combined with the task's own
unit tests, and run in the sandbox.
--dataset humaneval 164 function-level problems, scored pass@k
--dataset classeval 100 class-level tasks, scored at class level (feeding
pass@k) and method level, matching upstream
pass@k reuses the existing chunk mechanism: --n_cases at a multiple of the
dataset size gives that many samples per problem, aggregated with the unbiased
estimator from the Codex paper.
Program construction and result interpretation are dataset-owned hooks, because
the two suites differ: one HumanEval problem is a function judged by a single
assert, whereas one ClassEval task is a class judged by many independent test
methods, and reducing that to pass/fail would throw away partial credit. Suites
also declare their own package set, provisioning steps, interpreter and timeout.
The package set is derived from the dataset's own imports rather than a
requirements file. HumanEval turns out to need no third-party packages at all;
ClassEval needs thirteen, two of which must be declared because no import
mentions them -- lxml is selected by string in ClassEval_44, and scipy is a
gensim requirement.
Notes on ClassEval specifically:
* gensim publishes no wheel past cp313, so 3.13 is preferred; on 3.14+ the two
pure-Python APIs it uses are provided by a small shim instead.
* The three nltk corpora it needs are pre-seeded at venv build time. These are
the modern names -- the dataset asks for the since-renamed punkt and
averaged_perceptron_tagger.
* It defaults to a patched fork, ilintar/ClassEval, because three upstream
tests cannot be passed by anything: a hard-coded 2024 date checked against
datetime.now(), an assertion on the dataset author's own hostname, and an
exact float comparison. Reference solutions go from 96/100 to 100/100 classes
and 2182/2196 to 2196/2196 test methods. No skeleton is touched, so what the
model is asked to write is unchanged; --dataset-source selects the original.
--exec-seed (default 0) seeds the RNG before each test, so suites whose tests
draw random inputs score reproducibly. ClassEval_58 fails its own reference
solution about 30% of the time without it.
The simulator gains both suites, so the harness can be exercised without a
model.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
* llama-eval : fix silent isolation downgrade, generalise the sandbox runner
The isolation probe ran the interpreter without binding it back, so whenever
this tool is itself started from a venv under $HOME -- the usual case, since it
needs `datasets` -- bwrap's tmpfs hid the interpreter, the probe failed, and the
sandbox quietly fell back to a tier that does not hide $HOME at all. Nothing in
the output looked wrong when that happened, and the existing suites have been
running that way. The probe now uses the same interpreter and the same binds the
suite will actually get.
Two additions the agentic suite needs, neither of which changes the existing
suites:
* run_argv() runs an arbitrary command in a caller-owned directory that survives
the call. run() is now a thin wrapper over it, so HumanEval and ClassEval take
exactly the same path as before. A multi-step agent needs its edits to persist
across many invocations, and a suite for a language other than Python needs to
drive a linter and a test runner rather than a single source string.
* RLIMIT_AS is now per-sandbox, defaulting to what it was. It caps reserved
address space rather than resident memory, which is the wrong instrument for a
runtime that reserves a large region and commits little of it: node's
TypeScript type-stripping goes through WebAssembly, reserves gigabytes per
instance, and aborts at 4 GiB with an out-of-memory error that reads like a
fault in the code under test.
HumanEval stays at 164/164 and ClassEval at 100/100.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* llama-eval : add SACB, a multi-turn agentic coding suite
`--dataset agentic` is a different shape from the other code suites. Rather than
answering a single prompt, the model is given a small but real repository, a
narrow tool set and a bug report, drives its own multi-turn conversation, and is
graded by running a test suite it was never allowed to see.
The corpus is SACB (Simple Agent Coding Benchmark), 60 tasks over three
hand-written repositories in Python and TypeScript, hosted at ilintar/SACB.
Grading needs no container: a cached venv gives ruff and mypy for Python, and
node's own test runner plus tsc for TypeScript, so the whole dependency set is
four packages.
Three constraints are deliberate:
* No shell, and no way to run the tests. The only feedback channel is `lint`. A
syntax or type error can be driven out mechanically; a logic error has to be
reasoned about. A harness that lets an agent iterate against the test suite
measures something closer to search than to understanding.
* Two edit tools, one line-addressed and one content-addressed. Which one a
model reaches for, and whether it keeps line numbers straight after its own
earlier edit, is itself a signal.
* The tests never touch the working tree. They are held outside it and copied
into a throwaway copy only once the agent has stopped, so they can be neither
read nor edited. Every tool path is resolved and confinement-checked against
the repository root, so a symlink planted inside the tree cannot reach out of
it.
A task counts as resolved when every test that was failing passes and nothing
that already passed broke. Both sets are derived by running the tests against
the defective tree and against the reference fix, never written by hand, so a
task whose defect no test exercises cannot enter the corpus. Many tasks carry
several independent defects, so the fraction of target tests passed is reported
alongside, separating "fixed two of the three faults" from "changed nothing".
An episode that arrives with no tool call and no `finish` is nudged rather than
ended. Models routinely narrate their analysis mid-task, and treating that as
completion ends the episode with no edits and scores a reasoning failure that
never happened; about one episode in four exits that way.
Nothing here is installed unless this suite is selected, and then only for the
language being run: agentic_eval is imported lazily and each toolchain is keyed
per language.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* llama-eval : make the agentic hooks typed rather than discovered
`ty` flagged nine unresolved attributes, of which the seven in main() predate
the agentic suite: `EvalState.dataset` is Optional and was dereferenced without
narrowing.
* `loaded_dataset` turns the Optional into a value in one place, raising rather
than handing back None, so callers do not each have to narrow it.
* `is_agentic`, `prepare()` and `run()` are declared on BaseDataset. They were
reached through getattr with a default at three call sites, which no checker
can see through; `run()` now raises a clear error naming the suite instead of
failing as a missing attribute.
* The sandbox setup is one branch on `is_agentic` rather than two tests of
`grader_type`, which also drops a duplicated condition.
No behaviour change: HumanEval stays at 164/164, and `ty check` on
examples/llama-eval is clean.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* feat: API key
* fix reasoning and response length for agentic runs, add option to save entire transcripts
* llama-eval : fix ty unresolved-attribute on loaded_dataset.ae
Assisted-by: AI coding agent
* llama-eval : trim comments and remove non-ASCII from README
Assisted-by: AI coding agent
* enable OPENSSL=ON as default in build script, update it in CI and dockerfile
* change to LLAMA_BUILD_BORINGSSL on windows
* refactor
* rm Download UI build
* Apply suggestion from @CISC
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Apply suggestion from @CISC
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Apply suggestion from @CISC
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* examples : fix draft sampling and rejection handling
This commit fixes two issues in the speculative example which are
described in detail in the linked issue.
Resolves: https://github.com/ggml-org/llama.cpp/issues/29975
* examples : add kv-unified requirement for parallel > 1
* server : preserve context checkpoints across slot save/restore
Append the checkpoints after the packed server_tokens payload added in #26640
and count them in n_written / n_read, so a restored slot can still roll back to
a checkpoint instead of re-processing the whole prompt.
* server : drop draft checkpoint data that does not match the draft context
Restoring a slot saved with a different draft KV cache type aborted in
load_dft(). Test-load one draft checkpoint on restore and drop the draft
data if it does not fit, instead of crashing. Adds a regression test.
Co-authored-by: Igor Okulist <okigan@gmail.com>
* server : harden the checkpoint appendix of slot save files
Bound each blob size by the bytes left in the file before allocating, open the
file with UTF-8 paths on Windows like the llama state payload, fall back to full
prompt re-processing when a checkpoint restored from a slot file fails to load,
and replace the 1024 count cap by keeping the last n_ctx_checkpoints while reading.
* server : report an incomplete checkpoint appendix as a failed slot save
Return an error to the client when the appendix cannot be written, like a
failed payload write, and make the oversized-blob test declare a size that
cannot be allocated, so an unbounded allocation fails the test.
* server : reject an empty target state in the checkpoint appendix
A saved checkpoint always holds a target state, an empty blob would roll back
without restoring anything. Also log with the slot id, and load the draft test
model from the HF cache instead of a second download.
* common : return bool from checkpoint load_tgt / load_dft
A checkpoint restored from a slot file falls back to full prompt re-processing
when it fails to load, a checkpoint created in memory still aborts.
---------
Co-authored-by: Igor Okulist <okigan@gmail.com>
This commit adds an optional --add-bos token command line option to the
run-org-model.py script.
The motivation for this is that there are models, for example Gemma4,
that explicitely set the add_bos value to true in llama-vocab.cpp even
if the original model does not set this value to True.
It would be nice to be able to force the models to agree on the bos
token so that logit verification can proceed.
Refs: https://github.com/ggml-org/llama.cpp/pull/21500
* adapt common
* add common_batch
* wip
* wip: spec
* cont
* common_speculative_process
* server_batch to use common_batch
* rm some stale calls
Assisted-by: Claude Fable 5.1
* migrate mtmd
* handle imrope, handle return val of add()/add_embd()
* add spec zeros vector
* add warning on zero fill path
This commit adds a new recipe/target to the Makefile which allows the
logits verification to be run on pre-existing model outputs.
The motivation for this is that for large models it can take a long time
to run them models, and especially for the original model which seldom
changes this is very time consuming. With this change we can run the
original model one which will store the tokens and logits, and then
manually run the converted model and the run use this recipe to verify
them against the orignal model.
* metal : add top-k MoE fusion
Adds a Metal fusion for SOFT_MAX + ARGSORT + GET_ROWS with optional
routing-weight normalization and scale, matching the top-k MoE fusion
available in the CUDA and Vulkan backends. The fused kernel writes the
selected expert ids and routing weights directly, eliding the separate
softmax, argsort, get-rows, sum-rows, clamp, div and scale kernels.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : add MoE weighted reduction fusion
Fuses MUL(experts, weights) plus the expert VIEW/ADD chain into one kernel
that computes the weighted sum directly. The graph_optimize hook keeps the
expert and weight buffers alive until the fused output so the allocator cannot
reuse them while the kernel is still reading them.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* tests : expose MoE weighted reduction in fusion baseline
Use 2 experts per token in the generated MoE test models so the Metal
MoE weighted reduction fusion (MUL + ADD) is exercised by test-fusion.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : fuse RMS_NORM + SCALE
Adds NORM/RMS_NORM + SCALE fusion to the Metal backend by reusing the
norm+mul kernel with a scalar scale flag. Adds test coverage for both
NORM+SCALE and RMS_NORM+SCALE and regenerates the fusion baseline.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : use function constant for RMS_NORM + SCALE
Replaces the runtime use_scale karg with a Metal function constant. The
norm+mul kernel is compiled with FC_norm_use_scale=false for MUL fusion and
FC_norm_use_scale=true for SCALE fusion, so the fused kernel has no runtime
branch.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : use function constant for top-k MoE with_norm
Replaces the runtime with_norm karg with a Metal function constant. The
top-k MoE kernel is compiled separately for the normalized and non-normalized
routing variants, removing the runtime branch.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : rename moe_weighted_reduction suffix to moe_reduce
Shortens the MoE weighted-reduction fusion identifiers, kernel, pipeline,
matcher, args struct, and test op name from moe_weighted_reduction to
moe_reduce.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : add MUL_MAT + UNARY and MUL_MAT + ADD + UNARY fusion
Adds dense mat-vec activation fusion for sigmoid/silu and bias+softplus.
The mat-vec kernels apply the activation/bias epilogue via function
constants, avoiding the separate unary/add passes.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : revert MUL_MAT + UNARY and MUL_MAT + ADD + UNARY fusion
The mat-vec activation fusion regressed decode throughput on Qwen3.6-35B-A3B
by ~8% (tg32 81.5 vs 88.5 t/s). The regression is caused by loss of
concurrency: the standalone unary kernels previously overlapped with other
mat-vec work, while fusing the activation into the mat-vec kernel serializes
it on the critical path.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : add SSM_CONV + UNARY (silu) fusion
The SSM_CONV kernels apply silu directly via a function constant, eliding
the separate unary pass. Regenerates the fusion baseline.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : address fusion review comments
- Fix declaration/table alignment
- Rename top-k MoE kargs fields to val_clamp / val_scale
- Move moe-reduce alloc-deps handling into a general fusion helper
- Remove the public moe-reduce matcher API
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : fix unused parameter in top-k MoE fusion check
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : guard SSM_CONV fusion lookup behind use_fusion
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : track all fused outputs in graph reorder
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : keep top-k MoE logits alive until fused output
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : refactor alloc deps to pattern-driven approach
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : check fused kernel destination in concurrency tracking
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* meta : forward graph_optimize to underlying backends
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : use vector for fusion table
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* meta : keep graph_optimize unimplemented
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* parallel : fix non-deterministic prompt selection
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* parallel : support dummy models and add global logits run hash
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : sync cross-device copies with destination completion event
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : avoid const_cast in fusion alloc deps
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : skip fusions with aliased sources
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : hide fusion pattern definition
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : use vector fusion op sequences
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : drop redundant struct keywords
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : add alloc deps comment separator
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : generalize fusion output memory ranges
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : rename fusion out_offsets to outs
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : avoid dst vector in memory range check
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : optimize fusion matching and multi-output handling
- use pointer arithmetic for fusion info count lookup
- avoid heap allocations in top-k MoE and MoE reduce pattern matchers
- use fusion outs for multi-output subgraph checks
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* Revert "parallel : support dummy models and add global logits run hash"
This reverts commit 57c7caf941c1b43c270fd5009c9f175063522e96.
* fusion : update MTL.csv
* metal : unroll constant loops in top-k MoE kernel
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : use function constants for top-k MoE n_expert and top_k
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : rename fusion kargs to scale and clamp
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* metal : use function constants for moe_reduce and ssm_conv
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* fusion : update MTL.csv
This commit updates cmake to use PROJECT_SOURCE_DIR instead of CMAKE_SOURCE_DIR for paths in function calls.
The motivation for this is that when using add_subdirectory,
CMAKE_SOURCE_DIR is fixed to the top-level projects source directory,
that is the caller of add_subdirectory and not the llama.cpp root
which means that common/common.h header will not be resolved.
Refs: https://github.com/ggml-org/llama.cpp/pull/28091#issuecomment-5636106377
* server: fix speculation after an image
Pass the actual position to the drafter after an image, instead of the
token count. Affects every drafter, not just DFlash.
* rename draft n_past to pos0
n_past is used to denote number of tokens and this parameter is meant to be a position
This commit adds the printing of the ggml version and commit to the
test-cmake example.
The motivation is just to be able to quickly verify that the correct
version of ggml is being used.
Example output:
```console
test-cmake] llama.cpp version: 0.4.0-dev, build: 10837 (5202104b5)
[test-cmake] ggml version: 0.23.0, commit: 5202104b5
[test-cmake] Initializing backend...
...
```
* xcframework : fix build
* mtmd : remove unused include path
* vendor : use vendor::hash alias target in cmake
CMake reserves "::" in target names for imported/alias targets, so the real
target keeps the name vendor-hash and a vendor::hash ALIAS target is added.
Consumers (mtmd, llama-gguf-hash) now link against the namespaced alias.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* vendor : add cmake targets for all vendored libs with vendor:: aliases
Add INTERFACE targets for the header-only vendor libs (miniaudio, nlohmann,
sheredom, stb) and ALIAS targets named vendor::<lib> for all of them,
including cpp-httplib and hash. Each exposes the vendor/ root so includes
are namespaced, e.g. <nlohmann/json.hpp>.
Consolidate the per-lib add_subdirectory calls into a single
add_subdirectory(vendor), keeping the cpp-httplib gate on LLAMA_BUILD_COMMON.
Consumers (llama-common, mtmd) now link the aliases instead of relying on
raw vendor/ include paths.
hash: consumers now include via "hash/hash.h"; the vendor/hash dir is kept
as a PRIVATE include so the synced upstream sources compile unmodified.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* readme : use foo/bar names in acknowledgements
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ocd : fix valign
Replace the deprecated --mmap, --no-mmap, --mlock, and --direct-io flags with
the unified --load-mode argument across scripts, examples, and documentation.
Internal warning message and env var docs updated accordingly.
Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
Scrub developer-specific /home/<user>/ paths from example docs and test
fixtures so they don't leak into the tree.
- examples/test-cmake/README.md: /home/danbev/... -> /path/to/llama.cpp/...
- tests/test-chat.cpp: /home/jarvis/... -> /home/user/... (input and
expected string kept identical so the parser test still passes)
Co-authored-by: Jim Wu <ywu@xilinx.com>
* cmake : introduce semantic versioning (wip)
This commit introduces semantic versioning to llama.cpp.
* squash! cmake : introduce semantic versioning (wip)
* cmake : update test-cmake README notes [no ci]
* include libmtmd in output so show its semversioned
* ci : add make-release workflow
* ci : fix build number check in build-cmake-pkg.yml
* examples : remove trailing whitespace
* ci : abort if upstream ggml version does not exist
* ci : extract step contents into scripts
* ci : add GGML_NATIVE=OFF to ubuntu job
* examples : remove CI build information from test-cmake [no ci]
This commit removes the nightly/release information that I added
previously to keep this focused only on using building and installing
llama.cpp with cmake and being able to quickly verify changes or
troubleshoot issues.
* ci : merge scripts into single script
* remove -dev-build_number support
This commit removes the incremental build number (versioning) support
that I added. This was incorrect and we should only use the semver for
the version. Releases will be tag a nightly build and package
maintainers/managers that build from source can use the tag and it is
therefor important that the correct version is reported. So a
nightly-build will report the semver without the build number. The build
number and commit as availble via cmake and test-cmake has been updated
to include an example of using them:
```console
$ ./build.sh
[test-cmake] version: 0.1.0, build: 10360 (08c69e381)
...
```
Refs: https://github.com/ggml-org/llama.cpp/pull/26839#discussion_r3755836969
* docs: add initial release.md documentation
* cmake : clean-up and add LLAMA_BUILD_IS_DEV option
* ci : remove version input from make-release job
* ci : add LLAMA_BUILD_IS_DEV=OFF to build-cmake-pkg.yml
Refs: https://github.com/danbev/llama.cpp/actions/runs/31576801921/job/94050639145
* docs : update release notes with LLAMA_BUILD_IS_DEV info [no ci]
* ci : add TODO to winget workflow [no ci]
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This commit updates the python script that runs the original model to
generate embeddings for the causal model, to use save_output_data which
stores the token ids and the prompt in addition to logits.
The motivation for this is that the embedding logits verification will
fail as it expects these files (-prompt.txt and -tokens.bin) to exist.
With the changes in this commit the causal-verify-embeddings target
works again.
* Enable backend sampling with token speculation
* Clamp the mask sum before converting it into the sampled index
* Add a numeric context parameter declaring the maximum outputs one sequence
* More fixes
* Don't reuse memory for output views.
* Match dist between CPU and GPU
* Fix CPU and backend sampling mismatches
* Simpify some of the changes
* Fix tests on Vulkan
* More test fixes
* Rebase changes
* Rebase and address review comments
* Address review comments
* Address review comments
* Update src/llama-sampler.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This commit adds the --model-name flag to the causual and embedding
model conversion scripts.
The motivation for this is that this is the name used for the metadata
field general.name and it can be useful to specify this explicitely if
the default (the basename of the model path) is not what we want.
* server: add get_info tool
* fix --rpc in docs
* server: harden get_info probe result handling
Report the OS as unknown when the probe process fails to spawn or times
out, so the diagnostic text from run() is never returned as an OS name.
Strip the probe output on both ends, which also drops the blank line
that ver prints before the version on Windows. Name the output and
timeout limits, and report an unreadable working directory as unknown
instead of an empty string.
* server: simplify get_info result handling
Drop the named limits and the working directory error branch, keeping
the probe result handling to a single expression.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
dict.get("key", default) returns None (not default) when the key
exists but its value is explicitly None. This caused an AttributeError
in _escape_html() when a task errored before grading and answer was
set to None.
Assisted-by: pi:llama.cpp/Qwen3.6-27B
* requirements: relax torch~=2.6.0 to torch>=2.6.0 for convert_hf_to_gguf
The ~=2.6.0 operator resolves to >=2.6.0, <2.7.0, which fails on
PyPI for platform/CPython combinations where 2.6.x is not present.
The accompanying comment already says 'PyTorch 2.6.0 or later', so
the looser >=2.6.0 matches the documented intent and unblocks
pip install -r requirements/requirements-convert_hf_to_gguf.txt.
Fixes#23408
* requirements: bump torch floor to 2.11.0 per maintainer
* requirements: pin torch to ==2.11.0 per project policy
* requirements: pin mtmd torch and torchvision to 2.11.0/0.26.0 per project policy
* requirements: suppress check_requirements pin warning on mtmd
The check_requirements script flags '==' on lines in files matched by
*/**/requirements*.txt. Append the documented suppression comment to the
pinned torch and torchvision lines (and to the s390x platform marker lines)
so the check passes while keeping the pins required by project policy.
* ty: silence Tensor/Module union check on model[0].auto_model
With torch 2.11.0 stubs, nn.Sequential.__getitem__ now returns
Tensor | Module rather than Module, so model[0].auto_model fails ty
on the SentenceTransformer code path. The runtime behavior is
unchanged because SentenceTransformer always wraps a Module at
index 0. Adding a targeted unresolved-attribute ignore keeps the
type-check green without altering behavior. A follow-up issue
tracks typing the variable explicitly.
* pi : update
* ci : fix ios build
* ci : fix andoroid
* ci : fix apple builds
* cmake : add install() for impl libraries
Add install(TARGETS <target> LIBRARY) for all -impl libraries that were
changed from STATIC to shared (controlled by BUILD_SHARED_LIBS) in
commit bb28c1fe2. Without this, cmake --install fails to copy the shared
libraries, causing runtime errors like:
llama-server: error while loading shared libraries: libllama-server-impl.so
Ref: https://github.com/ggml-org/llama.cpp/issues/23494#issuecomment-4512912515
Assisted-by: llama.cpp:local pi
* ci : fix xcframework build
* tests : move save-load-state from examples to tests
- Move examples/save-load-state/ to tests/test-save-load-state.cpp
- Remove subdirectory reference from examples/CMakeLists.txt
- Add test to tests/CMakeLists.txt as a model test
- Remove CODEOWNERS entry for removed example directory
Assisted-by: llama.cpp:local pi
* cont : update ci
* save-load-state : refactor into separate phase functions
- Split monolithic main() into 4 self-contained phase functions, each
managing its own context/sampler/batch lifecycle
- Each function tokenizes internally using its local ctx instance
- main() is now a clean orchestrator: init -> run phases -> assert results
- Proper resource cleanup on every exit path (return {} on error)
Assisted-by: llama.cpp:local pi
* save-load-state : use params.out_file instead of separate state_file
- Remove state_file parameter from all phase functions
- Each function accesses params.out_file directly
- Initialize params.out_file in main alongside params.prompt
Assisted-by: llama.cpp:local pi
* save-load-state : use smart pointers for ctx and smpl
- Replace raw llama_context* with llama_context_ptr
- Replace raw llama_sampler* with llama_sampler_ptr
- Remove all manual llama_free() and llama_sampler_free() calls
- Keep llama_batch as raw (managed manually with llama_batch_free)
Assisted-by: llama.cpp:local pi
* save-load-state : add local llama_batch_ptr RAII wrapper
- Add llama_batch_ptr struct holding llama_batch by value
- Calls llama_batch_free() in destructor
- Eliminates all manual llama_batch_free() calls
Assisted-by: llama.cpp:local pi
* save-load-state : replace printf/fprintf with logging macros
- Add log.h include
- Replace fprintf(stderr, ...) errors with LOG_ERR
- Replace fprintf(stderr, ...) info with LOG_TRC
- Replace printf output with LOG
Assisted-by: llama.cpp:local pi
* save-load-state : refactor tests to check results inline
Each follow-up phase now accepts an expected result and performs
the comparison internally instead of collecting results in main().
Assisted-by: llama.cpp:local pi
* save-load-state : improve test output readability
Add phase labels, remove redundant run prefixes, and show
PASS after each test.
Assisted-by: llama.cpp:local pi
* pi : add rule about git signing
* save-load-state : simplify llama_batch_ptr
Change get() to return a reference and remove operator*().
Use batch.get() throughout for consistency.
Assisted-by: llama.cpp:local pi
* save-load-state : extract generate_tokens helper
Factor out the repeated token generation loop into a shared
helper function used by all phases.
Assisted-by: llama.cpp:local pi
* save-load-state : update comments to use test terminology
Replace "Phase" with "Test" and list each test's steps
as bullet points.
Assisted-by: llama.cpp:local pi
* save-load-state : rename test functions
Rename to test_baseline, test_state_load, test_seq_cp_host,
test_seq_cp_device. Update comments and logs accordingly.
Assisted-by: llama.cpp:local pi
* pi : add rule to never git push without confirmation
Assisted-by: llama.cpp:local pi
* common : add model_only option to common_init_from_params
Add bool model_only parameter to skip context creation,
sampler init, and context-dependent setup.
Use in save-load-state to initialize only the model,
with each test creating its own context.
Assisted-by: llama.cpp:local pi
---------
Co-authored-by: ggerganov <ggerganov@users.noreply.github.com>
* llama-eval : add per-problem summary table to HTML reports
- Add chunk_idx and problem_idx to TaskState and saved case dicts
- Group completed cases by problem_idx in dump_html()
- Render per-problem summary table before individual task table
- Columns: Problem (zero-padded), Runs, Correct (n/r),
Tokens (min/avg/max), T/s (min/avg/max), Gen s (min/avg/max)
- Sorted by problem index, monospace font, right-aligned numbers
- Colspan headers for grouped stats, auto width
- Simulator: add /v1/models endpoint, timings in response,
template-aware question matching, --dataset arg (aime/aime2025)
Assisted-by: llama.cpp:local pi
* llama-eval : add tabs for Detailed and Summary tables, apply monospace font globally
- Wrap Detailed and Summary tables in switchable tabs (Detailed active by default)
- Remove summary-section wrapper, use tab labels instead
- Apply monospace font to all tables and the top bar
Assisted-by: llama.cpp:local pi
* llama-eval : redesign top bar as CSS grid label/value pairs
- Replace flat span list with 4-column grid layout (2 pairs per row)
- Labels in muted color (#888), values in dark (#222)
- Bold dataset name and model name
- Removed media query, always uses 4 columns
Assisted-by: llama.cpp:local pi
* llama-eval : use realistic token counts and throughput in simulator
- comp_tokens: [30, 80] → [10000, 60000]
- tps_gen: derived → uniform [90.0, 110.0]
- t_gen_ms: now computed from tokens/tps
Assisted-by: llama.cpp:local pi
* llama-eval : color Answer column green/red based on correctness
Use the same .correct/.incorrect CSS classes on the Answer column
to make correct answers green and incorrect answers red.
Assisted-by: llama.cpp:local pi
* llama-eval : fix pyright errors from max(..., key=len) type inference
Use key=lambda x: len(x) instead of key=len so the type checker
infers the return type as str instead of Sized, fixing:
- unresolved-attribute: Object of type Sized has no attribute lower
- not-subscriptable: Cannot subscript object of type Sized
Assisted-by: llama.cpp:local pi
