2b808d5215
* build: llama.cpp to upstream master of 2026-10-18 The fork branch bmoe/expert-ready-hook-1810-dl is upstream master 5aeac37 plus the same single expert-ready-hook commit, in the variant that also answers the CPU registry's get_proc_address, so a build that loads the CPU backend as a module reaches it too. 523 upstream commits; the new base loads architectures the old one did not (Kolibri-1 among them). Upstream reworked common's chat and speculative APIs. The engine follows with the smallest change that keeps its behaviour, each one the way upstream's own callers now do it: - the rendered prompt arrives as parts; it is joined back with .str(), as upstream's server does, and tokenized exactly as before; - the parser arena's JSON serialization is gone; build_parse_params copies the arena instead, as upstream's chat session does, and reasoning_format already lives on the template inputs; - common_speculative_process reads a common_batch; the batch the engine decodes is mirrored into one at the three MTP call sites. The decode itself stays on the public llama_batch API. Same backends and build options, so docs/backends.md holds. All 23 host gates pass, a tiny Mixtral-style and a tiny OLMoE model pass all 90 byte-identity gates, and on a real model the greedy output is unchanged, MTP drafts and accepts, and the Metal prefill still runs. * test(gates): G18c compares decide and perplexity to within the kernels' batch-shape rounding The two paths put the same choice tokens through different batch shapes, a wide prefill and one row at a time, and the ARM CPU backend from the 2026-10 base accumulates them in another order: the three log-probabilities differ by the same 1e-12 relative, the log-softmax normaliser one ulp apart. Upstream never promised that comparison bitwise; the gates that prove streamed == resident stay bit-exact. A bound of 1e-9 relative keeps the check nine orders of magnitude below a wrong answer.
9.6 KiB
Compute backends: what llama.cpp carries, and what this engine has run on
The engine does not implement a compute backend. Every device it can use is one llama.cpp
registers, reached through the public ggml_backend_dev_* registry, so the list of what is
possible is llama.cpp's and the list of what is known to work is much shorter. This page keeps
the two apart.
The same facts cut by operating system, next to the rest of what each platform has, are in platforms.md.
What the pinned llama.cpp carries
Read from ggml/src/ and ggml/CMakeLists.txt of the pinned submodule (efaae9929, read again at that bump: same backends, same columns). A bump can
add or remove rows; redo the reading rather than trusting this table.
The last three columns come from reading the sources, not from running them: the type each backend
registers its devices as, whether its sources name GGML_OP_MUL_MAT_ID (the op every routed expert
goes through, so a backend without it sends the experts back to the CPU), and whether it can wrap
memory the caller already owns (buffer_from_host_ptr: the capability it advertises, and whether
the function is actually wired in). A backend naming an op is not proof it accepts every weight type
for it.
| backend | build flag | hardware | registers as | names the expert op | wraps host memory (advertised / wired) |
|---|---|---|---|---|---|
| CPU | GGML_CPU (on) |
every machine | CPU | yes | yes / yes |
| BLAS | GGML_BLAS |
a host BLAS library (Accelerate, OpenBLAS, ...) | accelerator | no | yes / yes |
| Metal | GGML_METAL |
Apple GPUs | GPU | yes | yes / yes |
| CUDA | GGML_CUDA |
NVIDIA GPUs | GPU, or integrated GPU | yes | no / no |
| HIP | GGML_HIP |
AMD GPUs through ROCm; compiles the CUDA sources | as CUDA | as CUDA | as CUDA |
| MUSA | GGML_MUSA |
Moore Threads GPUs; compiles the CUDA sources | as CUDA | as CUDA | as CUDA |
| Vulkan | GGML_VULKAN |
any GPU with a Vulkan driver | GPU, or integrated GPU | yes | no / yes |
| OpenCL | GGML_OPENCL |
Adreno first, other OpenCL GPUs | GPU | yes | no / a function that declines |
| SYCL | GGML_SYCL |
Intel GPUs through oneAPI | GPU, or integrated GPU | yes | no / yes |
| WebGPU | GGML_WEBGPU |
browsers and native WebGPU | GPU | yes | no / no |
| Hexagon | GGML_HEXAGON |
Qualcomm NPU (HTP) | GPU | yes | no / no |
| CANN | GGML_CANN |
Huawei Ascend NPU | GPU | yes | no / no |
| OpenVINO | GGML_OPENVINO |
Intel CPU, GPU and NPU through OpenVINO | GPU | yes | no / no |
| ET | GGML_ET |
Esperanto ET-SoC | GPU | yes | no / no |
| zDNN | GGML_ZDNN |
IBM Z | accelerator | no | no / no |
| ZenDNN | GGML_ZENDNN |
AMD CPUs through ZenDNN | accelerator | yes | yes / yes |
| VirtGPU | GGML_VIRTGPU |
a host GPU remoted into a virtual machine | (the remoted device's) | (the remoted device's) | no / yes |
| RPC | GGML_RPC |
another machine, or a loopback server | GPU | (the remote's) | no / no |
Three things in that table decide what this engine can do with a backend.
- The registered type is not a description of the hardware. NPUs register as
GPU, and Metal registers asGPUon unified memory. Nothing here reads the type as "has memory of its own"; that is measured (hardware-planning.md). - The advertised capability and the wired function disagree on three backends. Vulkan, SYCL
and VirtGPU report
buffer_from_host_ptrfalse and implement it. OpenCL reports it false and wires a function that returns nothing, so there the false holds in effect, and the call is safe to make. The planner therefore recordstrueas yes and anything else as unknown. - There is no backend for the Apple Neural Engine, for MediaTek or Samsung NPUs, or for an Intel NPU outside OpenVINO. On those the engine has the CPU and whatever GPU backend the platform has.
What this engine has run on
| backend | where | state | what it does today |
|---|---|---|---|
| CPU | Android, Windows, macOS, Linux | measured on phones and desktops; gates run in CI on Linux | decode, prefill, expert streaming |
| Hexagon | Android, Snapdragon with Hexagon v73 or later | measured on a phone; in the release APK (scripts/build-hexagon-android.sh) |
prefill on the NPU, decode on the CPU (npu-prefill.md) |
| Metal | macOS | measured on one 16 GB Apple-silicon laptop; on in every macOS build | prefill on the GPU, decode on the CPU (npu-prefill.md) |
| RPC | host test builds | a test fixture only: a loopback server fronting the CPU, so the prefill-device gates have a second device | nothing a front-end can select |
| BLAS | macOS, where llama.cpp turns it on by default | linked, never given a weight | nothing placed on it |
| OpenCL | Android | built, in the release APK beside Hexagon (scripts/build-hexagon-android.sh), importing the Khronos ICD loader so a phone without OpenCL still starts the engine, and the vendor's own library where there is one; in July the dense set on a phone's integrated GPU was measured and lost 27 % (RunConfig::dense_on_device) |
run on a Snapdragon-class phone with an Adreno GPU: 344 GiB/s at prefill width against the cores' 36, and it did not reproduce the host's result, so the plan refuses it on correctness and the prefill stays where the decode is; named by hand it is refused by the arena, whose per-expert placement its padded allocations do not fit (npu-prefill.md) |
| Vulkan | Android, Linux, Windows | built, in the release APK and in the Linux and Windows archives, as a module that loads only where a Vulkan driver is present. One 2020 phone ran it and its driver ended the probe, which is what the isolated probe contains | run on the same phone: 44 GiB/s one token wide against the cores' 36, and 36 at prefill width, so no prefill win; its fp16 path did not reproduce the host's result and its fp32 one did. Forced by hand, a 1727-token prompt prefilled in 85 s against 65 on the cores (npu-prefill.md) |
| CUDA, HIP, SYCL | Linux, Windows | not in any archive: a module built where the toolchain is (-DGGML_CUDA=ON, ...) and dropped beside the binary is loaded like the shipped ones; none has been run here |
unknown |
| MUSA, WebGPU, CANN, OpenVINO, ET, zDNN, ZenDNN, VirtGPU | - | never built here | unknown |
Three backends, then, with a measurement behind them, out of eighteen.
Two decisions behind the archives, written down so they are not re-made by accident. CUDA and HIP stay out of the default archives: their runtime libraries weigh more than the rest of the archive, and Vulkan reaches the same GPUs; a module built where the toolchain is loads from beside the binary like the shipped ones. And a module that is present and never given a weight is not known to be free: on one laptop with an integrated GPU, a build with Vulkan linked in loaded the model five times slower than one without, cause not found. The modules build owes that measurement, on the same machine, before the load time with an idle GPU module is called a non-event.
What each role asks of a backend
The engine uses a device in one of four ways, and each asks for something different.
| role | what the backend must do | where it stands |
|---|---|---|
| decode on the host | be the CPU backend, with the expert-ready hook for the overlap | every platform |
prefill device (--prefill-device) |
run wide graphs, accept weights through ggml_backend_tensor_set, and take the model's weight types or a type the engine can convert to |
Hexagon and Metal measured; the mechanism names no backend, so the others are untested, not excluded |
layers on a device (--gpu-layers, from llama.cpp's capacity fitter) |
have memory of its own, or the placement frees nothing | applied by the planner where a device has its own memory; never run on such a machine here. On unified memory the fitter's answer is set aside, and forcing it on the 16 GB laptop ended in the GPU's out-of-memory error |
| streamed experts computed on a device | wrap memory the streamer already reserved (buffer_from_host_ptr), which is what the fourth column above is about |
not built. Metal offers it; Vulkan, OpenCL and SYCL implement it without advertising it; CUDA has no route through the public API |
How a backend reaches the planner
No rule names a backend. A device that registers is enumerated, asked for its memory, and measured: the same matmul on the model's own quantized format as the host, checked against the host's result, and the cost of one host/device crossing. A device that does not reproduce the host's result is excluded on correctness, whatever its speed.
So a backend llama.cpp adds needs no code here to be seen. It does need a module beside the
executable, and a machine to run it on. Every release is built that way (BMOE_BACKEND_MODULES,
the top-level CMakeLists): the engine knows no backend at compile time, each backend is a library
ggml loads from the executable's directory at start-up, the CPU one comes in one library per
instruction-set tier with the best picked on the machine, and a module whose driver library is
missing fails to load and is simply not there, instead of keeping the binary from starting. The
plan then reports what registered, and keeps "this machine has no accelerator" apart from "this
build carries none". The Metal backend, the first one tried on a desktop, surfaced three
faults no test had (one memory pool counted twice, an idle device taking graph nodes, a cache the
kernel compressed). The honest reading of "unknown" above is that the same should be expected of
each of the others.
scripts/rented-box.sh is there for that: one session on a machine nobody here owns, which
builds, plans, measures, and writes down what the plan predicted next to what the engine did.