Thresholds that decided something as bare literals now have a name and a reason: the lane tie fraction behind io-threads, the rate-spread warning and the density tie tolerance in the allocator, and the form bounds of the parameter table. The two device-bandwidth comparisons that spelled the backend margin as 1.10 read PlannerPolicy::min_backend_win.
Removed, unused: is_known(Tri), StorageFacts::mapping_penalty(), ModelProfile::expert_working_set_bytes(), Lane::MmapWarm.
Plan::to_flags() is rendered from to_args() instead of a hand-written list, so the reproduce line carries every non-default parameter of the plan - the device choice and the fitter's tensor overrides were missing from it - in table order, with arguments quoted for a shell.
Comments corrected where they said something the code does not do: how a device name is used, who includes llama.cpp's common library, and how much the storage probe reads.
The compute buffers are reserved for the widest graph a prompt is computed
in (--ubatch), which follows the context, while the ledger read them from
the capacity fitter at llama.cpp's default of 512 tokens and with a logit
row per position. At a 2048 context the two errors roughly cancel; at
16384 the ledger charged 1.2 GB for 8 GB reserved, the cache lock was
refused part way and decode lost a third of its rate.
probe_placement takes the session's own width, projects with the row cap
the session sets under a prefill device, and projects again at half the
width: two points give what a token of width costs on this model, context
and set of backends (Placement::at_width). The planner then takes the
widest graph whose buffers stay inside a share of what the process may
hold (PlannerPolicy::compute_share_max, a tenth), halving from the context
and never below ubatch_min. A device's buffers count where they come out
of this pool. A pinned width is kept and charged in full; an unmeasured
slope decides nothing.
On a 16 GB Apple-silicon laptop, Qwen3.6-35B-A3B Q4_K_M, 16384 context,
--auto alone: decode 22.4 and 22.3 tok/s (14.6 and 16.7), the cache locked
at 7.7 GB (settled at 3.5), a 10850-token prompt in 31.8 s (20.4, at a
width that cost the decode). A 2048 context plans as before. Measured on
that one machine: docs/hardware-planning.md lists where it is still owed.
Split models, and what a declined plan costs.
The storage probe read the path the caller names. On a model split into shards that is the first
shard, which can be metadata alone: a 5 MB file the drive answers from its own cache. The model
profile now names the file holding the most expert bytes and the probe reads there.
A plan that declined streaming because the cache fell under the token cycle kept the allocation it
had refused, and printed its cost. It is now restated as the file left mapped with nothing held.
The cache floor.
A budget under the model's token cycle declined streaming altogether and
left the file mapped, on the machine where a mapped file is slowest: the
memory a cache lacks is the memory the page cache lacks too.
The floor now takes the cache alone. The plan keeps --moe-stream with
--cache-mb 0 when the ledger prices the uncached stream under the mapping,
and declines as before where the drive's read rate was never measured, so
the stream is never chosen without a price.
On a 16 GB Apple-silicon laptop, Qwen3.6-35B-A3B Q4_K_M at a 16384 context
with about 3 GB reported available: 0.26 tok/s declined and mapped, 7.3
tok/s from --auto now. The floor itself holds: a 536 MiB cache forced on
returned 0.0% hits at 6.3 tok/s against 8.6 with none. With 8.9 GB
reported available the plan is unchanged.
Every probe that touches a device does so by using it, and a device is
only as sound as its driver. The planner is meant to be handed a binary
that carries every backend and to find out for itself what the machine
has, which only works if finding out cannot end the program. It could:
on a phone from 2020 a build carrying the Vulkan backend loaded it,
listed the GPU, and died at the first allocation, in a function that
driver does not have and the backend calls without checking.
Each device beyond the host is now asked in a second process first: this
same executable started again with the device's name, which runs every
probe that touches that device and prints its facts as one line. A line
that comes back is applied whole or not at all. A child that dies or
does not answer in two minutes marks the device unusable, with the
reason. Either way the probes in this process leave that device alone.
An unusable device is weighed by nothing and kept out of the run: the
capacity fitter is not asked about it, no layer is placed on it, and the
host carries the run unless another device was armed for the prefill.
The plan prints it as UNUSABLE with what it did.
Starting a second process is the front-end's business, so the probe
takes a runner. bmoe-cli gives it one (--probe-device is the child's
mode, --no-isolated-probes turns it off). Where a platform has no way to
start one the runner is empty and the probes run in-process as before:
Windows today, and the desktop server.
Also: a build that carries no accelerator backend no longer reports "no
device" as a measured fact about the machine in the prefill line.
The same phone and build: the plan completes in 23 s, reports the GPU
unusable (signal 11) and the run decodes at 1.87 and 2.02 tok/s on the
cores. On a laptop whose device works the plan is unchanged and the
second process costs about 3.5 s.
An iOS harness (examples/ios, scripts/build-ios.sh).
An app on iOS may not start a process, so examples/ios compiles cli/main.cpp into an
app and calls it on a thread. On iOS the planner reads the process's own memory share
and probes devices in-process. CI compiles the harness for the simulator.
Not run on a device.
The planner chooses where a prompt is computed, and what that costs the
decode in memory.
The prefill device. The planner measures the model's matmul at prefill
width on the host and on every device - the shape of a prefill, which
ranks devices in the opposite order to a one-token decode - and arms the
fastest one that reproduces the host's result, wins by a margin, and whose
slots still fit the memory ledger. Every refusal names its reason and its
figures. A device the plan armed is best-effort: if it cannot be set up at
load the prefill runs on the host.
One expert slot or two on the device, chosen by the ledger: two where a
layer can be read while the one before it computes, one where two do not
fit.
Slots in turn with the cache. The expert slots and the expert cache are
never busy at once, so the slots a plan arms are held only while a prefill
runs; the cache yields its coldest entries for them and gets its budget
back afterwards.
A device with memory of its own and no stated size is charged to the host.
Measured on a 16 GB Apple-silicon laptop, Qwen3.6-35B-A3B Q4_K_M, --auto
alone: a 1461-token prompt prefills in 3.4 s against 21.2 s on the cores.
Three pieces that sit on one another, for every target the engine builds
on.
The parameter table (bmoe/params.h, core/src/params.cpp). Every RunConfig
tunable is one row: key, type, group, bounds written with the constants
validate() checks, help, reader and writer. bmoe-cli parses its engine
flags and prints their usage from it; --describe-params prints it as JSON
for a front-end to render, --show-config the resolved configuration.
The hardware planner (--auto, --plan, --probe; core/src/plan). The engine
measures the machine - memory it can keep and what a reclaim costs, core
classes and a thread sweep, the bandwidth of the CPU and of every device
on the model's own quantized matmul, what a host/device crossing costs,
the storage rate by request size and lane count - reads the model from
its gguf metadata, runs llama.cpp's capacity fitter, and resolves the
streaming knobs with the fact behind each one. The rules are pure
functions over two profiles and name no platform, vendor, device or
architecture; the *_probe adapters are the only files that may. A knob
set by hand is never touched, a missing fact declines instead of
guessing, and nothing lossy arms itself.
One memory ledger with two ceilings. Everything a run holds - context and
compute buffers, a device armed for prefill, the dense set, the expert
cache - is a row charged against what the process may hold and,
separately, against what it may hold reclaim-exempt, net of what is
already locked. The dense set is pinned whole or not at all, and the
cache is sized to what can be locked where it can be locked at all.
--cache-pin-fit steps the cache budget down to what the platform grants
when a lock is refused part way, instead of ending pinning; off by
default, set by the planner when it sized the cache to the lock.
macOS. Metal in every build, the planner, --release-mmap and the memory
telemetry on Darwin, --devices auto (a device with nothing placed on it
stays out of the graph), the expert cache pinned in RAM, an evicted
expert's pages really released, and --dense-weights ahwb as wired
memory.
Also: --release-mmap (+46% decode on Windows, where a live mapping
serialises the streamer's unbuffered reads), a portable build that
collects every binary and backend library in build/bin, and the planner
in the layer maps of docs/architecture.md and AGENTS.md.
Measured on a 16 GB Apple-silicon laptop with --auto alone: DeepSeek V4
Flash UD-IQ2_M 2.46 to 2.48 tok/s (0.27 before the ledger, when the plan
asked for more locked memory than the machine grants), Qwen3.6-35B-A3B
Q4_K_M 23.1 to 23.5 tok/s.
ci: the workflows, the Android staging script and two bench scripts look for
the binaries in build/bin, where the build now collects them.
docs: docs/backends.md lists the eighteen compute backends in the pinned
llama.cpp and which three have a measurement behind them here.
