vrr/BigMoeOnEdge
2
0
Fork 0
mirror of https://github.com/Helldez/BigMoeOnEdge.git synced 2026-10-19 15:33:46 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

feat(moe): --drop-cold-experts — spend quality only where it buys I/O (#95)

Browse source 
* feat(moe): --drop-cold-experts, spend quality only where it buys I/O

Turbo top-k drops the tail of a routing whether or not those experts were
already in RAM. A resident expert costs no flash read, so that trade pays
quality for nothing on the ~80% of decode routings that are cache hits.

This skips a routed expert only when it is a cache MISS and the router
weighted it below frac x (1/top-k). Replayed over the committed route
traces at frac 1.0, decode phase, that avoids 66% of flash reads for 9.5%
of the router's weight mass, where --n-expert-used 5 avoids 23% for a
comparable 10.6% -- about 3x the reads at the same quality cost.

Implementation. The decision needs the FINAL router weights, which arrive
several nodes after the topk where the streamer normally loads, so
load_layer() is deferred to the terminal node of the layer's weight chain.
Which node that is depends on the model's gating, so the hook learns it
from the graph rather than carrying an architecture table; if it fails to
arrive the hook forgets it and re-learns rather than re-betting. A dropped
slot has its weight zeroed and its expert id repointed at the routing's
top-weighted expert: an unread expert can sit in reserved-but-uncommitted
VM and mul_mat_id would touch it anyway, so the kernel is given memory
that is certainly resident and multiplies it by exactly zero.

Requires the LRU cache -- with --cache-mb 0 residency reads all-miss and
the policy would silently degenerate into an unconditional weight cut.
Prefill is excluded by default. The top expert is always pinned, so no
routing can be emptied at any threshold.

Gates: G8a/G8a' prove the deferral and the learned terminal node are
transparent (byte-identical output, zero drops, at a threshold below any
producible weight); G8b that full strength against a constantly-evicting
cache never reaches an unloaded slot; G8c that at top-k 1 dropping is a
no-op, pinning both the top-expert guarantee and the threshold tracking
the effective top-k.

Three existing metrics shift meaning under dropping and the docs now say
so: cache_hit_pct rises without the cache serving more (a dropped routing
is a miss that is never looked up), and token/layer_demand measure what
was staged rather than routed. prefetch.md's "cannot change output" is
scoped, limitations.md gains the non-reproducibility entry, and
benchmark-method.md warns that reversing the run order cannot distinguish
a moved drop rate from a contaminated cell.

Off by default in the CLI and in the app. The output is not reproducible
-- what gets dropped depends on what the cache held -- so it carries no
rows in the README tables, and switching it on by default waits on a
published on-device A/B rather than on the replay argument alone.

* feat(app): default cache-aware dropping to 75%, measured on device

Qwen3.6-35B-A3B (top-k 8 of 256), in-app, cache 3000, one variable
changed: 2.549 tok/s off, 3.938 at F=0.75 (+55%), 4.702 at F=1.0 (+84%),
with flash reads falling 248 -> 163 -> 48 GiB. Per-token bootstrap
intervals separate every pair except off vs 0.50, which overlaps -- at
half the uniform share the policy drops 2.7% of routings and buys
nothing, which doubles as a negative control that the machinery is free
when it does not fire.

Run order was 1.0, off, 0.5, 0.75, so the two fastest cells are the first
and the LAST; thermal drift would have made the last the worst. The
mechanism orders by threshold even though the run order does not.

The replay turned out conservative rather than optimistic. It is
documented as an upper bound because it cannot model the cache changing
in response to dropping: at F=0.75 it was accurate (37% predicted, 34%
measured), at F=1.0 it understated (66% predicted, 81% measured). Avoided
reads free cache capacity, which raises the hit rate, which leaves fewer
misses to drop.

75% rather than 100% is deliberate: it takes the larger part of the win
for half the discarded routings (14% against 28%). Quality is still
unquantified -- no perplexity number and no side-by-side exists -- so the
conservative end of a measured range is the defensible default. The CLI
stays off; the byte-identity gates need a deterministic default.

Also records cache_hit_pct rising 67.8 -> 90.7% as the documented
accounting artefact rather than the cache serving more, and majflt/token
as dominated by each run's starting memory state, not by the threshold.
This commit is contained in:
Helldez 2026-07-22 17:21:55 +02:00 • committed by GitHub
parent 79611654cd
commit bea5a0b99e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
36 changed files with 6650 additions and 39 deletions
Download patch file Download diff file Expand all files Collapse all files

70
CHANGELOG.md
View file

@ -4,6 +4,76 @@ All notable changes to this project are documented here. The format follows
[Keep a Changelog](https://keepachangelog.com/), and the project aims to follow
Semantic Versioning.
## [Unreleased]
### Added
- **`--drop-cold-experts F` — cache-aware expert dropping.** Skips a
routed expert when it is a cache **miss** *and* the router weighted it below `F × (1/top-k)`. An
expert already resident costs no flash read, so it always runs however small its weight: quality
is spent only where it buys I/O. Replayed over the committed route traces at `F = 1.0`, decode
phase, this avoids **66% of flash reads for 9.5% of the router's weight mass**, where
`--n-expert-used 5` avoids 23% for a comparable 10.6% — roughly 3x the reads at the same quality
cost. (At equal *reads* instead, `--n-expert-used 3` avoids 59% but discards 37% of the mass.)
`--drop-no-renorm` and `--drop-in-prefill` are the A/B switches. Requires the LRU cache:
`validate()` rejects it with `--cache-mb 0`, where every expert reads as a miss and the policy
would silently degenerate into an unconditional weight cut.
Unlike turbo top-k the output is **not reproducible** — what gets dropped depends on what the
cache held — so it carries no rows in the README benchmark tables, which are a deterministic
protocol. See [docs/expert-dropping.md](docs/expert-dropping.md).
- `scripts/route-drop-replay.py`: the offline model the numbers above come from, including the
static-`k` baseline replayed on the same rows so the two policies are comparable at equal I/O.
- Route trace gains a `dropped` column, and the metrics summary `experts_routed` /
`experts_dropped` — the flag fixes a threshold, not a drop rate, so only these say what a run
actually traded. New CLI summary line `moe-drop:`.
- Example app: **Speed / quality → Drop cold experts** (off / 50% / 75% / 100% of the uniform
share), **defaulting to 75%**, disabled in mmap mode and with the cache off. 75% rather than 100%
takes the larger part of the win for half the discarded routings, which is the defensible choice
while the quality cost is unquantified. The CLI keeps defaulting to off — the byte-identity gates
need a deterministic default.
- Gates **G8a/G8a'/G8b/G8c**: a threshold below any producible weight leaves the output
byte-identical to the undropped stream (the deferred load and the learned terminal weight node are
transparent) and is asserted to have examined routings while dropping none; at full strength
against a constantly-evicting cache generation still completes, so no matmul reaches a
reserved-but-uncommitted slot; and at `--n-expert-used 1` dropping is a proven no-op, which pins
both the top-expert guarantee and the threshold being taken against the *effective* top-k.
### Measured
- On device, in-app, Qwen3.6-35B-A3B-Q4_K_M (top-k 8 of 256), cache 3000, one variable changed:
**2.549 tok/s off → 3.938 at `F = 0.75` (+55%) → 4.702 at `F = 1.0` (+84%)**, with flash reads
falling 248 → 163 → 48 GiB. Per-token bootstrap intervals separate every pair except off vs 0.50,
which overlaps — at half the uniform share the policy drops 2.7% of routings and buys nothing,
which doubles as the negative control that the machinery is free when it does not fire.
Data: [docs/bench-data/2026-07-22-drop-cold-experts/](docs/bench-data/2026-07-22-drop-cold-experts/findings.md).
- Run order was 1.0, off, 0.5, 0.75, so the two fastest cells are the first and the **last**: thermal
drift would have made the last the worst. The mechanism orders by threshold even though the run
order does not, which is what run order cannot fake.
- **The replay was conservative, not optimistic.** It is documented as an upper bound because it
cannot model the cache changing; at `F = 0.75` it was accurate (37% predicted, 34% measured), at
`F = 1.0` it understated (66% predicted, **81%** measured). Avoided reads free cache capacity,
which raises the hit rate, which leaves fewer misses to drop.
- **Quality remains unmeasured.** No perplexity number and no side-by-side exists for this knob.
Throughput is settled; the cost of discarding 14% (at 0.75) or 28% (at 1.0) of routings is not.
### Changed
- With the policy armed, `load_layer()` moves from the topk node to the terminal node of the layer's
weight chain — the decision needs the final router weights. Which node that is depends on the
model's gating, so the hook **learns** it from the graph rather than carrying an architecture
table; until it is known a layer loads at its topk node undropped, exactly as before. No behaviour
changes when `--drop-cold-experts` is off.
- README no longer calls turbo top-k "the one lossy knob" — it is now the *measured* one.
- Docs that assumed a deterministic engine are scoped: `prefetch.md` ("cannot change output" holds
only with the lossy knobs off — under dropping, a correct guess un-drops an expert),
`moe-streaming.md`, `architecture.md`, `limitations.md` (new entry for non-reproducibility) and
`runtime.h`'s contract.
- `cache_hit_pct`, `token_demand_MiB` and `layer_demand_MiB` shift meaning under dropping — a
dropped routing is a miss that is never looked up, so the hit rate rises without the cache serving
more, and the demand figures measure what was *staged* rather than routed. Documented in
`telemetry.md`, `pressure.md` (which tells you to size the cache first, dropping off) and
`metrics.h`; `benchmark-method.md` gains the axis plus a warning that its reverse-the-run-order
check cannot distinguish a moved drop rate from a contaminated cell.
- `scripts/route-analyze.py` reports when a trace was recorded with dropping on, so its
working-set figures are not misread as flash traffic.
## [0.14.0] - 2026-07-21
### Added

30
README.md
View file

@ -41,7 +41,7 @@ Highlights:
- **gpt-oss-120b (Q4_K_M), ~5× device RAM**: **1.3 tok/s** at the model's own routing width against
0.09 tok/s for the same file loaded the ordinary way (mmap), a **14×** difference at matched settings.
**2.2 tok/s** with the one lossy knob on (fewer experts).
**2.2 tok/s** with the measured lossy knob on (fewer experts).
- **Lossless on models past RAM**: Qwen3-30B-A3B (Q4_K_M, 18.5 GB) up to **5.2 tok/s**,
Qwen3.6-35B-A3B (Q4_K_M, 22.3 GB) up to **5.0 tok/s** and Gemma-4-26B-A4B (Q4_K_M, 17.0 GB) up to
**4.1 tok/s** on the same phone, output identical to the resident model.
@ -95,8 +95,16 @@ and a manual copy to the device: steps in the
device.
- **I/O–compute overlap** (`--overlap`): hides flash latency behind compute. Byte-identical;
needs a small optional add-on to llama.cpp (see [docs/seam.md](docs/seam.md)).
- **Turbo top-k** (`--n-expert-used N`): the one lossy knob. Fewer experts per token, ~+22–24%
- **Turbo top-k** (`--n-expert-used N`): the measured lossy knob. Fewer experts per token, ~+22–24%
speed, output quality is yours to judge.
- **Cache-aware expert dropping** (`--drop-cold-experts F`): skips a routed expert only when it would
cost a flash read *and* the router barely weighted it, so quality is spent only where it buys I/O.
Measured at **+55%** on Qwen3.6-35B-A3B (2.55 → 3.94 tok/s at `F = 0.75`, flash reads down a
third; +84% at `F = 1.0`), bootstrap intervals disjoint
([data](docs/bench-data/2026-07-22-drop-cold-experts/findings.md)). It is the one setting whose
output is **not reproducible** — what gets skipped depends on what the cache held — so it has no
rows in the tables below, which are a deterministic protocol, and its quality cost is not yet
quantified. See [docs/expert-dropping.md](docs/expert-dropping.md).
- **Multi-turn sessions and live telemetry**: the model stays loaded across chat turns, and every
run can emit a per-token breakdown of where the time went.
- **Android demo app** ([`examples/android`](examples/android)): a chat app with a live telemetry
@ -127,7 +135,7 @@ phone.
ordinary way (no streaming), which is what the streamed rows are compared against. *k* is how many
experts each token routes to — for us the number of experts, i.e. `n_expert_used` (set with
`--n-expert-used`). Each table shows the model's default width and, where measured, a reduced *k*,
the one lossy setting — see [Turbo top-k — the one lossy option](#turbo-top-k--the-one-lossy-option)
the measured lossy setting — see [Turbo top-k — the measured lossy option](#turbo-top-k--the-measured-lossy-option)
below.
- **tok/s**: generation speed; higher is better.
- **Flash/token**: data read from storage per generated token; lower means the cache is working.
@ -167,7 +175,7 @@ dense weights kept out of the page cache (`--dense-weights anon`) runs it stably
| **streamed, k=6, cache 3000 MiB, 4 lanes, overlap** | **5.8** | 91 MiB | 68% |
All streamed rows use `--overlap --dense-weights anon`. A larger cache is the main lossless lever
(cache 3000 is worth +16% over 2000); the k=6 rows are the one lossy option (turbo top-k, below),
(cache 3000 is worth +16% over 2000); the k=6 rows are the measured lossy option (turbo top-k, below),
worth a further ~16% by routing to six experts instead of eight. The lossless best here is cache
3000 at the model's own width, **5.0 tok/s** — output byte-identical to the resident model.
@ -206,7 +214,7 @@ Gemma keeps more of itself permanently resident, so the 4000 MiB cache fits only
free at launch; cache 2000 + overlap is the dependable everyday setting on this device. Turbo top-k
(k=6) is the fastest here (+22%) but changes the output.
### Turbo top-k — the one lossy option
### Turbo top-k — the measured lossy option
Every model here ships a routing width — the number of experts each token uses (8 for the Qwen and
Gemma models, 4 for gpt-oss). Forcing it lower with `--n-expert-used` cuts both compute and flash
@ -214,9 +222,15 @@ reads; the `k=6` rows folded into the tables above are that knob, measured A/B a
own width. It is worth **+22–24%** on the Qwen and Gemma models, and takes gpt-oss from 1.3 to
**2.2 tok/s** (k=2).
Everything else in this README changes *how* weights are fetched, never the math. This knob changes
*what* the model computes: output differs from the full model and quality can degrade. Judge it on
your own task before relying on it.
Every benchmarked setting other than this one changes *how* weights are fetched, never the math.
This knob changes *what* the model computes: output differs from the full model and quality can
degrade. Judge it on your own task before relying on it.
It also spends quality indiscriminately: the tail of the routing goes whether or not those experts
were already in RAM, and a resident expert costs no flash read at all.
[Cache-aware dropping](docs/expert-dropping.md) is the experimental answer to that — same kind of
trade, but only where it buys I/O. It has no measured rows here yet, which is why the tables above
are still turbo top-k's.
### What to expect in the app

20
cli/main.cpp
View file

@ -363,6 +363,12 @@ static void print_usage(const char * argv0) {
" --force-cache allow a cache-mb in the pathological band\n"
" --overlap overlap async expert reads with FFN compute (needs the fork)\n"
" --prefetch K temporally prefetch the next K layers' experts (needs the cache)\n"
" --drop-cold-experts F skip a routed expert that is a cache MISS and carries less than\n"
" F x (1/top-k) of the routing's weight. F in (0, 1]; 1.0 is the\n"
" uniform share and the useful maximum. LOSSY and cache-dependent:\n"
" it changes the output, and not reproducibly. Off by default.\n"
" --drop-no-renorm do not rescale the surviving weights after a drop (A/B)\n"
" --drop-in-prefill drop during prefill too (off: the cold cache makes it expensive)\n"
" --list-archs print supported MoE architectures and exit\n"
"\n"
" Env overrides (flag wins): BMOE_CACHE_MB, BMOE_IO_THREADS, BMOE_PROGRESS, BMOE_OVERLAP, BMOE_PREFETCH, "
@ -478,6 +484,12 @@ int main(int argc, char ** argv) {
cfg.moe.prefetch_layers = std::atoi(next("--prefetch"));
else if (a == "--prefetch-sync") // debug: complete speculative reads synchronously
cfg.moe.prefetch_sync = true;
else if (a == "--drop-cold-experts")
cfg.moe.drop_cold_frac = (float) std::atof(next("--drop-cold-experts"));
else if (a == "--drop-no-renorm")
cfg.moe.drop_renorm = false;
else if (a == "--drop-in-prefill")
cfg.moe.drop_prefill = true;
else if (a == "--list-archs") {
std::printf("supported MoE architectures:\n");
for (int k = 0; k < n_moe_recipes(); ++k)
@ -622,6 +634,14 @@ int main(int argc, char ** argv) {
std::printf("moe-prefetch: %.1f MiB speculative, %lld/%lld experts useful (%.0f%%)\n", s.moe_spec_read_mib,
s.moe_spec_useful, s.moe_spec_experts,
s.moe_spec_experts > 0 ? 100.0 * s.moe_spec_useful / s.moe_spec_experts : 0.0);
// How hard the policy actually bit. The flag sets a threshold, not a drop rate: what gets
// discarded depends on what the cache held, so this is the only honest report of the trade
// a given run made.
if (cfg.moe.drop_cold_frac > 0.0f)
std::printf("moe-drop: %lld/%lld routed experts dropped (%.1f%%), threshold %.2f x uniform\n",
s.experts_dropped, s.experts_routed,
s.experts_routed > 0 ? 100.0 * s.experts_dropped / s.experts_routed : 0.0,
(double) cfg.moe.drop_cold_frac);
}
return 0;
}

29
core/include/bmoe/config.h
View file

@ -94,6 +94,35 @@ struct MoeStreamConfig {
// and the policy the Android app ships by default — the CLI matches it here.
DenseWeightsMode dense_weights = DenseWeightsMode::Anonymous;
// ── cache-aware expert dropping (lossy; opt-in) ──────────────────────────────────
// Skip a routed expert when it is a cache MISS *and* the router weighted it below
// drop_cold_frac × (1 / n_expert_used) — i.e. below that fraction of the uniform share a
// top-k routing would give each expert. 0 (the default) disables it and the engine is
// bit-exact as before.
//
// The asymmetry is the whole idea: an expert already resident costs no flash read, so it
// always runs however small its weight. Quality is spent only where it buys I/O. Because
// the largest weight in a routing is always >= the uniform share, a frac of 1.0 can never
// empty a routing; validate() rejects anything above it, and the implementation additionally pins the
// top-weighted expert so no cell is ever left with nothing to compute.
//
// This changes the output — it is a quality/throughput trade like n_expert_used, not an
// optimisation. Unlike n_expert_used it is *state-dependent*: the same prompt can decode
// differently depending on what the cache happened to hold, so a run is no longer
// reproducible token-for-token. See docs/expert-dropping.md.
float drop_cold_frac = 0.0f;
// Rescale the surviving weights so the routing still sums to what it did before the drop.
// Without it the layer's expert output is systematically scaled down by the discarded mass
// (~10% at frac 1.0), which perturbs the residual stream more than the missing expert does.
bool drop_renorm = true;
// Apply dropping during prefill too. Off by default and deliberately so: the cache is cold
// there, so nearly every expert is a miss and the same threshold discards ~4x the weight
// mass it does in decode (measured; see docs/expert-dropping.md). Prefill is also
// compute-bound, so there is little to win.
bool drop_prefill = false;
// Test/debug only: complete each prefetch's speculative reads synchronously, on the eval
// thread, before returning. This defeats the latency-hiding purpose (the reads no longer
// overlap compute) but makes speculative integration deterministic, so the byte-identity

12
core/include/bmoe/metrics.h
View file

@ -108,6 +108,17 @@ struct RunSummary {
double token_demand_mib = 0.0;
// The widest layer's routed bytes: the mechanical floor a cache must be able to stage.
double layer_demand_mib = 0.0;
// Both measure what reached the streamer. Under MoeStreamConfig::drop_cold_frac that is what a
// token STAGES, not what it routed — a dropped expert is never handed over — so the "floor a
// cache must clear" reading stops being mechanical there: the floor shrinks because the cache
// was small. Size the cache with dropping off, then turn it on.
// Cache-aware expert dropping (zero when --drop-cold-experts is off). Routed counts what the
// router selected across the generation, dropped how much of it the policy declined to read;
// their ratio is the lever's actual bite, which depends on the cache and so cannot be read off
// the flag. Both cover generation only — prefill drops nothing unless armed for it.
long long experts_routed = 0;
long long experts_dropped = 0;
// Temporal prefetch (zero when --prefetch is off): speculative bytes read during generation,
// experts successfully prefetched, and how many of those a later routing actually used.
@ -142,6 +153,7 @@ struct RunInfo {
bool overlap = false;
int prefetch_layers = 0;
std::string dense_weights = "anon"; // dense (non-expert) policy: "mmap" | "warm" | "anon"
float drop_cold_frac = 0.0f; // cache-aware expert dropping threshold (0 = off)
};
// Optional per-token sink (e.g. CSV for benchmarks). The engine calls on_run_info once before the

6
core/include/bmoe/route_trace.h
View file

@ -38,6 +38,12 @@ struct RouteTraceRow {
float weight = 0.0f; // final applied routing weight; NaN if the graph exposed none
uint8_t residency = 0; // RouteResidency
uint64_t expert_bytes = 0; // flash bytes this routing reads; 0 unless residency == route_miss
// Cache-aware dropping (MoeStreamConfig::drop_cold_frac) discarded this routing: the expert was
// a miss weighted below the threshold, so it was never read and its weight was zeroed. `weight`
// and `residency` stay as the ROUTER produced them — the trace records the routing that was
// chosen, and this flag records what the policy then did with it — but expert_bytes is 0,
// because a dropped expert costs no read. Always 0 when dropping is off.
uint8_t dropped = 0;
};
// Run-level facts the rows cannot carry. Emitted once, before any row.

3
core/include/bmoe/runtime.h
View file

@ -5,7 +5,8 @@
// them to the expert source, then greedily generates n_predict tokens — reporting each
// token to the optional callback/sink and returning a RunSummary. Greedy sampling makes
// the output a deterministic function of the graph, which is what the byte-identity
// gates rely on.
// gates rely on — true of every configuration except MoeStreamConfig::drop_cold_frac, which
// decides from live cache state and so is reproducible only within a single run.
#pragma once
#include "bmoe/config.h"

11
core/src/config.cpp
View file

@ -85,6 +85,17 @@ ValidationResult validate(const RunConfig & cfg) {
"speculative reads land in the per-layer cache buffers, which do not exist "
"with the cache off.");
}
if (m.drop_cold_frac > 0.0f && !cache_on) {
return fail("moe.drop_cold_frac requires the LRU cache (cache_mb > 0 or cache_auto): with the "
"cache off every expert is a miss, so the policy stops being cache-aware and "
"degenerates into an unconditional weight cut — which is what n_expert_used already "
"does, without pretending to consult residency.");
}
if (m.drop_cold_frac < 0.0f || m.drop_cold_frac > 1.0f) {
return fail("moe.drop_cold_frac must be in [0, 1] (0 = off). Above 1.0 the threshold can "
"exceed the largest weight in a routing, which would discard every expert of a "
"layer; 1.0 is the uniform share 1/n_expert_used and the useful maximum.");
}
}
return r;

11
core/src/engine/session.cpp
View file

@ -370,6 +370,7 @@ std::unique_ptr<Session> Session::open(const SessionConfig & cfg,
im.hook = std::make_unique<RouterHook>(recipe ? *recipe : MoeRecipe{}, im.n_layer);
im.hook->set_prefetch_layers(cfg.moe.prefetch_layers);
im.hook->set_drop_policy(cfg.moe.drop_cold_frac, cfg.moe.drop_renorm, cfg.moe.drop_prefill);
llama_context_params cparams = llama_context_default_params();
cparams.n_ctx = cfg.n_ctx;
@ -561,6 +562,7 @@ std::unique_ptr<Session> Session::open(const SessionConfig & cfg,
ri.o_direct = cfg.moe.enabled && cfg.moe.o_direct;
ri.overlap = cfg.moe.enabled && cfg.moe.overlap;
ri.prefetch_layers = cfg.moe.enabled ? cfg.moe.prefetch_layers : 0;
ri.drop_cold_frac = cfg.moe.enabled ? cfg.moe.drop_cold_frac : 0.0f;
// The CSV keeps the two familiar flags, derived from the resolved dense-weights policy.
ri.dense_weights = cfg.moe.dense_weights == DenseWeightsMode::Mmap ? "mmap"
: cfg.moe.dense_weights == DenseWeightsMode::Anonymous ? "anon"
@ -757,6 +759,9 @@ RunResult Session::generate(const GenerateRequest & req,
// The frame the I/O rows are stamped with at flush; the other traces carry their own.
int trace_phase = 0, trace_step = 0;
auto trace_begin = [&](int base_pos, int n_tokens, int phase) {
// Not a trace concern, but the same per-decode frame: the drop policy is decode-only
// unless armed for prefill, so it has to be told which phase this batch is.
im.hook->set_batch_phase(phase);
if (im.route_trace) im.hook->begin_trace_batch(base_pos, n_tokens, phase, im.turn);
// A node is computed once for the whole batch, not per token, so a prefill chunk's graph is
// attributed to its last position rather than pretending to split across the chunk.
@ -842,6 +847,10 @@ RunResult Session::generate(const GenerateRequest & req,
long long prev_spec_bytes = moe.enabled ? (long long) im.source.stats().spec_read_bytes : 0;
long long prev_spec_experts = moe.enabled ? im.source.stats().spec_experts : 0;
long long prev_spec_useful = moe.enabled ? im.source.stats().spec_useful : 0;
// Taken after prefill, so the drop counters describe generation — the phase the policy is armed
// for and the one the tok/s number is about.
const long long prev_routed = im.hook->experts_routed();
const long long prev_dropped = im.hook->experts_dropped();
for (int t = 0; t < req.n_predict; ++t) {
// Greedy stays argmax (byte-identical to the resident reference the gates check); with a
@ -931,6 +940,8 @@ RunResult Session::generate(const GenerateRequest & req,
s.moe_spec_experts = st.spec_experts - prev_spec_experts;
s.moe_spec_useful = st.spec_useful - prev_spec_useful;
}
s.experts_routed = im.hook->experts_routed() - prev_routed;
s.experts_dropped = im.hook->experts_dropped() - prev_dropped;
if (sink) sink->on_summary(s);
{

9
core/src/metrics/csv_metrics_sink.cpp
View file

@ -23,9 +23,9 @@ public:
r.model.c_str(), r.arch.c_str(), r.n_layer, r.n_expert, r.n_expert_used, r.n_threads, r.n_ctx);
std::fprintf(f_,
"# moe_stream=%d cache_mb=%d cache_auto=%d cache_ceil_mb=%d force_cache=%d "
"io_threads=%d o_direct=%d overlap=%d prefetch=%d dense_weights=%s\n",
"io_threads=%d o_direct=%d overlap=%d prefetch=%d dense_weights=%s drop_cold_frac=%.4g\n",
r.moe_stream, r.cache_mb, r.cache_auto, r.cache_ceil_mb, r.force_cache, r.io_threads, r.o_direct,
r.overlap, r.prefetch_layers, r.dense_weights.c_str());
r.overlap, r.prefetch_layers, r.dense_weights.c_str(), (double) r.drop_cold_frac);
write_header();
}
@ -49,13 +49,14 @@ public:
"n_prompt=%d load_s=%.3f prefill_s=%.3f prefill_tps=%.2f stall_s/tok=%.3f mgmt_s/tok=%.3f "
"cache_resident_MiB=%.1f cache_budget_MiB=%.1f cache_resizes=%lld "
"spec_read_MiB=%.1f spec_experts=%lld spec_useful=%lld "
"majflt/tok=%.2f cpu_s/tok=%.4f token_demand_MiB=%.1f layer_demand_MiB=%.1f\n",
"majflt/tok=%.2f cpu_s/tok=%.4f token_demand_MiB=%.1f layer_demand_MiB=%.1f "
"experts_routed=%lld experts_dropped=%lld\n",
s.n_generated, s.s_per_token, s.tokens_per_second, s.moe_read_mib, s.moe_io_seconds,
s.moe_compute_s_per_token, s.moe_io_s_per_token, s.cache_hit_pct, s.n_prompt, s.load_seconds,
s.prefill_seconds, s.prefill_seconds > 0 ? s.n_prompt / s.prefill_seconds : 0.0,
s.moe_stall_s_per_token, s.moe_mgmt_s_per_token, s.cache_resident_mib, s.cache_budget_mib,
s.cache_resizes, s.moe_spec_read_mib, s.moe_spec_experts, s.moe_spec_useful, s.majflt_per_token,
s.cpu_s_per_token, s.token_demand_mib, s.layer_demand_mib);
s.cpu_s_per_token, s.token_demand_mib, s.layer_demand_mib, s.experts_routed, s.experts_dropped);
std::fflush(f_);
}

11
core/src/metrics/route_trace_sink.cpp
View file

@ -26,7 +26,7 @@ public:
std::fprintf(f_, "# layer=%zu expert_bytes=%llu dense_bytes=%llu\n", il,
(unsigned long long) s.expert_bytes_per_layer[il], (unsigned long long) dense);
}
std::fprintf(f_, "turn,phase,step,layer,slot,expert,weight,residency,expert_bytes\n");
std::fprintf(f_, "turn,phase,step,layer,slot,expert,weight,residency,expert_bytes,dropped\n");
std::fflush(f_);
}
@ -36,12 +36,13 @@ public:
// A weight the graph never exposed prints as nan, not 0: "unknown" must not read as
// "the router gave this expert no mass".
if (std::isnan(r.weight))
std::fprintf(f_, "%d,%d,%d,%d,%d,%d,nan,%u,%llu\n", r.turn, r.phase, r.step, r.layer, r.slot,
(int) r.expert, (unsigned) r.residency, (unsigned long long) r.expert_bytes);
std::fprintf(f_, "%d,%d,%d,%d,%d,%d,nan,%u,%llu,%u\n", r.turn, r.phase, r.step, r.layer, r.slot,
(int) r.expert, (unsigned) r.residency, (unsigned long long) r.expert_bytes,
(unsigned) r.dropped);
else
std::fprintf(f_, "%d,%d,%d,%d,%d,%d,%.6g,%u,%llu\n", r.turn, r.phase, r.step, r.layer, r.slot,
std::fprintf(f_, "%d,%d,%d,%d,%d,%d,%.6g,%u,%llu,%u\n", r.turn, r.phase, r.step, r.layer, r.slot,
(int) r.expert, (double) r.weight, (unsigned) r.residency,
(unsigned long long) r.expert_bytes);
(unsigned long long) r.expert_bytes, (unsigned) r.dropped);
}
std::fflush(f_); // once per decode, not per row
}

10
core/src/moe/expert_stream_source.cpp
View file

@ -655,8 +655,10 @@ bool ExpertStreamSource::load_layer(int il, const int32_t * ids, int n_ids) {
// Already staged in this batch: still promote so the LRU order reflects the LAST
// token that used this expert (ids arrive token-major), not its first touch — this
// keeps the prompt tail's experts hot across prefill. Reads are scheduled only once
// (the seen_ guard below), so this is bookkeeping only. In decode (n=1) the top-k ids
// are distinct, so this branch never runs and behaviour is unchanged.
// (the seen_ guard below), so this is bookkeeping only. The router's own top-k ids are
// distinct, so in decode (n=1) this used to be unreachable — but cache-aware dropping
// repoints a dropped slot's id at the routing's top expert, which makes duplicates the
// normal case there. Promoting the same entry twice is idempotent, so it stays correct.
if (cache_max_) {
const int32_t id = il * n_expert_ + e;
lru_unlink(id);
@ -822,7 +824,9 @@ bool ExpertStreamSource::load_layer_async(int il, const int32_t * ids, int n_ids
// Promote every touched expert in raw id order (token-major) so the LRU order reflects
// the LAST token that used it, not the sorted/first-touch order staged_ imposes — this
// keeps the prompt tail's experts hot across prefill. Bookkeeping only; every id staged
// above is now valid and linked. In decode (n=1, distinct top-k) this is a no-op reshuffle.
// above is now valid and linked. In decode (n=1) this is a no-op reshuffle when the ids are
// distinct, and an idempotent re-promote of the same entry when cache-aware dropping has
// repointed a slot at the routing's top expert.
// Skipped in load_all (everything is resident, so LRU order is meaningless).
if (!load_all_) {
for (int i = 0; i < n_ids; ++i) {

177
core/src/moe/router_hook.cpp
View file

@ -62,15 +62,27 @@ static bool match_weights(const char * name, int & il_out) {
// (token j at nb[2], slot k at nb[1]) — except the norm variant, whose callback fires on the
// pre-reshape 2-D [nu, nt] (token j at nb[1], slot k at nb[0]). ne[0] == 1 tells the two apart.
// As with the topk ids, these are views: only the strides say where a token's row really starts.
static void gather_weights(const ggml_tensor * t, int nu, int nt, std::vector<float> & out) {
// Where token j's slot k lives inside a weight node. Single source of truth for the layout: the
// reader below and the drop policy's writer must agree, or the policy would zero another token's
// slot. Returns a mutable pointer; the gather takes a const tensor and only reads through it.
static float * weight_at(const ggml_tensor * t, int j, int k) {
const bool three_d = t->ne[0] == 1;
const size_t tok_nb = three_d ? t->nb[2] : t->nb[1];
const size_t slot_nb = three_d ? t->nb[1] : t->nb[0];
return (float *) ((char *) t->data + (size_t) j * tok_nb + (size_t) k * slot_nb);
}
static void gather_weights(const ggml_tensor * t, int nu, int nt, std::vector<float> & out) {
out.assign((size_t) nu * nt, 0.0f);
for (int j = 0; j < nt; ++j)
for (int k = 0; k < nu; ++k)
out[(size_t) j * nu + k] =
*(const float *) ((const char *) t->data + (size_t) j * tok_nb + (size_t) k * slot_nb);
out[(size_t) j * nu + k] = *weight_at(t, j, k);
}
// Same, for the selected-expert ids. The node is a VIEW of the full argsort, so the row stride is
// nb[1] (n_expert * 4), not n_expert_used * 4 — see the gather at the topk node.
static int32_t * id_at(ggml_tensor * t, int j, int k) {
return (int32_t *) ((char *) t->data + (size_t) j * t->nb[1] + (size_t) k * t->nb[0]);
}
void RouterHook::begin_capture() {
@ -83,6 +95,114 @@ void RouterHook::end_capture() {
capturing_ = false;
}
void RouterHook::set_drop_policy(float frac, bool renorm, bool in_prefill) {
drop_frac_ = frac > 0.0f ? frac : 0.0f;
drop_renorm_ = renorm;
drop_prefill_ = in_prefill;
term_node_.assign(n_layer_ > 0 ? n_layer_ : 0, std::string{});
drop_ = PendingDrop{};
chain_last_.clear();
experts_routed_ = experts_dropped_ = 0;
}
// Is dropping live for the batch being decoded? Needs a source to ask about residency, a non-zero
// threshold, and — unless armed for prefill — a decode batch: with a cold cache the same threshold
// discards several times the weight mass for a phase that is not I/O-bound anyway.
bool RouterHook::drop_armed() const {
return drop_frac_ > 0.0f && source_ != nullptr && (batch_phase_ == 1 || drop_prefill_);
}
// Apply the policy to the layer held in drop_, then load only what survives.
//
// `wt` is the terminal node of the weight chain: the weights as the expert matmul will apply them.
// Both edits happen here, before any node consumes them:
// - the dropped slot's weight is zeroed (and, with renorm, the survivors are scaled back up so
// the routing keeps the total mass it had);
// - the dropped slot's ID is repointed at the routing's top-weighted expert. That second edit is
// not cosmetic. An expert we decline to read may sit in a reserved-but-uncommitted slot, and
// mul_mat_id would still touch it; pointing the slot at an expert that is certainly resident
// makes the kernel read valid memory and multiply it by exactly zero. It costs a duplicate
// matmul, which is the right trade on a decode bound by flash rather than arithmetic.
// The top-weighted expert is never dropped, so a routing always keeps at least one live expert
// whatever the threshold — the guarantee does not rest on frac <= 1 alone.
void RouterHook::apply_drop(ggml_tensor * wt) {
PendingDrop & D = drop_;
const int nu = D.nu, nt = D.nt;
gather_weights(wt, nu, nt, drop_w_);
// Classify against the cache BEFORE anything is loaded; settle landed prefetches first, or an
// expert a prefetch correctly guessed would look like a miss and be dropped for nothing.
source_->settle_spec();
drop_res_.assign(drop_ids_.size(), (uint8_t) 0);
source_->query_residency(D.layer, drop_ids_.data(), (int) drop_ids_.size(), drop_res_.data());
const float thr = drop_frac_ / (float) nu; // frac of the uniform share each of k experts would get
const bool tracing = trace_on_ && pending_.layer == D.layer;
drop_mask_.assign((size_t) nu * nt, (uint8_t) 0);
for (int j = 0; j < nt; ++j) {
const size_t row = (size_t) j * nu;
int best = 0;
float total = 0.0f;
for (int k = 0; k < nu; ++k) {
total += drop_w_[row + k];
if (drop_w_[row + k] > drop_w_[row + best]) best = k;
}
const int32_t best_id = drop_ids_[row + best];
float kept = 0.0f;
int n_dropped = 0;
for (int k = 0; k < nu; ++k) {
const size_t idx = row + k;
const bool drop = k != best && drop_res_[idx] == route_miss && drop_w_[idx] < thr;
if (!drop) {
kept += drop_w_[idx];
continue;
}
*weight_at(wt, j, k) = 0.0f;
*id_at(D.ids, j, k) = best_id;
drop_ids_[idx] = best_id;
drop_mask_[idx] = 1;
++n_dropped;
}
experts_dropped_ += n_dropped;
// Restore the routing's total mass. Without this the layer's expert output is scaled down
// by whatever was discarded, which perturbs the residual stream in a direction the model
// never sees in training — a systematic shrink, unlike the one missing contribution.
if (drop_renorm_ && n_dropped > 0 && kept > 0.0f) {
const float g = total / kept;
for (int k = 0; k < nu; ++k)
if (!drop_mask_[row + k]) *weight_at(wt, j, k) *= g;
}
}
if (tracing) pending_.dropped = drop_mask_;
source_->load_layer(D.layer, drop_ids_.data(), (int) drop_ids_.size());
D.deferred = false;
}
// Finish with the layer whose topk we last saw: record which node ended its weight chain, so the
// next graph can decide there, and make sure nothing was left waiting on a node that never came.
void RouterHook::close_drop_layer() {
PendingDrop & D = drop_;
if (D.layer < 0) return;
if (D.layer < (int) term_node_.size() && term_node_[D.layer].empty() && !chain_last_.empty())
term_node_[D.layer] = chain_last_;
if (D.deferred && source_) {
// The node we learned as terminal did not appear this time, so the deferral was never
// honoured and this layer's matmul has already run against slots nothing loaded. Load the
// routing now to keep the cache's accounting straight, and — more importantly — FORGET the
// terminal node, so the next graph re-learns it and loads at the topk node meanwhile.
// Deferring again on the same stale guess would repeat the fault every single token; one
// bad layer in one token is recoverable, a standing bet against a graph that moved is not.
source_->load_layer(D.layer, drop_ids_.data(), (int) drop_ids_.size());
if (D.layer < (int) term_node_.size()) term_node_[D.layer].clear();
D.deferred = false;
}
D.layer = -1;
}
void RouterHook::set_trace(bool on) {
trace_on_ = on;
pending_ = PendingLayer{};
@ -143,7 +263,11 @@ void RouterHook::flush_pending() {
r.expert = P.ids[idx];
r.weight = have_w ? P.weights[idx] : std::numeric_limits<float>::quiet_NaN();
r.residency = idx < P.residency.size() ? P.residency[idx] : (uint8_t) 0;
if (r.residency == route_miss && charged_.insert(r.expert).second) r.expert_bytes = ebytes;
r.dropped = idx < P.dropped.size() ? P.dropped[idx] : (uint8_t) 0;
// A dropped routing is never read, so it is neither charged nor allowed to claim the
// charge for its expert: if another token of the batch routes the same expert and keeps
// it, that routing pays the read.
if (!r.dropped && r.residency == route_miss && charged_.insert(r.expert).second) r.expert_bytes = ebytes;
trace_rows_.push_back(r);
}
}
@ -277,9 +401,12 @@ bool RouterHook::on_eval(ggml_tensor * t, bool ask) {
// ── stream: the routing nodes get the single-node barrier so we see the selected ids ──
int il = -1;
const bool is_topk = std::sscanf(t->name, "ffn_moe_topk-%d", &il) == 1 && il >= 0;
// Only a traced run asks for the weight nodes: each extra ask is another barrier.
// The weight nodes are asked for by a traced run, and by the drop policy, which decides on the
// weights the matmul will actually apply. Each extra ask is another barrier — a handful per MoE
// layer, on tensors of a few floats — so neither is on by default.
int wl = -1;
const bool is_weights = trace_on_ && match_weights(t->name, wl);
const bool want_weights = trace_on_ || drop_armed();
const bool is_weights = want_weights && match_weights(t->name, wl);
// The compute trace wants every node isolated — or, at layer granularity, only the first
// node of each layer: the cursor advances on the ask stream (every node passes through
// here), so one isolation request per layer transition. Layerless names (embeddings, the
@ -302,15 +429,29 @@ bool RouterHook::on_eval(ggml_tensor * t, bool ask) {
}
// Weights follow their layer's topk, so the pending record is already open; keep the last
// one offered (match_weights explains why) and let the flush read it.
// one offered (match_weights explains why) and let the flush read it. This runs BEFORE the drop
// policy edits the same tensor, so the trace records the routing the router produced, not the
// one the policy left behind — `dropped` is what says which is which.
if (is_weights && t->data && t->type == GGML_TYPE_F32 && pending_.layer == wl && pending_.nu > 0)
gather_weights(t, pending_.nu, pending_.nt, pending_.weights);
// Learn which node ends this layer's weight chain, and — once known — use it as the point where
// the routing is decided: everything the drop policy needs is final here, and nothing has
// consumed it yet. The learning pass and the deferral are the same walk, so a layer whose chain
// shape the hook has not seen yet simply keeps the undropped behaviour.
if (is_weights && t->data && t->type == GGML_TYPE_F32 && drop_.layer == wl) {
chain_last_ = t->name;
if (drop_.deferred && wl >= 0 && wl < (int) term_node_.size() && term_node_[wl] == t->name) apply_drop(t);
}
if (source_ && is_topk && t->data && t->type == GGML_TYPE_I32) {
// selected_experts is [n_expert_used, n_tokens] but a VIEW of the full argsort
// [n_expert, n_tokens]: its row stride is nb[1] (= n_expert*4), not
// n_expert_used*4. Gather respecting the strides — a flat read would grab token
// 0's sorted tail as token 1's experts, corrupting the KV cache.
// The previous layer's weight chain has been fully offered by now.
close_drop_layer();
gathered_.clear();
const int nu = (int) t->ne[0], nt = (int) t->ne[1];
for (int j = 0; j < nt; ++j)
@ -325,6 +466,7 @@ bool RouterHook::on_eval(ggml_tensor * t, bool ask) {
pending_.nt = nt;
pending_.ids = gathered_;
pending_.weights.clear();
pending_.dropped.clear();
// Classify against the cache BEFORE load_layer makes these experts resident —
// afterwards everything reads as a hit. Settle landed prefetches first, or an expert
// a prefetch correctly guessed would be recorded as a miss.
@ -333,7 +475,26 @@ bool RouterHook::on_eval(ggml_tensor * t, bool ask) {
source_->query_residency(il, gathered_.data(), (int) gathered_.size(), pending_.residency.data());
}
source_->load_layer(il, gathered_.data(), (int) gathered_.size());
// Count what the ROUTER selected, here rather than inside apply_drop: a layer that is not
// deferred yet (the first graph, or a phase the policy is not armed for) still routed these
// experts, and a denominator that skipped them would report the drop rate as a fraction of
// the wrong thing.
if (drop_frac_ > 0.0f) experts_routed_ += (long long) gathered_.size();
// Open the layer for the drop policy. Deferring the load is only safe once this layer's
// terminal weight node is known — otherwise there is no callback left to decide in, and the
// expert matmul would run against slots nothing loaded. First graph of a run: load here.
const bool defer = drop_armed() && il >= 0 && il < (int) term_node_.size() && !term_node_[il].empty();
drop_.layer = il;
drop_.nu = nu;
drop_.nt = nt;
drop_.ids = t;
drop_.deferred = defer;
chain_last_.clear();
if (defer)
drop_ids_ = gathered_;
else
source_->load_layer(il, gathered_.data(), (int) gathered_.size());
// Temporal prefetch: hint the next K layers with what the PREVIOUS token routed there,
// to be read on idle lanes while this layer computes; then record this layer's routing

50
core/src/moe/router_hook.h
View file

@ -73,6 +73,22 @@ public:
// has returned. Off by default: asking for the extra nodes costs a barrier per layer.
void set_trace(bool on);
// ── cache-aware expert dropping (lossy; see MoeStreamConfig::drop_cold_frac) ──────
// `frac` > 0 arms the policy: a routed expert that is a cache MISS and carries less than
// frac × (1/n_expert_used) of the routing's weight is discarded — not read, weight zeroed,
// its slot pointed at the routing's top-weighted expert so the matmul still reads memory
// that is certainly resident. `renorm` rescales the survivors to preserve the routing's
// total mass. Off (frac == 0) the hook behaves exactly as before, bit for bit.
void set_drop_policy(float frac, bool renorm, bool in_prefill);
// Which phase the batch being decoded belongs to (0 = prefill, 1 = decode). The drop policy
// is decode-only unless armed for prefill, and unlike the traces it must know this on every
// run, so it cannot ride on begin_trace_batch.
void set_batch_phase(int phase) { batch_phase_ = phase; }
long long experts_routed() const { return experts_routed_; }
long long experts_dropped() const { return experts_dropped_; }
// ── compute trace (diagnostics; see bmoe/decode_trace.h) ────────────────────────
// When on, the hook asks for EVERY node, which makes ggml compute and synchronize each one
// alone — so the wall delta between consecutive callbacks is that node's real compute time,
@ -119,8 +135,12 @@ private:
std::vector<int32_t> ids;
std::vector<float> weights;
std::vector<uint8_t> residency;
std::vector<uint8_t> dropped; // set by the drop policy; all zero when it is off
};
void flush_pending();
bool drop_armed() const;
void apply_drop(ggml_tensor * weights);
void close_drop_layer();
void ctrace_close_segment(int interval_layer, const char * tail_name);
// Stored by value, not by reference: the caller often constructs us from a temporary
@ -140,6 +160,36 @@ private:
int prefetch_layers_ = 0;
std::vector<std::vector<int32_t>> prev_ids_;
// Cache-aware dropping. Inert unless drop_frac_ > 0.
//
// The decision needs the FINAL router weights, and those are produced several nodes after the
// topk that opens the layer — so load_layer() is postponed from the topk node to the terminal
// node of the layer's weight chain, and the ids/weights are edited there, before the expert
// matmul consumes either. Which node is terminal depends on the model's gating (norm, softmax,
// scaled, or none of them), so it is LEARNED from the graph rather than tabulated per
// architecture: term_node_[il] fills in on the first graph, and until it does the layer loads
// at its topk node undropped, exactly as with the policy off. That costs the first token of a
// run its dropping and nothing else.
float drop_frac_ = 0.0f;
bool drop_renorm_ = true;
bool drop_prefill_ = false;
int batch_phase_ = 1; // 0 prefill, 1 decode
long long experts_routed_ = 0, experts_dropped_ = 0;
struct PendingDrop {
int layer = -1;
int nu = 0, nt = 0;
ggml_tensor * ids = nullptr; // the topk view, rewritten in place for dropped slots
bool deferred = false; // true when load_layer is waiting for the terminal weight node
};
PendingDrop drop_;
std::vector<std::string> term_node_; // per layer, "" until learned
std::string chain_last_; // last weight node seen for drop_.layer while its chain runs
std::vector<int32_t> drop_ids_; // this layer's routed ids, kept across the deferral
std::vector<float> drop_w_; // scratch: the final weights
std::vector<uint8_t> drop_res_; // scratch: residency of each routed id
std::vector<uint8_t> drop_mask_; // scratch: which slots this layer dropped
// Route trace. All of this is inert unless trace_on_.
bool trace_on_ = false;
int trace_base_pos_ = 0, trace_batch_n_ = 1, trace_phase_ = 0, trace_turn_ = 0;

3
docs/README.md
View file

@ -21,7 +21,8 @@ for the idea the project is built on.
| [telemetry.md](telemetry.md) | The `BMOE_*` line protocol and CSV schema — the integration contract. |
| [session.md](session.md) | Session lifecycle, KV prefix reuse, cancellation. |
| [cache-sizing.md](cache-sizing.md) | `--cache-mb auto`, the cache ceiling, and dense warm-up. |
| [prefetch.md](prefetch.md) | `--prefetch K`: the design and why it cannot change output. |
| [prefetch.md](prefetch.md) | `--prefetch K`: the design and why it cannot change output (with the lossy knobs off). |
| [expert-dropping.md](expert-dropping.md) | `--drop-cold-experts F`: spending quality only where it buys a flash read, and why it is the one setting whose output is not reproducible. |
| [android-memory.md](android-memory.md) | What reclaims the engine's memory on a phone, which levers exist (almost none), and why the cache hit rate is what the kernel judges you by. |
| [pressure.md](pressure.md) | Cache policy under memory pressure: why an unaffordable budget starts a reclaim war, why the adaptive governor was retired, and what the fixed `--cache-mb` / `--dense-weights` levers do. |

9
docs/architecture.md
View file

@ -45,8 +45,11 @@ Streaming experts serially needs three things from the inference engine. All thr
already public in llama.cpp:
1. **A hook at routing time.** `llama_context_params.cb_eval` is called for every graph
node. We ask for only the routing nodes (`ffn_moe_topk-<il>`); ggml computes and
node. We ask for the routing nodes (`ffn_moe_topk-<il>`); ggml computes and
synchronizes each alone, then calls us back with the selected expert ids materialized.
The route trace and [cache-aware dropping](expert-dropping.md) additionally ask for each
layer's `ffn_moe_weights*-<il>` chain — and dropping is the one path that *writes into* a
graph tensor's contents rather than only rebinding `->data`. See [seam.md](seam.md).
2. **The expert tensor pointers.** During a one-token warm-up we scan each graph node's
sources for tensors named `blk.<il>.ffn_{gate,up,down}_exps.weight` and record the
live `ggml_tensor*`. We then rebind their `->data`.
@ -89,4 +92,6 @@ The composition root is `Session` (core/src/engine/session.cpp):
so the gates and the interactive session share the same code path.
Greedy sampling makes the output a deterministic function of the graph — the property the
[byte-identity gates](../tests/moe_gates.cpp) assert.
[byte-identity gates](../tests/moe_gates.cpp) assert. That holds with the lossy knobs off. Under
[`--drop-cold-experts`](expert-dropping.md) the hook edits routing weights from live cache state,
which is not in the graph, so output becomes a function of the graph *and* the run's history.

1361
docs/bench-data/2026-07-22-drop-cold-experts/bmoe-20260722-134209.csv Normal file
View file

File diff suppressed because it is too large Load diff

1359
docs/bench-data/2026-07-22-drop-cold-experts/bmoe-20260722-135113.csv Normal file
View file

File diff suppressed because it is too large Load diff

1355
docs/bench-data/2026-07-22-drop-cold-experts/bmoe-20260722-140454.csv Normal file
View file

File diff suppressed because it is too large Load diff

1474
docs/bench-data/2026-07-22-drop-cold-experts/bmoe-20260722-142300.csv Normal file
View file

File diff suppressed because it is too large Load diff

79
docs/bench-data/2026-07-22-drop-cold-experts/findings.md Normal file
View file

@ -0,0 +1,79 @@
# Cache-aware expert dropping — measured POSITIVE on device (2026-07-22)
**Verdict: `--drop-cold-experts` is a real throughput win, and the offline replay understated it at
full strength. 0.75 and 1.00 are each separable from the baseline and from each other; 0.50 does
nothing measurable. Quality was not measured and remains the open question.**
## Setup
In-app, one device, `Qwen3.6-35B-A3B-Q4_K_M` (`qwen35moe`, 40 layers, 256 experts, top-k 8), cache
3000 MiB, `--dense-weights ahwb`, overlap on, prefetch off, 4 compute + 4 read lanes, ~1350-1470
token generations. Every cell differs in `drop_cold_frac` **only**; the four `# moe_stream=…`
preambles in this directory are identical apart from that field.
## Whole-run numbers
| `drop_cold_frac` | tok/s | flash read | I/O s/token | cache hit | routings dropped |
|---|---|---|---|---|---|
| 0 (off) | 2.549 | 248.2 GiB | 0.299 | 67.8% | 0 |
| 0.50 | 2.564 | 231.4 GiB | 0.279 | 69.1% | 2.7% |
| 0.75 | **3.938** | 162.8 GiB | 0.190 | 77.2% | 14.2% |
| 1.00 | **4.702** | 48.1 GiB | 0.070 | 90.7% | 28.4% |
Per-token, first 50 tokens trimmed (cache warm-up, plus the one token the hook spends learning the
graph), 2000-sample bootstrap on the mean:
| `drop_cold_frac` | median ms/token | mean ms/token | 95% CI on the mean |
|---|---|---|---|
| 0 | 249.5 | 391.1 | [375.3, 407.1] |
| 0.50 | 263.2 | 391.6 | [375.3, 408.5] |
| 0.75 | 202.8 | 253.5 | [246.9, 260.4] |
| 1.00 | 153.4 | 214.9 | [206.2, 225.0] |
**Every pair is disjoint except 0 vs 0.50, which overlaps.** At half the uniform share the policy
finds almost nothing to drop (2.7% of routings) and buys nothing — exactly what the replay
predicted, and a useful negative control: the machinery costs nothing measurable when it is not
firing.
## Why this is not the device drifting
Run order was **1.00 (13:42), off (13:51), 0.50 (14:04), 0.75 (14:23)** — deliberately not in
threshold order. The two fastest cells are the **first and the last**. Thermal drift or accumulated
memory pressure would make the last cell the worst; it is the second best. The baseline also ran
early, on a relatively cool device, and still lost.
The mechanism is monotone in the threshold even though the run order is not: flash read
248 → 231 → 163 → 48 GiB and I/O 0.299 → 0.279 → 0.190 → 0.070 s/token order themselves perfectly by
`drop_cold_frac`. That dose-response is what run order cannot fake.
## The replay was conservative, not optimistic
`docs/expert-dropping.md` calls the offline replay an **upper bound**, because it cannot model the
cache changing as a result of dropping. At `F = 0.75` it was accurate (predicted ~37% of reads
avoided, measured 34%). At `F = 1.0` it **understated**: predicted 66%, measured **81%**.
The extra comes from a compounding effect the static replay is blind to: reads avoided free cache
capacity, which raises the hit rate, which leaves fewer misses to drop in the first place. The
caveat in the doc should be read as "the bound holds where dropping is light, and is pessimistic
where it is heavy".
## Read `cache_hit_pct` with the documented caveat
The rise from 67.8% to 90.7% is **not** the cache serving more. A dropped routing is a miss that is
never looked up, so it leaves both sides of the ratio (see `docs/telemetry.md`). `read_MiB` is the
honest number here, and it is unaffected by that accounting: 248 GiB → 48 GiB is real.
`majflt/token` (224 / 178 / 14 / 611) does **not** order by threshold and is dominated by what the
device's memory looked like when each run started. Nothing should be concluded from it here.
## What this does not show
- **Quality is unmeasured.** At `F = 0.75` the policy discards 14.2% of routings and at `F = 1.0`
28.4%. There is no perplexity number and no side-by-side comparison in this experiment. Throughput
is settled; whether the output holds up is not.
- **One model, one device, one run per cell.** The confidence intervals are within-run (over ~1300
tokens), not across repeats. The reversed-order pair is owed, as it was for `ahwb`.
- **Top-k 8 only.** Qwen3.6 routes 8 of 256. A model routing 2 or 4 experts has a much larger uniform
share, so the same `F` bites differently there. gpt-oss in particular is unmeasured.
- **Prefetch was off.** The interaction between speculation and dropping (a correct guess un-drops an
expert) is untested.

11
docs/benchmark-method.md
View file

@ -47,6 +47,7 @@ Vary one axis at a time:
| threads (-t) | 2, 4, 8 | U-shape, 4 optimal, 8 regresses |
| overlap | off, on | net gain **only over a warm cache** (hides residual flash wait behind compute); a net loss on a cold cache-0 stream, where I/O dwarfs compute |
| n-expert-used | default, 6 | fewer active experts cut compute + I/O ~linearly (8→6 ≈ −25%), changes the output |
| drop-cold-experts | off, 0.75, 1.0 | the second lossy axis, and the only **non-deterministic** one: what is skipped depends on cache state, so cells are noisier and the drop rate must be reported with the tok/s. Needs the cache on |
| dense-weights | warm, anon | decisive well past RAM, near-neutral near it: on gpt-oss (5.2× RAM) `anon` drops majflt/token from the hundreds to **6–10** and compute with it; on Qwen (1.64× RAM) there is little dense-fault pressure to remove. Watch `majflt/token`, not just tok/s |
When sweeping `--n-expert-used`, run it as a **matched A/B against the model's own default**
@ -125,6 +126,16 @@ So:
Re-run such a cell; do not publish it. And sanity-check any matrix by **reversing the run order** —
cells that move were measuring device state.
**The reversal check does not work under `--drop-cold-experts`.** There a cell can move because the
*drop rate* moved — the policy reads live cache state, so the same command legitimately discards a
different number of experts on a different run. That is the feature working, not the device
contaminating the cell, and the two tells above cannot tell them apart. Always record
`experts_dropped`/`experts_routed` (or the `moe-drop:` line) next to the tok/s: a dropping cell
without its drop rate is uninterpretable, because the flag fixes a threshold and not a rate. Note
also that a dropping run pays the same extra per-MoE-layer barriers a route-traced run does, so an
A/B against `--n-expert-used` is not overhead-matched — see
[expert-dropping.md](expert-dropping.md).
### Caveats
- **Thermal.** Sustained decode throttles. Warm up, then measure a steady window; discard

195
docs/expert-dropping.md Normal file
View file

@ -0,0 +1,195 @@
# Cache-aware expert dropping
`--drop-cold-experts F` skips a routed expert when it is **not in the cache** *and* the router
weighted it below `F × (1 / top-k)` — that is, below `F` of the uniform share each of the `k`
selected experts would get if the router split its mass evenly. Off by default.
It is the second lossy knob in the engine, after
[turbo top-k](../README.md#turbo-top-k--the-measured-lossy-option), and it exists because the first one
spends quality in a place it does not have to.
## Why cache state belongs in the decision
`--n-expert-used k` drops the routing's tail unconditionally: slot 7 and slot 8 go, whether or not
they were already sitting in RAM. But an expert that is already resident costs **no flash read** —
and on a streamed decode, flash reads are what the token is waiting for
([decode is I/O-bound](benchmarks.md)). Dropping a resident expert pays quality for nothing.
Turn that around and the policy writes itself: **spend quality only where it buys I/O**. Keep every
resident expert however small its weight; consider dropping only the ones that would cost a read,
and only when the router says they barely matter.
## What it costs and what it buys
Replayed over the committed route traces (`docs/bench-data/2026-07-15-route-trace/`), decode phase,
threshold at the uniform share (`F = 1.0`):
| policy | flash reads avoided | router weight discarded |
|---|---|---|
| `--drop-cold-experts 1.0` | **66%** | **9.5%** |
| `--n-expert-used 5` | 23% | 10.6% |
| `--n-expert-used 3` | 59% | 36.8% |
(Qwen3-30B-A3B at k=6; Gemma-4-26B-A4B is within a point and a half on both columns: 67.4% / 8.2%. On gpt-oss-120b at k=2 the
policy matches `--n-expert-used 1`'s read saving while discarding 25% of the weight mass instead of
42%.)
At a comparable quality cost the cache-aware policy avoids roughly **three times** the reads. The
reason is visible in the third column of the trace: about 80% of decode routings are cache hits, and
the policy leaves every one of them alone.
`F` is a curve, not a switch. At `F = 0.75` the same model trades 4.4% of the weight mass for 37% of
the reads — still better than `--n-expert-used 5` on **both** axes.
These are replay numbers, and they were an **upper bound** only where dropping is light — see the
measurement below, where at full strength reality beat the prediction.
## Measured on device
Qwen3.6-35B-A3B (top-k 8 of 256), in-app, cache 3000 MiB, one variable changed:
| `F` | tok/s | flash read | routings dropped |
|---|---|---|---|
| off | 2.549 | 248 GiB | 0 |
| 0.50 | 2.564 | 231 GiB | 2.7% |
| 0.75 | **3.938** | 163 GiB | 14.2% |
| 1.00 | **4.702** | 48 GiB | 28.4% |
Per-token bootstrap intervals separate every pair **except off vs 0.50**, which overlaps: at half the
uniform share the policy finds almost nothing to drop and buys nothing. That is also the useful
negative control — the machinery costs nothing measurable when it is not firing.
Run order was 1.00, off, 0.50, 0.75, so the two fastest cells are the first and the *last*: thermal
drift would have made the last the worst. Full data, caveats and the run-order argument:
[bench-data/2026-07-22-drop-cold-experts](bench-data/2026-07-22-drop-cold-experts/findings.md).
**Quality was not measured.** Throughput is settled; whether the output holds up at 14% or 28%
discarded routings is not, and no perplexity number exists for this knob yet.
## Two properties worth knowing
**A routing is never emptied.** The largest weight in a routing is always at least the uniform
share, so at `F ≤ 1.0` the top expert can never fall below the threshold. `validate()` rejects
`F > 1.0` for that reason, and the implementation additionally pins the top-weighted expert, so the
guarantee does not rest on the bound alone.
**It requires the expert cache.** With `--cache-mb 0` every expert reads as a miss, so the policy
would stop being cache-aware and become an unconditional weight cut — which is what
`--n-expert-used` already does, without claiming to consult residency. `validate()` rejects the
combination, the same way it rejects `--prefetch` without a cache.
**It changes what `--prefetch` means.** Speculation is normally output-neutral by construction. Here
residency is an *input* to the policy, so a correct guess un-drops an expert that would otherwise
have been discarded: prefetch depth becomes an output-affecting setting. The decision point also
settles pending speculation a few nodes after it was issued, which shortens the overlap window the
prefetch exists for — treat the two as interacting, not composable.
**Prefill is excluded by default.** With a cold cache almost every expert is a miss, and the same
threshold discards ~42% of the weight mass instead of ~9%. Prefill is compute-bound anyway, so there
is little to win. `--drop-in-prefill` arms it for experiments.
## The output is no longer reproducible
This is the real novelty, and the reason the flag is off by default and named the way it is.
`--n-expert-used` is lossy but **deterministic**: same prompt, same config, same tokens. Dropping is
lossy and **state-dependent** — what gets discarded depends on what the cache happened to hold,
which depends on everything decoded before it. The same prompt can produce different text across
runs, and a benchmark cell is noisier because the drop rate itself varies.
The greedy byte-identity gates therefore do not cover the policy's output, and cannot: there is
nothing stable to compare against. They cover the machinery instead (see below).
## How it is implemented
The decision needs the **final** router weights, and those are produced several graph nodes after
the topk node where the streamer normally loads. So with the policy armed, `load_layer()` is
postponed from the topk node to the terminal node of the layer's weight chain — the last node before
the expert matmul consumes either the ids or the weights.
Which node is terminal depends on the model's gating (`_norm`, `_softmax`, `_scaled`, or none), so
the hook **learns** it from the graph instead of carrying a per-architecture table: the first graph
of a run records the chain, and dropping starts from the second. A layer whose shape has not been
seen yet simply loads at its topk node, undropped. That costs a run its first token's dropping and
nothing else, and it keeps [hard rule 4](../CLAUDE.md) — no model-specific constants in the
streaming path.
At the decision point two edits happen, both before anything reads them:
1. the dropped slot's **weight is zeroed**, and with `drop_renorm` (default on) the survivors are
scaled so the routing keeps its original total mass;
2. the dropped slot's **id is repointed** at the routing's top-weighted expert.
The second edit is not cosmetic. An expert the engine declines to read may sit in a
reserved-but-uncommitted slot, and `mul_mat_id` would still touch it. Pointing the slot at an expert
that is certainly resident makes the kernel read valid memory and multiply it by exactly zero. It
costs a duplicate matmul — the right trade on a decode bound by flash rather than arithmetic.
Renormalisation matters more than it looks: without it the layer's expert output is systematically
scaled down by the discarded mass, a perturbation of the residual stream the model never sees in
training. `--drop-no-renorm` exists to A/B that claim.
Cost of the extra barriers: the policy asks for each layer's weight nodes, a handful more
synchronisation points per MoE layer on tensors of a few floats. The same asks a route trace makes.
## Measuring it
The engine reports what the policy actually did, which the flag alone cannot tell you — the
threshold is fixed, the drop rate is not:
```
moe-drop: <dropped>/<routed> routed experts dropped (<pct>%), threshold <F> x uniform
```
The route trace gains a `dropped` column: `weight` and `residency` stay as the **router** produced
them, `dropped` records what the policy then did, and `expert_bytes` is 0 for a dropped routing
because it costs no read. That is enough to replay a real run against the offline model and check
whether the upper bound held. See [telemetry.md](telemetry.md).
## Gates
`bmoe_moe_gates` covers the machinery, not the policy's output:
- **G8a** — with a threshold below any weight the router can produce, nothing is dropped and the
output is **byte-identical** to the undropped stream. This proves the deferral and the learned
terminal node are transparent, separating "the plumbing is correct" from "the policy is lossy" —
a regression in the first would otherwise hide behind the expected difference. **G8a'** asserts
the count separately (`experts_routed > 0`, `experts_dropped == 0`), so "a weight happened to fall
under the threshold" fails legibly instead of as a mysterious byte mismatch.
- **G8b** — at full strength against a cache small enough to be evicting constantly, so dropped
experts really do land on slots the cache has released. Generation still completes: the id
repointing means no matmul ever reads reserved-but-uncommitted memory. (The gates deliberately do
*not* run this with the cache off — there the shared-slot path has no uncommitted memory, so the
safety property the repointing exists for would go untested.)
- **G8c** — forcing top-k to 1 makes every routed expert the top one, so dropping must be a no-op at
any threshold and the output must match the undropped k=1 run byte for byte. This pins both the
top-expert guarantee and the fact that the threshold is taken against the **effective** top-k
discovered at runtime — a hardcoded width would not survive the override.
## Defaults, and where the numbers do and do not come from
The **CLI defaults it off**, and will keep doing so: the byte-identity gates need a deterministic
default, and an instrument should not quietly change the thing it measures.
The **app ships it at 75%**, under **Speed / quality → Drop cold experts**, with rungs 50 / 75 /
100 as percentages of the uniform share. It is disabled there in mmap mode and with the cache off,
the same two conditions `validate()` enforces.
75% rather than 100% is the deliberate choice: it is where the measured curve turns, taking the
larger part of the throughput win (+55%) for half the discarded routings (14% against 28%). With
quality unquantified, the conservative end of a measured range is the defensible default.
The **CLI keeps defaulting to off**, and should: the byte-identity gates need a deterministic
default, and an instrument that quietly changes what it measures is not an instrument.
What is still owed before this is recommended beyond that:
- a published decode A/B against `--n-expert-used` at matched tok/s, with the device state recorded
the way [benchmark-method.md](benchmark-method.md) requires;
- a quality comparison at that matched speed — the whole thesis is that this knob buys the same
throughput for less damage, and only a side-by-side can support it;
- a re-run of the replay against a real traced run with the `dropped` column, to see how far the
static upper bound overstated the win.
The [`layer-lfu` entry in the roadmap](roadmap.md) is the standing reminder for why the third one
matters: it simulated exactly as predicted and was ~30% slower in reality.

5
docs/limitations.md
View file

@ -18,6 +18,11 @@ serial path, and only a single ~25-line hook (with an explicit sunset) for the o
## Limitations
- **One setting makes output non-reproducible.** Every other knob is deterministic given a
configuration: `--n-expert-used` changes the output, but changes it the same way on every run.
[`--drop-cold-experts`](expert-dropping.md) decides per routing from live cache state, so the
same prompt and the same flags can decode differently run to run, and the byte-identity gates
cannot cover its output — only its machinery. Off by default in the CLI.
- **n=1 only.** The expert sparsity exists only for single-token decode, so streaming is
incompatible with speculative decoding or batching. Prefill streams the union of the
prompt's routed experts (still far below the full bank, but larger than one token's).

5
docs/moe-streaming.md
View file

@ -32,7 +32,10 @@ Ordering is guaranteed by ggml's eval-callback loop: the node we mark is compute
buffers until this layer's matmul has synchronized. Correct on any backend.
The result is **lossless**: byte-identical to running with every expert resident, asserted
by the gates.
by the gates. That is the streaming path itself; two opt-in knobs deliberately trade output for
speed on top of it — `--n-expert-used` (fewer experts per token) and
[`--drop-cold-experts`](expert-dropping.md) (skip an expert that would cost a read and was barely
weighted). Both are off unless asked for, which is what keeps the sentence above true by default.
## Residency modes

6
docs/prefetch.md
View file

@ -32,7 +32,11 @@ captures most of the benefit. Prefetch requires the LRU cache to be on — eithe
## How it stays correct and out of the way
The speculative path never delays real work and never changes output:
The speculative path never delays real work and never changes output — with the lossy knobs off.
(Under [`--drop-cold-experts`](expert-dropping.md) residency is an *input* to the routing policy,
so a correct guess un-drops an expert that would otherwise have been discarded. Prefetch depth
becomes output-affecting there; everything below still holds for the bytes themselves.)
- **Same bytes.** A speculative read is the *identical* read a real miss would issue — same file
offset, same destination buffer (`lbuf_[p][il] + e*slice`). A prefetched expert is therefore

10
docs/pressure.md
View file

@ -30,7 +30,10 @@ is:
## Why a budget cannot be a constant
The expert cache is the one lever that trades RAM for flash reads, so the temptation is to set it as
The expert cache is the one lever that trades RAM for flash reads (
[`--drop-cold-experts`](expert-dropping.md) is the other kind of trade — quality for flash reads —
and the two interact: a squeezed cache raises the miss rate, which raises the drop rate, so memory
pressure degrades output quality there instead of only throughput). The temptation is to set it as
large as the device seems to allow. On a phone that is the wrong shape of decision, for three
reasons that are measured rather than argued:
@ -115,6 +118,11 @@ not a floor), `layer_demand_MiB` (the mechanical floor), `cache_budget_MiB` (the
effect), `cache_hit_pct`. Per token, `dense_resident_frac` says whether the dense set is holding in
RAM (the live signal now that the cache-residency governor sensor is gone).
This sizing procedure assumes dropping is off. With
[`--drop-cold-experts`](expert-dropping.md) on, dropped routings are misses that never reach the
cache, so `cache_hit_pct` reads high and `token_demand_MiB` reads low for the same budget — size
the cache first, then turn dropping on.
Reading `cache_hit_pct` against `token_demand_MiB` is how you tell whether a fixed `--cache-mb N` is
earning its RAM: a budget near or below one token's demand holds no history between tokens and its
hits are only inter-token correlation; well above it, a high hit rate means real reuse.

13
docs/roadmap.md
View file

@ -113,6 +113,19 @@ routed) are supported; other `build_moe_ffn` models are one recipe row each. The
frontier is architectures whose routing node is not the shared `ffn_moe_topk` — custom gating,
which the capture/stream hook would need to learn. See [adding-a-model.md](adding-a-model.md).
## Skipping reads the router barely wants — built, unmeasured
`--drop-cold-experts` ([expert-dropping.md](expert-dropping.md)) is the first lever that treats
quality and I/O as a *joint* budget rather than two separate knobs: an expert already in the cache
runs however small its weight, and only a routing that would cost a flash read can be dropped. On
the recorded traces that is worth ~3× the reads of turbo top-k for a comparable weight cost, which
is the strongest offline case any remaining lever has shown.
What it does **not** have is a device measurement, and the previous entry on this page is the reason
that matters: `layer-lfu` simulated well and was ~30% slower in reality. The open questions are the
device A/B against turbo top-k at matched throughput, the quality comparison at that speed, and how
far the static replay overstated the win once dropping starts changing what the cache holds.
## Expert quantization on the fly
Storing streamed experts at a lower precision than the resident parts to cut read volume, if it

14
docs/seam.md
View file

@ -24,10 +24,22 @@ come from the arch's recipe — `ffn_{gate,up,down}_exps` for the split layout,
throughout — capture observes, it does not isolate. `ggml_tensor` is a public struct, so
reading `->name`, `->ne`, `->nb` and writing `->data` is public API surface.
**Stream phase** (real generation). We return true only for `ffn_moe_topk-<il>`. The
**Stream phase** (real generation). We return true for `ffn_moe_topk-<il>`. The
non-ask callback then hands us that node with the selected expert ids materialized; we
gather them (stride-aware) and trigger the slice reads.
Two optional jobs ask for more: the route trace and
[cache-aware dropping](expert-dropping.md) also want each layer's `ffn_moe_weights*-<il>` chain,
which is another barrier per node but no new kind of access — same public struct, same read of
`->data`.
Dropping does go one step further, and it is the only place the engine **writes into** a graph
tensor's contents rather than repointing `->data` at its own buffer: at the terminal node of the
weight chain it zeroes a dropped slot's weight and repoints that slot's expert id. Both tensors are
scratch the graph produced and has not yet consumed, so this alters the values flowing through the
run — deliberately, that is what the lossy policy *is* — and never llama.cpp's own state, its
weights, or its control flow. It stays inside the same callback contract; nothing is patched.
## 2. gguf offsets
`gguf_init_from_file(..., no_alloc=true)` + `gguf_get_data_offset` +

31
docs/telemetry.md
View file

@ -51,6 +51,10 @@ BMOE_PROGRESS {"step":<int>,"steps":<int>,"wall_ms":<float>,"io_ms":<float>,
can be a large share of the token; at steady state it is near zero. Surfacing it stops the "all
compute" reading on warm-up tokens where the real cost is cache churn, not matmul.
- `cache_hit_pct` is the cumulative cache hit rate, or `-1` when no cache is used.
**Under [`--drop-cold-experts`](expert-dropping.md) read it with care:** a dropped routing is a
miss that is never looked up, so it leaves both sides of the ratio and the reported hit rate
rises without the cache having served anything more. Compare runs at the same drop rate, or read
`experts_dropped` next to it.
- `majflt` / `cpu_ms` **decompose the `compute_ms` residual** — the whole point being that "compute"
above is a catch-all that silently absorbs page faults and scheduler stalls, not just matmul.
They are measured directly around `llama_decode` (no submodule patch needed): `majflt` is the
@ -101,6 +105,16 @@ moe-prefetch: <mib> MiB speculative, <useful>/<prefetched> experts useful (<pct>
`<prefetched>` the experts fully read ahead, and `<useful>` how many of those a later routing
actually hit. See [prefetch.md](prefetch.md).
With `--drop-cold-experts F` a `moe-drop:` line is added:
```
moe-drop: <dropped>/<routed> routed experts dropped (<pct>%), threshold <F> x uniform
```
The flag fixes a *threshold*, not a rate: how much is actually discarded depends on what the cache
held, so this line — not the flag — is what a run traded. See
[expert-dropping.md](expert-dropping.md).
Under `--overlap` the `moe-stream:` line additionally reports `stall_s/tok=<s>` — the mean
wall time per token that compute threads waited for expert reads to complete. It is `0` in
serial mode (where the read wait is already folded into decode time).
@ -130,7 +144,10 @@ sampled dense-weight residency, `-1` when unmeasured. All are additive: older CS
so consumers must read by column NAME (from the header row) and treat any as optional. The `# summary`
line likewise gains `stall_s/tok=<s>`, `mgmt_s/tok=<s>`, `majflt/tok=<f>`, `cpu_s/tok=<s>`,
`token_demand_MiB=<f>` (the expert bytes one token routes, measured — where cache hits start, NOT a
floor to defend; see [pressure.md](pressure.md)) and `layer_demand_MiB=<f>` (the widest layer's routed
floor to defend; see [pressure.md](pressure.md)), `experts_routed=<n>` / `experts_dropped=<n>` (what
[cache-aware dropping](expert-dropping.md) actually discarded during generation — the flag sets a
threshold, not a rate, so this is the only record of the trade a run made) and
`layer_demand_MiB=<f>` (the widest layer's routed
bytes: the mechanical floor the cache must be able to stage); see the `io_ms` note above for how the
read-time columns are reinterpreted under overlap.
@ -161,6 +178,10 @@ per routed expert. **A traced run is not a benchmark run** — the numbers in th
traced run are slower than the real thing, and `mgmt_ms` in particular shifts, because settling
speculative prefetch moves outside the window that times it.
Columns are **append-only** within `v1`, like the metrics CSV: `dropped` was added after
`expert_bytes`, so consumers must read by column NAME and treat any column as optional rather than
indexing by position.
The file is long format: a `#` preamble carrying the run's static facts, then one row per routed
expert. Conceptually it is a matrix — rows are steps, columns are layers — and a **cell** is the
`n_expert_used` rows sharing `(turn, phase, step, layer)`.
@ -169,7 +190,7 @@ expert. Conceptually it is a matrix — rows are steps, columns are layers — a
# route_trace v1
# model=<path> arch=<string> n_layer=<int> n_expert=<int> n_expert_used=<int>
# layer=<int> expert_bytes=<int> dense_bytes=<int> (one per layer)
turn,phase,step,layer,slot,expert,weight,residency,expert_bytes
turn,phase,step,layer,slot,expert,weight,residency,expert_bytes,dropped
```
| column | meaning |
@ -183,6 +204,7 @@ turn,phase,step,layer,slot,expert,weight,residency,expert_bytes
| `weight` | the final applied routing weight, after whatever softmax/normalise/scale the architecture uses. `nan` when the graph exposed no weight node — "unknown", never `0`. |
| `residency` | `0` = miss (this routing reads from flash), `1` = hit, `2` = hit on a speculative prefetch's first touch. |
| `expert_bytes` | flash bytes this routing reads; `0` unless `residency=0`. |
| `dropped` | `1` when [cache-aware dropping](expert-dropping.md) discarded this routing — a miss weighted below the threshold, never read, weight zeroed. Always `0` with `--drop-cold-experts` off. |
`(turn, phase, step, layer, slot)` is unique. Two asymmetries are deliberate:
@ -195,6 +217,11 @@ turn,phase,step,layer,slot,expert,weight,residency,expert_bytes
streamed, so there is nothing to measure per step: `dense_bytes` is what a cold layer costs to
page in, stated once. Per-layer *I/O time* is absent for the same kind of reason — under
`--overlap` reads complete asynchronously, so any per-layer timing would be fiction.
- **`weight` and `residency` describe the router; `dropped` describes the policy.** When dropping is
on, a discarded routing keeps the weight the router gave it and the residency it faced — the trace
records the routing that was *chosen* — while `expert_bytes` falls to `0`, because a dropped
expert is never read. Summing `expert_bytes` therefore still measures real flash traffic, and
`dropped` is what explains the gap against `residency==0`.
**The last layer has only one prefill step, and that is real.** Before the final layer's FFN,
llama.cpp gathers only the tokens whose logits were asked for (`inp_out_ids`; see `il == n_layer

17
examples/android/app/src/main/java/io/bigmoeonedge/example/AppSettings.kt
View file

@ -34,6 +34,10 @@ data class AppSettings(
val overlap: Boolean = true, // read the next experts while the current layer computes
val denseWeights: DenseWeights = DenseWeights.ANON, // dense (non-expert) weight residency policy
val prefetchLayers: Int = 0, // temporal prefetch depth K (0 = off); needs the cache
// Cache-aware expert dropping, as a PERCENTAGE of the uniform share 1/top-k (0 = off, 100 = the
// share itself). Stored as an Int because the settings are integer rungs; the flag takes a
// fraction. LOSSY and cache-dependent — it changes the output, and not reproducibly.
val dropColdPct: Int = 75,
val thinking: Boolean = false, // reasoning; off passes --no-think (enable_thinking=false)
val metricsCsv: Boolean = true, // write the engine's per-token CSV for this session (--csv)
) {
@ -89,6 +93,11 @@ data class AppSettings(
// Auto sizing is a live LRU cache, so it satisfies the prefetch cache requirement.
val cacheOn = cacheMb == CACHE_AUTO || cacheMb > 0
if (prefetchLayers > 0 && cacheOn) a += listOf("--prefetch", prefetchLayers.toString())
// Cache-aware dropping needs a live cache to ask about residency — with the cache off
// every expert reads as a miss and the engine rejects the combination outright, so the
// same cacheOn condition that guards prefetch guards this. The engine takes a fraction
// of the uniform share; the setting is stored as a percentage.
if (dropColdPct > 0 && cacheOn) a += listOf("--drop-cold-experts", (dropColdPct / 100.0).toString())
}
return a
}
@ -101,7 +110,7 @@ data class AppSettings(
*/
fun sessionSignature(modelPath: String): String =
listOf(modelPath, mmap, cacheMb, cacheCeilMb, ioThreads, threads, nExpertUsed, oDirect,
overlap, denseWeights, prefetchLayers)
overlap, denseWeights, prefetchLayers, dropColdPct)
.joinToString("|")
fun save(ctx: Context) {
@ -114,6 +123,7 @@ data class AppSettings(
.putBoolean("overlap", overlap)
.putString("denseWeights", denseWeights.name)
.putInt("prefetchLayers", prefetchLayers)
.putInt("dropColdPct", dropColdPct)
.putBoolean("thinking", thinking)
.putBoolean("metricsCsv", metricsCsv)
.apply()
@ -178,6 +188,10 @@ data class AppSettings(
// 0 = model default (top-k as trained). 6/4/3/2 trade output quality for tok/s (fewer routed experts).
val N_EXPERT_CHOICES = intArrayOf(0, 6, 4, 3, 2)
val PREFETCH_CHOICES = intArrayOf(0, 1, 2, 4)
// Percent of the uniform share 1/top-k. 100 is the share itself and the useful maximum:
// above it the threshold could exceed every weight in a routing. The rungs below it are the
// conservative half of the curve, where the replay already beats a top-k cut on both axes.
val DROP_COLD_CHOICES = intArrayOf(0, 50, 75, 100)
val THREAD_CHOICES = intArrayOf(2, 4, 6, 8)
val NPREDICT_CHOICES = intArrayOf(16, 32, 48, 64, 128, 256, 512, 1024, 2048)
@ -206,6 +220,7 @@ data class AppSettings(
}
},
prefetchLayers = p.getInt("prefetchLayers", d.prefetchLayers),
dropColdPct = p.getInt("dropColdPct", d.dropColdPct),
thinking = p.getBoolean("thinking", d.thinking),
metricsCsv = p.getBoolean("metricsCsv", d.metricsCsv),
)

19
examples/android/app/src/main/java/io/bigmoeonedge/example/SettingsScreen.kt
View file

@ -144,6 +144,25 @@ fun SettingsScreen(current: AppSettings, onChange: (AppSettings) -> Unit, onBack
"but the output changes — a speed/quality trade-off.",
fontSize = 12.sp, color = MaterialTheme.colorScheme.onSurfaceVariant,
)
IntSetting(
"Drop cold experts (% of even share)", AppSettings.DROP_COLD_CHOICES, current.dropColdPct,
format = { if (it == 0) "off" else "$it%" },
// Unlike top-k, this one asks the expert source what is resident, so it needs
// both the streamer and a live cache — the same condition prefetch is under.
enabled = !current.mmap &&
(current.cacheMb == AppSettings.CACHE_AUTO || current.cacheMb > 0),
) { onChange(current.copy(dropColdPct = it)) }
Text(
"Experimental. What slows a token down is reading an expert that is not already in RAM. " +
"This skips such an expert when the router barely wanted it anyway — below the chosen " +
"share of an even split. With 8 active experts an even split is 12.5% each, so 75% " +
"means \"skip it if it carries less than 9.4% of the routing\".\n\n" +
"Experts already in RAM always run, however small their weight: they cost no read. The " +
"strongest expert of each routing is never skipped.\n\n" +
"Higher is faster and rougher. Like Active experts, the reply changes — but unlike it, " +
"not the same way twice: what gets skipped depends on what the cache happened to hold.",
fontSize = 12.sp, color = MaterialTheme.colorScheme.onSurfaceVariant,
)
}
Section("Compute") {

14
scripts/route-analyze.py
View file

@ -52,6 +52,7 @@ class Trace:
self.expert_bytes = {} # layer -> bytes of one expert
self.dense_bytes = {} # layer -> non-streamed bytes
self.rows = [] # (step, layer, slot, expert, weight, residency, ebytes)
self.n_dropped = 0 # routings discarded by --drop-cold-experts (0 unless it was on)
self._read(path, phase, turn)
def _read(self, path, phase, turn):
@ -72,6 +73,12 @@ class Trace:
continue
w = float("nan") if p[6] == "nan" else float(p[6])
self.rows.append((int(p[2]), int(p[3]), int(p[4]), int(p[5]), w, int(p[7]), int(p[8])))
# `dropped` (appended after expert_bytes) marks a routing cache-aware dropping
# discarded: the router asked for it, the engine never read it. Counted, not folded
# into the rows, so every figure below keeps meaning "what the ROUTER asked for" —
# see docs/expert-dropping.md for why the two diverge.
if len(p) > 9 and p[9].strip() not in ("", "0"):
self.n_dropped += 1
def _preamble(self, line):
kv = kv_tokens(line)
@ -391,6 +398,13 @@ def main():
print("arch %s n_layer %d n_expert %d n_expert_used %d"
% (tr.arch, tr.n_layer, tr.n_expert, tr.n_expert_used))
print("phase %s rows %d" % (args.phase, len(tr.rows)))
if tr.n_dropped:
# Say it once, loudly: on such a trace "routed" and "read" are no longer the same set, and
# every reuse/working-set figure below describes the router's demand, not flash traffic.
print("NOTE cache-aware dropping was ON: %d of %d routings (%.1f%%) were discarded and "
"never read.\n Figures below are what the ROUTER asked for; use "
"scripts/route-drop-replay.py for what it cost."
% (tr.n_dropped, len(tr.rows), 100.0 * tr.n_dropped / len(tr.rows)))
names = DEFAULT_VIEWS if args.view == "default" else (
list(VIEWS) if args.view == "all" else [args.view])

159
scripts/route-drop-replay.py Normal file
View file

@ -0,0 +1,159 @@
#!/usr/bin/env python3
"""Replay a route trace against the cache-aware expert-dropping policy (docs/expert-dropping.md).
The policy skips a routed expert when it is a cache MISS and the router weighted it below
`frac x (1 / n_expert_used)`. A resident expert costs no flash read, so it is never dropped:
quality is spent only where it buys I/O. This script answers, per threshold, what that trade
would have been on an already-recorded run:
io_saved fraction of MISS BYTES the policy never reads -- the win
mass_lost fraction of total router weight discarded -- the proxy for the damage
The static-k baseline (`--n-expert-used`) is replayed on the same rows, because the only question
that matters is comparative: at equal io_saved, which policy discards less weight?
Two limits, both deliberate:
* This is a STATIC replay. Skipping a read changes what the cache holds later, so the real hit
pattern drifts from the recorded one. io_saved is an UPPER BOUND, not a prediction.
* mass_lost is a proxy. It says how much of the router's mass went away, not what that did to
the output. Only a quality A/B answers that.
A trace recorded with dropping already ON reports its `dropped` column instead of re-deriving it,
which is how the upper bound above gets checked against a real run.
Usage: route-drop-replay.py <route.csv> [<route.csv> ...]
Stdlib only, like the other analysis scripts here.
"""
import os
import sys
from collections import defaultdict
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from trace_io import read_preamble_csv # noqa: E402
MISS = 0
DECODE, PREFILL = 1, 0
def cells(rows, phase):
"""Group rows into routing cells: (turn, step, layer) -> [(slot, weight, residency, bytes, dropped)]."""
out = defaultdict(list)
for r in rows:
if int(r["phase"]) != phase:
continue
try:
w = float(r["weight"])
except ValueError:
continue # 'nan': the graph exposed no weight node, so no threshold can be applied
out[(r["turn"], r["step"], r["layer"])].append(
(int(r["slot"]), w, int(r["residency"]), int(r["expert_bytes"]), int(r.get("dropped", 0) or 0))
)
return out
def replay_threshold(cs, thr):
"""Drop a miss weighted below thr, never the cell's top expert (which the engine also pins)."""
miss_bytes = dropped_bytes = 0
total_mass = lost_mass = 0.0
kept_hist = defaultdict(int)
for entries in cs.values():
best = max(range(len(entries)), key=lambda i: entries[i][1])
kept = 0
for i, (_slot, w, res, nb, _d) in enumerate(entries):
total_mass += w
if res == MISS:
miss_bytes += nb
if i != best and w < thr:
dropped_bytes += nb
lost_mass += w
continue
kept += 1
kept_hist[kept] += 1
return miss_bytes, dropped_bytes, total_mass, lost_mass, kept_hist
def replay_static_k(cs, keep_k):
"""Baseline --n-expert-used: keep the top keep_k slots whatever the cache holds."""
miss_bytes = dropped_bytes = 0
total_mass = lost_mass = 0.0
for entries in cs.values():
for slot, w, res, nb, _d in entries:
total_mass += w
if res == MISS:
miss_bytes += nb
if slot >= keep_k:
lost_mass += w
if res == MISS:
dropped_bytes += nb
return miss_bytes, dropped_bytes, total_mass, lost_mass
def observed(cs):
"""What a trace recorded with dropping ON actually did. (dropped rows, weight mass, miss bytes)."""
n_dropped = 0
lost_mass = total_mass = 0.0
for entries in cs.values():
for _slot, w, _res, _nb, d in entries:
total_mass += w
if d:
n_dropped += 1
lost_mass += w
return n_dropped, lost_mass, total_mass
def pct(num, den):
return 100.0 * num / den if den else 0.0
def report(path):
meta, rows = read_preamble_csv(path)
k = int(meta.get("n_expert_used", 0) or 0)
if k <= 0:
print(f"{path}: no n_expert_used in the preamble; cannot express a threshold")
return
print("=" * 78)
print(f"{os.path.basename(path)} arch={meta.get('arch')} n_expert={meta.get('n_expert')} k={k}")
print("=" * 78)
for phase, label in ((DECODE, "DECODE"), (PREFILL, "PREFILL")):
cs = cells(rows, phase)
if not cs:
print(f"\n[{label}] no rows")
continue
n_tot = sum(len(e) for e in cs.values())
n_miss = sum(1 for e in cs.values() for x in e if x[2] == MISS)
print(f"\n[{label}] {len(cs)} routing cells, {n_tot} routed experts, {pct(n_miss, n_tot):.1f}% misses")
n_drop, lost, total = observed(cs)
if n_drop:
print(f" recorded: dropping was ON for this run -- {n_drop} routings dropped "
f"({pct(n_drop, n_tot):.1f}%), {pct(lost, total):.2f}% of the weight mass")
uniform = 1.0 / k
print(f"\n cache-aware threshold, as a fraction of the uniform share 1/k = {100 * uniform:.2f}%")
print(f" {'frac':>6} {'thr':>8} {'io_saved':>9} {'mass_lost':>10} surviving experts per cell")
for frac in (0.25, 0.5, 0.75, 1.0):
thr = uniform * frac
mb, db, tm, lm, hist = replay_threshold(cs, thr)
h = " ".join(f"{kk}:{pct(v, len(cs)):.0f}%" for kk, v in sorted(hist.items()))
print(f" {frac:6.2f} {100 * thr:7.2f}% {pct(db, mb):8.1f}% {pct(lm, tm):9.2f}% {h}")
print(f"\n static-k baseline (--n-expert-used), same rows, for comparison at equal io_saved")
print(f" {'keep_k':>6} {'io_saved':>9} {'mass_lost':>10}")
for keep in range(k - 1, 0, -1):
mb, db, tm, lm = replay_static_k(cs, keep)
print(f" {keep:6d} {pct(db, mb):8.1f}% {pct(lm, tm):9.2f}%")
print()
def main(argv):
if len(argv) < 2:
print(__doc__)
return 2
for p in argv[1:]:
report(p)
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))

22
tests/config_test.cpp
View file

@ -143,6 +143,28 @@ int main() {
expect_ok("out-of-range sampling knobs are inert under greedy", c);
}
// Cache-aware expert dropping. The upper bound is not cosmetic: above the uniform share the
// threshold can exceed every weight in a routing, and a config that can empty a layer must not
// be accepted just because the implementation happens to guard against it too.
{
RunConfig c = ok_base();
c.moe.enabled = true;
c.moe.drop_cold_frac = 0.5f;
expect_fail("dropping without a cache is rejected (nothing to be aware of)", c);
c.moe.cache_mb = MoeStreamConfig::cache_min_mb;
c.moe.drop_cold_frac = 0.0f;
expect_ok("dropping off is the default and valid", c);
c.moe.drop_cold_frac = 0.5f;
expect_ok("a threshold below the uniform share is valid", c);
c.moe.drop_cold_frac = 1.0f;
expect_ok("the uniform share itself is valid", c);
c.moe.drop_cold_frac = 1.01f;
expect_fail("a threshold above the uniform share is rejected", c);
c.moe.drop_cold_frac = -0.1f;
expect_fail("a negative threshold is rejected", c);
}
if (failures == 0) {
std::printf("all config checks passed\n");
return 0;

79
tests/moe_gates.cpp
View file

@ -350,6 +350,85 @@ int main(int argc, char ** argv) {
std::printf("[SKIP] S2 (expert-ready hook not built)\n");
#endif
// G8 — cache-aware expert dropping, plumbing vs policy.
//
// Arming the policy moves load_layer() from the topk node to the terminal node of the layer's
// weight chain, and has the hook learn which node that is. That machinery must be transparent:
// with a threshold below any weight the router can produce, nothing is dropped and the output
// must stay byte-identical to the undropped stream. This separates "the deferral is correct"
// from "the policy is lossy" — only the second is allowed to change bytes, and a regression in
// the first would otherwise hide behind the expected difference.
// The policy needs a real LRU cache: with the cache off every expert reads as a miss, so it
// would degenerate into an unconditional weight cut and the repointing below would never face
// the reserved-but-uncommitted slot it exists to avoid. The small forced budget is the same one
// G2 uses to provoke evictions, so misses and hits both occur.
RunConfig drop_inert = base(model);
drop_inert.moe.enabled = true;
drop_inert.moe.cache_mb = 2;
drop_inert.moe.force_cache = true;
drop_inert.moe.io_threads = 4;
drop_inert.moe.drop_cold_frac = 1e-6f;
std::string s_drop_inert;
if (!gen(drop_inert, s_drop_inert, err)) {
std::fprintf(stderr, "drop(inert threshold) run failed: %s\n", err.c_str());
return 2;
}
fails += check("G8a drop(threshold below any weight) == streaming(cached, undropped)", s_sc, s_drop_inert);
// The identity above only means anything if the policy really was armed and really dropped
// nothing. Asserting the count separately turns "a weight happened to fall under the threshold"
// from a mysterious byte mismatch into a legible failure.
{
RunResult r = run(drop_inert);
if (!r || r.summary.experts_dropped != 0 || r.summary.experts_routed <= 0) {
std::printf("[FAIL] G8a' inert threshold must examine routings and drop none (routed=%lld dropped=%lld)\n",
r.summary.experts_routed, r.summary.experts_dropped);
++fails;
} else {
std::printf("[PASS] G8a' inert threshold examined %lld routings, dropped none\n", r.summary.experts_routed);
}
}
// G8b — the same at full strength, against a cache small enough to be evicting constantly, so
// dropped experts really do land on slots the cache has released. There is no reference output
// to compare against (it is lossy by design), so the gate is that the engine survives it: a
// dropped expert's slot is repointed at one that is certainly resident, so the matmul must never
// read reserved-but-uncommitted memory and generation must still complete.
RunConfig drop_hard = drop_inert;
drop_hard.moe.drop_cold_frac = 1.0f;
drop_hard.moe.drop_prefill = true;
std::string s_drop_hard;
if (!gen(drop_hard, s_drop_hard, err)) {
std::fprintf(stderr, "drop(full strength) run failed: %s\n", err.c_str());
return 2;
}
if (s_drop_hard.empty()) {
std::printf("[FAIL] G8b drop(full strength) produced no output\n");
++fails;
} else {
std::printf("[PASS] G8b drop(full strength) generates without touching unloaded experts\n");
}
// G8c — the top-weighted expert is pinned, so a routing can never be emptied. Forcing top-k to
// 1 makes every routed expert the top one, and dropping must then be a no-op at ANY threshold:
// the output has to match the same k=1 run with the policy off, byte for byte. This also pins
// down that the threshold is taken against the EFFECTIVE top-k discovered at runtime — a
// hardcoded width would not survive the override.
RunConfig k1 = base(model);
k1.moe.enabled = true;
k1.moe.cache_mb = 2;
k1.moe.force_cache = true;
k1.moe.io_threads = 4;
k1.n_expert_used = 1;
RunConfig k1_drop = k1;
k1_drop.moe.drop_cold_frac = 1.0f;
k1_drop.moe.drop_prefill = true;
std::string s_k1, s_k1_drop;
if (!gen(k1, s_k1, err) || !gen(k1_drop, s_k1_drop, err)) {
std::fprintf(stderr, "top-k=1 drop run failed: %s\n", err.c_str());
return 2;
}
fails += check("G8c drop(full strength, top-k=1) == top-k=1 undropped (top expert pinned)", s_k1, s_k1_drop);
if (fails == 0) std::printf("\nall MoE byte-identity gates passed\n");
return fails == 0 ? 0 : 1;
}