diff --git a/docs/adding-a-model.md b/docs/adding-a-model.md index 2c26653..c0ebcef 100644 --- a/docs/adding-a-model.md +++ b/docs/adding-a-model.md @@ -10,32 +10,42 @@ In `core/src/moe/arch_registry.cpp`: ```cpp static const MoeRecipe k_recipes[] = { - { "qwen3moe", "ffn_gate_exps", "ffn_up_exps", "ffn_down_exps" }, - { "your_arch", "ffn_gate_exps", "ffn_up_exps", "ffn_down_exps" }, // <-- new + { "qwen3moe", { "ffn_gate_exps", "ffn_up_exps", "ffn_down_exps" } }, + { "your_arch", { "ffn_gate_exps", "ffn_up_exps", "ffn_down_exps" } }, // <-- new }; ``` -`arch` is the gguf `general.architecture` string. The three suffixes are the tensor names -without the `blk..` prefix and `.weight` suffix. +`arch` is the gguf `general.architecture` string. `exps_suffix` lists the layer's expert +weight tensors — the names without the `blk..` prefix and `.weight` suffix. The common +split layout names three ({gate, up, down}); leave a trailing slot `nullptr` for layouts +with fewer (see the fused case below). The engine discovers the expert count and per-expert +stride at runtime, so a recipe is only these names. ## 2. Run the gates -Generate a tiny model for your architecture (adapt `scripts/make-tiny-moe.py`) and run: +Generate a tiny model for your architecture and run its gate: ```bash +# scripts/make-tiny-moe.py --arch emits a synthetic model for that layout; +# tests/CMakeLists.txt wires one generate-fixture + gate per arch it knows. cd build && ctest -R moe_gates --output-on-failure ``` -If G1 (streamed == resident) passes, the architecture streams losslessly. +If G1 (streamed == resident) passes, the architecture streams losslessly. For a layout +`make-tiny-moe.py` does not yet emit, either teach it that arch (preferred — permanent CI +coverage) or validate the recipe against a real model, as was done for `llada-moe`. ## When one row is not enough -Some models **pack gate and up into a single tensor** (`ffn_gate_up_exps`) instead of two. -The current recipe shape assumes three separate projections, and the engine will refuse -such a model rather than stream it incorrectly. Supporting the merged layout needs a -distinct recipe variant and a small change in `expert_stream_source` to treat the packed -tensor as two logical projections — contributions welcome. +Some models **pack gate and up into a single tensor** (`ffn_gate_up_exps`) instead of two — +`gemma4` (Gemma 4 MoE) is one. This is still a single row: put the fused suffix first and +leave the tail `nullptr`, because to the streamer a fused `gate_up` is just an expert tensor +with a larger per-expert stride, discovered at runtime like any other. + +```cpp + { "gemma4", { "ffn_gate_up_exps", "ffn_down_exps", nullptr } }, +``` Models with **shared/always-on experts** (a dense expert applied to every token, as in -DeepSeek and some Qwen variants) work, but the shared expert stays resident and reduces -the streaming saving proportionally. Note it in the model's entry when you add one. +`gemma4`, DeepSeek and some Qwen variants) work, but the shared expert stays resident and +reduces the streaming saving proportionally. Note it in the model's entry when you add one. diff --git a/docs/limitations.md b/docs/limitations.md index 066ab6f..5ac4ffa 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -23,8 +23,10 @@ expert-selective streaming that stays lossless and requires no fork. - **CPU experts.** Streamed experts are computed on CPU; the rebind targets host memory. GPU offload of the streamed experts is not supported (the dense parts can still use the GPU). Decode is flash-I/O-bound anyway, so this is rarely the bottleneck. -- **Three separate projections.** Models that pack gate+up into one expert tensor are not - yet supported (see [adding-a-model.md](adding-a-model.md)). +- **Shared experts stay resident.** Architectures with an always-on shared expert (e.g. + `gemma4`) stream the routed experts but keep the shared expert — and any dense layers — + mmap-resident, so the streamed fraction (and the memory saving) is smaller than for a + purely routed model like `qwen3moe`. - **Repack must stay off.** Loading uses `use_extra_bufts=false`; you cannot combine streaming with weight repacking. - **Windows throughput.** The cache's reserve-then-commit-per-slice path is heavier on diff --git a/docs/roadmap.md b/docs/roadmap.md index e044fbb..7275a86 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -20,8 +20,10 @@ large share of the I/O. Requires a routing predictor or speculative prefetch. ## More architectures -Beyond `qwen3moe`: other `build_moe_ffn` models (one recipe row each), then the merged -`ffn_gate_up_exps` layout and shared-expert models. See +Beyond `qwen3moe`: other `build_moe_ffn` models are one recipe row each. The merged +`ffn_gate_up_exps` layout and shared-expert models are supported too (`gemma4` / Gemma 4 +MoE exercises both). Remaining frontier: architectures whose routing node is not the shared +`ffn_moe_topk` (e.g. custom gating), which the capture/stream hook would need to learn. See [adding-a-model.md](adding-a-model.md). ## Expert quantization on the fly diff --git a/docs/seam.md b/docs/seam.md index ae499b6..ec862d9 100644 --- a/docs/seam.md +++ b/docs/seam.md @@ -18,10 +18,11 @@ updated. We use both phases: **Capture phase** (one warm-up decode). `ask` is called for every node, so we scan each -node's `src[]` for expert weight tensors (`blk..ffn_{gate,up,down}_exps.weight`) and -record the live `ggml_tensor*`. We return false throughout — capture observes, it does not -isolate. `ggml_tensor` is a public struct, so reading `->name`, `->ne`, `->nb` and writing -`->data` is public API surface. +node's `src[]` for expert weight tensors (`blk...weight`, where the suffixes +come from the arch's recipe — `ffn_{gate,up,down}_exps` for the split layout, a fused +`ffn_gate_up_exps` for others) and record the live `ggml_tensor*`. We return false +throughout — capture observes, it does not isolate. `ggml_tensor` is a public struct, so +reading `->name`, `->ne`, `->nb` and writing `->data` is public API surface. **Stream phase** (real generation). We return true only for `ffn_moe_topk-`. The non-ask callback then hands us that node with the selected expert ids materialized; we @@ -52,6 +53,14 @@ cd ../.. && git add third_party/llama.cpp && scripts/build-host.sh cd build && ctest --output-on-failure # gates must stay green ``` +Most bumps are exactly this: update, rebuild, gates green. The fragility is not the public +API but the *internal naming conventions* the seam attaches to — the tensor suffixes and +the `ffn_moe_topk` node name are how llama.cpp happens to build MoE graphs today, not a +guaranteed contract, so upstream can rename or restructure them (Gemma 4's fused +`ffn_gate_up_exps` is one such evolution we absorbed with a recipe row). The gates are the +enforcement: a rename breaks byte-identity before merge instead of silently corrupting +output. Each supported architecture adds one more gate to keep green across a bump. + If a future release moves the two hooks (a stable expert-residency API, say) upstream, this seam shrinks further or disappears — `core/` does not change.