mirror of
https://github.com/Helldez/BigMoeOnEdge.git
synced 2026-10-19 15:33:46 +00:00
010eeb1da3
LFM2 MoE is a hybrid short-convolution/attention stack, but its routed experts
use the standard split layout (ffn_{gate,up,down}_exps) and it routes through
build_moe_ffn, so the engine reads the same ffn_moe_topk node it already reads.
Support is therefore one registry row, with no change to the streaming path.
Two structural properties are worth recording rather than discovering later.
The first leading_dense_block_count blocks are dense and name no expert tensors,
so they never bind and stay mmap-resident, lowering the streamed fraction the
same way a shared expert does. And the router applies a per-expert bias
(ffn_exp_probs_b) before the top-k, which changes which experts are selected but
not the node the hook observes.
The byte-identity gates pass; the split-layout gate (qwen3moe) already covers
this streaming path, as it does for gpt-oss and qwen35moe.
Registering the architecture states that the layout streams losslessly, not that
streaming is the right way to run these models: at ~8B total, LFM2.5-8B-A1B fits
in phone RAM, where a resident load is faster. It is deliberately not added to
the in-app download catalog, which lists the models the engine is measured on.
2.9 KiB
2.9 KiB
Limitations and prior art
Prior art
BigMoeOnEdge is an engineering package, not a new technique. The ideas it combines:
- AirLLM — layer-by-layer streaming of >RAM models from disk.
- Apple, "LLM in a flash" — flash-aware weight streaming, windowing, sparsity-driven loading.
- FlexGen — offloading and I/O-bound throughput scheduling for large models.
- PowerInfer / EdgeMoE — hot/cold expert locality and expert-granularity residency on the edge.
The contribution here is a clean, modular, llama.cpp-native implementation of
expert-selective streaming that stays lossless and runs on the public API — no fork for the
serial path, and only a single ~25-line hook (with an explicit sunset) for the optional
--overlap feature. See seam.md § 3.
Limitations
- n=1 only. The expert sparsity exists only for single-token decode, so streaming is incompatible with speculative decoding or batching. Prefill streams the union of the prompt's routed experts (still far below the full bank, but larger than one token's).
- CPU experts. Streamed experts are computed on CPU; the rebind targets host memory. GPU offload of the streamed experts is not supported (the dense parts can still use the GPU). Decode is flash-I/O-bound anyway, so this is rarely the bottleneck.
- Shared experts stay resident. Architectures with an always-on shared expert (e.g.
gemma4) stream the routed experts but keep the shared expert — and any dense layers — resident (in the page cache, or in the engine's own buffers under--dense-weights anon), so the streamed fraction (and the memory saving) is smaller than for a purely routed model likeqwen3moe. The same applies to architectures whose first blocks are dense by design (lfm2moehas aleading_dense_block_count): those blocks name no expert tensors, so they are never streamed. - Streaming does not help a model that fits. The engine's reason to exist is a model larger than RAM. Registering an architecture says the layout streams losslessly, not that streaming is the fast way to run every model using it — a small MoE that fits in memory is faster loaded resident, and the registry rows are about coverage, not a recommendation.
- Repack must stay off. Loading uses
use_extra_bufts=false; you cannot combine streaming with weight repacking. - Windows throughput. The cache's reserve-then-commit-per-slice path is heavier on Windows than the POSIX lazy-commit path. The gates run on Windows; the throughput targets are stated for Android/Linux.
- Depends on a ggml scheduling behaviour (documented in seam.md) that is not a stability-guaranteed contract. Re-verified by the gates on each submodule bump.
Not goals
- Distributing a model across devices (a different axis).
- Beating a model that already fits in RAM — if it fits, run it resident.