BigMoeOnEdge/docs

Documentation

Start with architecture.md for the layer map, or moe-streaming.md for the idea the project is built on.

Understanding the design

Doc	What it answers
architecture.md	How the layers fit together, and why llama.cpp is not forked.
moe-streaming.md	Why streaming experts from flash makes a >RAM model run at all.
seam.md	The exact contract with llama.cpp's public API, and how to upgrade the submodule.
platforms.md	What runs on each operating system and backend, how far each has been taken, and what each still owes.
backends.md	Every compute backend the pinned llama.cpp carries, which of them this engine has actually run on, and what each use of a device asks of one.
limitations.md	What this does not do, what it cannot do, and the prior art it builds on.
roadmap.md	Where the project is going as a product, in order; then the research themes, with what was measured and what was refuted.

Using and extending it

Doc	What it answers
application-framework.md	Why bmoe-server is the application framework every UI is a client of, and what a client may and may not do.
screens.md	The five screens every UI shows, specified once.
design.md	The one visual language every UI uses.
which-models-stream.md	Which models stream: the rule read from the file, what is refused and why, what stays resident.
telemetry.md	The BMOE_* line protocol and CSV schema — the integration contract.
server-api.md	The desktop server's HTTP API: settings from the parameter table, OpenAI-compatible chat, live events, models and the plan.
session.md	Session lifecycle, KV prefix reuse, cancellation.
decide.md	--decide: picking one of a list of choices from a single prefill, with no decode, and keeping the state after a shared prefix between calls.
settings.md	Every streaming setting: the flag, its values and what it does.
hardware-planning.md	--auto: how the engine sizes a run to this machine and this model, the fact behind each choice, and what is still unmeasured.
cache-sizing.md	--cache-mb auto, the cache ceiling, and dense warm-up.
prefetch.md	--prefetch K: the design and why it cannot change output (with the lossy knobs off).
expert-dropping.md	--drop-cold-experts F: spending quality only where it buys a flash read, and why a cache-dependent setting's output is not reproducible.
row-gathered-tables.md	--row-stream: serving a dense table the graph only gathers rows from out of flash, and how the engine decides which tables those are without naming one.
cache-aware-substitution.md	--expert-substitute L: re-ranking a routing toward the experts already in the cache, the paper it comes from, and why a wide scoring batch cannot price it.
mtp.md	--mtp: drafting with the model's own MTP head and verifying a whole group per decode — lossless by construction, and why a wider decode can lose in the streamed regime.
ngram.md	--ngram: drafting from text that repeats, with no head and no draft decode — and why a source that abstains costs exactly an unspeculated step.
expert-prediction.md	--predict-log: how much of a routing can be known a layer early, measured against the predictor --prefetch already bets on — and why a good score still would not mean a faster decode.
route-ahead.md	--route-ahead N: committing the routing to the N-layers-early prediction, so a prefetch of it can never miss — and what that costs in quality (experimental, lossy).
npu-prefill.md	--prefill-device: prefill on the NPU and decode on the CPU, a model larger than RAM fed to the NPU two layers at a time, and why decode stays on the CPU.
android-memory.md	What reclaims the engine's memory on a phone, which levers exist (almost none), and why the cache hit rate is what the kernel judges you by.
pressure.md	Cache policy under memory pressure: why an unaffordable budget starts a reclaim war, why the adaptive governor was retired, and what the fixed --cache-mb / --dense-weights levers do.

Measurements

Doc	What it answers
benchmarks.md	Measured results per model on Android, with device-pressure numbers.
benchmarks-gpt-oss.md	gpt-oss-120b: a 58 GB model at 5.2× device RAM, and what it costs.
benchmark-method.md	How to measure this engine on any machine: what each knob does, when to move it, and the rules that keep a matrix honest.
community-benchmarks.md	Results on hardware we do not own, the one-command protocol (scripts/bench-report.sh), and how to submit a row.
warmup-analysis.md	Why first tokens are slow, and the two regimes behind it.
bench-data/	Raw per-run CSVs and session notes. A dated archive — see its README.

Every benchmark figure in these docs is measured on the hardware named beside it. If you re-measure, update benchmark-method.md and the affected table together.