mirror of
https://github.com/Helldez/BigMoeOnEdge.git
synced 2026-10-19 15:33:46 +00:00
cecfa397df
Three pieces that sit on one another, for every target the engine builds on. The parameter table (bmoe/params.h, core/src/params.cpp). Every RunConfig tunable is one row: key, type, group, bounds written with the constants validate() checks, help, reader and writer. bmoe-cli parses its engine flags and prints their usage from it; --describe-params prints it as JSON for a front-end to render, --show-config the resolved configuration. The hardware planner (--auto, --plan, --probe; core/src/plan). The engine measures the machine - memory it can keep and what a reclaim costs, core classes and a thread sweep, the bandwidth of the CPU and of every device on the model's own quantized matmul, what a host/device crossing costs, the storage rate by request size and lane count - reads the model from its gguf metadata, runs llama.cpp's capacity fitter, and resolves the streaming knobs with the fact behind each one. The rules are pure functions over two profiles and name no platform, vendor, device or architecture; the *_probe adapters are the only files that may. A knob set by hand is never touched, a missing fact declines instead of guessing, and nothing lossy arms itself. One memory ledger with two ceilings. Everything a run holds - context and compute buffers, a device armed for prefill, the dense set, the expert cache - is a row charged against what the process may hold and, separately, against what it may hold reclaim-exempt, net of what is already locked. The dense set is pinned whole or not at all, and the cache is sized to what can be locked where it can be locked at all. --cache-pin-fit steps the cache budget down to what the platform grants when a lock is refused part way, instead of ending pinning; off by default, set by the planner when it sized the cache to the lock. macOS. Metal in every build, the planner, --release-mmap and the memory telemetry on Darwin, --devices auto (a device with nothing placed on it stays out of the graph), the expert cache pinned in RAM, an evicted expert's pages really released, and --dense-weights ahwb as wired memory. Also: --release-mmap (+46% decode on Windows, where a live mapping serialises the streamer's unbuffered reads), a portable build that collects every binary and backend library in build/bin, and the planner in the layer maps of docs/architecture.md and AGENTS.md. Measured on a 16 GB Apple-silicon laptop with --auto alone: DeepSeek V4 Flash UD-IQ2_M 2.46 to 2.48 tok/s (0.27 before the ledger, when the plan asked for more locked memory than the machine grants), Qwen3.6-35B-A3B Q4_K_M 23.1 to 23.5 tok/s. ci: the workflows, the Android staging script and two bench scripts look for the binaries in build/bin, where the build now collects them. docs: docs/backends.md lists the eighteen compute backends in the pinned llama.cpp and which three have a measurement behind them here.
21 lines
745 B
Bash
Executable file
21 lines
745 B
Bash
Executable file
#!/usr/bin/env bash
|
|
# Configure and build BigMoeOnEdge for the host (Linux/macOS). Env overrides:
|
|
# BUILD_DIR (default build), BUILD_TYPE (Release), JOBS (nproc).
|
|
set -euo pipefail
|
|
|
|
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
BUILD_DIR="${BUILD_DIR:-build}"
|
|
BUILD_TYPE="${BUILD_TYPE:-Release}"
|
|
JOBS="${JOBS:-$(getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4)}"
|
|
|
|
cd "$ROOT"
|
|
if [ ! -f third_party/llama.cpp/CMakeLists.txt ]; then
|
|
echo "llama.cpp submodule missing — run: git submodule update --init --recursive"
|
|
exit 1
|
|
fi
|
|
|
|
cmake -S . -B "$BUILD_DIR" -DCMAKE_BUILD_TYPE="$BUILD_TYPE"
|
|
cmake --build "$BUILD_DIR" -j "$JOBS"
|
|
|
|
echo "built: $BUILD_DIR/bin/bmoe-cli"
|
|
echo "run gates: (cd $BUILD_DIR && ctest --output-on-failure)"
|