mirror of
https://github.com/Helldez/BigMoeOnEdge.git
synced 2026-10-20 11:00:56 +00:00
eb466c2893
The compute buffers are reserved for the widest graph a prompt is computed in (--ubatch), which follows the context, while the ledger read them from the capacity fitter at llama.cpp's default of 512 tokens and with a logit row per position. At a 2048 context the two errors roughly cancel; at 16384 the ledger charged 1.2 GB for 8 GB reserved, the cache lock was refused part way and decode lost a third of its rate. probe_placement takes the session's own width, projects with the row cap the session sets under a prefill device, and projects again at half the width: two points give what a token of width costs on this model, context and set of backends (Placement::at_width). The planner then takes the widest graph whose buffers stay inside a share of what the process may hold (PlannerPolicy::compute_share_max, a tenth), halving from the context and never below ubatch_min. A device's buffers count where they come out of this pool. A pinned width is kept and charged in full; an unmeasured slope decides nothing. On a 16 GB Apple-silicon laptop, Qwen3.6-35B-A3B Q4_K_M, 16384 context, --auto alone: decode 22.4 and 22.3 tok/s (14.6 and 16.7), the cache locked at 7.7 GB (settled at 3.5), a 10850-token prompt in 31.8 s (20.4, at a width that cost the decode). A 2048 context plans as before. Measured on that one machine: docs/hardware-planning.md lists where it is still owed. Split models, and what a declined plan costs. The storage probe read the path the caller names. On a model split into shards that is the first shard, which can be metadata alone: a 5 MB file the drive answers from its own cache. The model profile now names the file holding the most expert bytes and the probe reads there. A plan that declined streaming because the cache fell under the token cycle kept the allocation it had refused, and printed its cost. It is now restated as the file left mapped with nothing held. The cache floor. A budget under the model's token cycle declined streaming altogether and left the file mapped, on the machine where a mapped file is slowest: the memory a cache lacks is the memory the page cache lacks too. The floor now takes the cache alone. The plan keeps --moe-stream with --cache-mb 0 when the ledger prices the uncached stream under the mapping, and declines as before where the drive's read rate was never measured, so the stream is never chosen without a price. On a 16 GB Apple-silicon laptop, Qwen3.6-35B-A3B Q4_K_M at a 16384 context with about 3 GB reported available: 0.26 tok/s declined and mapped, 7.3 tok/s from --auto now. The floor itself holds: a 536 MiB cache forced on returned 0.0% hits at 6.3 tok/s against 8.6 with none. With 8.9 GB reported available the plan is unchanged. |
||
|---|---|---|
| .. | ||
| CMakeLists.txt | ||
| main.cpp | ||