docs(roadmap): Vulkan and OpenCL GPUs, a Connect view for clients, the planner drawn (#258)
Some checks are pending
ci / version (push) Waiting to run
ci / format (push) Blocked by required conditions
ci / host-linux (push) Blocked by required conditions
ci / ui (push) Blocked by required conditions
ci / host-windows (push) Waiting to run
ci / host-macos (push) Waiting to run
ci / ios-harness (push) Waiting to run
ci / changes (push) Waiting to run
ci / android-apk (push) Waiting to run

Three steps join the product plan: carry the Vulkan and OpenCL backends in a
build, measure the prefill device on each and then the host-memory route for
streamed experts; one Connect view that tells any OpenAI-compatible client what
it needs (#257); and one diagram of the planner's logic for
docs/hardware-planning.md and the README.
This commit is contained in:
Raffaele 2026-10-16 12:37:49 +02:00 • committed by GitHub
parent 35242b5226
commit 0207ed9382
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 29 additions and 1 deletions

View file

@ -104,7 +104,10 @@ Semantic Versioning.
same thing their own way (#236).
- `docs/roadmap.md` opens with the product plan, in the order the work has to happen: one engine,
one application framework, native UIs per OS, the OpenAI-compatible endpoint everywhere. The
research themes are its second half.
research themes are its second half. Two steps join it: the Vulkan and OpenCL GPUs (a build that
carries each, a measured prefill run, then the host-memory route for streamed experts), a
Connect view that tells any client what it needs (#257), and one diagram of the planner's logic
for `docs/hardware-planning.md` and the README.
### Changed
- **A load says it is planning.** `state` reads `planning` from the moment a load in auto mode

View file

@ -68,6 +68,31 @@ can return with a measurement behind it, a Windows run. The plan already prints
to what happened; collecting that pair from every run and from the community benchmarks is how the
coefficients get corrected on machines nobody here owns.
**The GPUs most machines have: Vulkan and OpenCL.** Of the eighteen backends the pinned llama.cpp
carries, three have a measurement behind them here ([backends.md](backends.md)), and the GPU in
most phones and PCs is reached through neither of the two GPU ones measured (Metal, Hexagon).
Vulkan reaches any GPU with a driver - discrete and integrated on Windows and Linux, and Android -
and OpenCL is llama.cpp's Adreno path. Both name the expert op, and both implement wrapping host
memory without advertising it, which is the capability streamed experts computed on a device would
need. The work, in order: carry each backend in a build (a desktop archive with Vulkan, an APK
with OpenCL beside Hexagon), so the planner sees the device with no code here; one measured run of
the prefill device on each, against the CPU and, on the phone, against the NPU; then the
`buffer_from_host_ptr` route for the streamed experts where the measurement says the device wins.
The OpenCL entry in [backends.md](backends.md) carries the one negative so far: the dense set on a
phone's integrated GPU lost 27%. That measured the dense weights, not the prefill.
**Connect a client from the UI.** Serving a coding agent or a chat client is what the endpoint is
for, and today it takes launch options and hand-typed values. One Connect view in every UI, fed by
the server: share on the network with a key, the base URL, model id, context and maximum output a
client must be told, a preset that sizes the context for an agent, and a filled-in snippet per
client. Researched and specified in [#257](https://github.com/Helldez/BigMoeOnEdge/issues/257).
**The planner, drawn.** One diagram of the whole solution, as logic rather than code: the facts
the probes measure, the rules that read them, the ledger that keeps memory honest, and the plan
that comes out with its reasons, for prefill and decode apart. It goes in
[hardware-planning.md](hardware-planning.md) and, in a short form, in the README, so a reader
sees in one picture why the same rules give a different plan on a phone, a laptop and a desktop.
Not now: a cache that resizes itself at run time. It is the agnostic answer to the available-memory
problem and it stays behind the static readings until those have been measured.