mirror of
https://github.com/Helldez/BigMoeOnEdge.git
synced 2026-10-19 07:23:46 +00:00
docs(roadmap): Vulkan and OpenCL GPUs, a Connect view for clients, the planner drawn (#258)
Some checks are pending
ci / version (push) Waiting to run
ci / format (push) Blocked by required conditions
ci / host-linux (push) Blocked by required conditions
ci / ui (push) Blocked by required conditions
ci / host-windows (push) Waiting to run
ci / host-macos (push) Waiting to run
ci / ios-harness (push) Waiting to run
ci / changes (push) Waiting to run
ci / android-apk (push) Waiting to run
Some checks are pending
ci / version (push) Waiting to run
ci / format (push) Blocked by required conditions
ci / host-linux (push) Blocked by required conditions
ci / ui (push) Blocked by required conditions
ci / host-windows (push) Waiting to run
ci / host-macos (push) Waiting to run
ci / ios-harness (push) Waiting to run
ci / changes (push) Waiting to run
ci / android-apk (push) Waiting to run
Three steps join the product plan: carry the Vulkan and OpenCL backends in a build, measure the prefill device on each and then the host-memory route for streamed experts; one Connect view that tells any OpenAI-compatible client what it needs (#257); and one diagram of the planner's logic for docs/hardware-planning.md and the README.
This commit is contained in:
parent
35242b5226
commit
0207ed9382
2 changed files with 29 additions and 1 deletions
|
|
@ -104,7 +104,10 @@ Semantic Versioning.
|
|||
same thing their own way (#236).
|
||||
- `docs/roadmap.md` opens with the product plan, in the order the work has to happen: one engine,
|
||||
one application framework, native UIs per OS, the OpenAI-compatible endpoint everywhere. The
|
||||
research themes are its second half.
|
||||
research themes are its second half. Two steps join it: the Vulkan and OpenCL GPUs (a build that
|
||||
carries each, a measured prefill run, then the host-memory route for streamed experts), a
|
||||
Connect view that tells any client what it needs (#257), and one diagram of the planner's logic
|
||||
for `docs/hardware-planning.md` and the README.
|
||||
|
||||
### Changed
|
||||
- **A load says it is planning.** `state` reads `planning` from the moment a load in auto mode
|
||||
|
|
|
|||
|
|
@ -68,6 +68,31 @@ can return with a measurement behind it, a Windows run. The plan already prints
|
|||
to what happened; collecting that pair from every run and from the community benchmarks is how the
|
||||
coefficients get corrected on machines nobody here owns.
|
||||
|
||||
**The GPUs most machines have: Vulkan and OpenCL.** Of the eighteen backends the pinned llama.cpp
|
||||
carries, three have a measurement behind them here ([backends.md](backends.md)), and the GPU in
|
||||
most phones and PCs is reached through neither of the two GPU ones measured (Metal, Hexagon).
|
||||
Vulkan reaches any GPU with a driver - discrete and integrated on Windows and Linux, and Android -
|
||||
and OpenCL is llama.cpp's Adreno path. Both name the expert op, and both implement wrapping host
|
||||
memory without advertising it, which is the capability streamed experts computed on a device would
|
||||
need. The work, in order: carry each backend in a build (a desktop archive with Vulkan, an APK
|
||||
with OpenCL beside Hexagon), so the planner sees the device with no code here; one measured run of
|
||||
the prefill device on each, against the CPU and, on the phone, against the NPU; then the
|
||||
`buffer_from_host_ptr` route for the streamed experts where the measurement says the device wins.
|
||||
The OpenCL entry in [backends.md](backends.md) carries the one negative so far: the dense set on a
|
||||
phone's integrated GPU lost 27%. That measured the dense weights, not the prefill.
|
||||
|
||||
**Connect a client from the UI.** Serving a coding agent or a chat client is what the endpoint is
|
||||
for, and today it takes launch options and hand-typed values. One Connect view in every UI, fed by
|
||||
the server: share on the network with a key, the base URL, model id, context and maximum output a
|
||||
client must be told, a preset that sizes the context for an agent, and a filled-in snippet per
|
||||
client. Researched and specified in [#257](https://github.com/Helldez/BigMoeOnEdge/issues/257).
|
||||
|
||||
**The planner, drawn.** One diagram of the whole solution, as logic rather than code: the facts
|
||||
the probes measure, the rules that read them, the ledger that keeps memory honest, and the plan
|
||||
that comes out with its reasons, for prefill and decode apart. It goes in
|
||||
[hardware-planning.md](hardware-planning.md) and, in a short form, in the README, so a reader
|
||||
sees in one picture why the same rules give a different plan on a phone, a laptop and a desktop.
|
||||
|
||||
Not now: a cache that resizes itself at run time. It is the agnostic answer to the available-memory
|
||||
problem and it stays behind the static readings until those have been measured.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue