mirror of
https://github.com/Helldez/BigMoeOnEdge.git
synced 2026-10-19 15:33:46 +00:00
5a07bd6b9a
Some checks failed
ci / changes (push) Has been cancelled
ci / host-windows (push) Has been cancelled
ci / version (push) Has been cancelled
ci / host-macos (push) Has been cancelled
ci / ios-harness (push) Has been cancelled
ci / android-apk (push) Has been cancelled
ci / format (push) Has been cancelled
ci / host-linux (push) Has been cancelled
ci / ui (push) Has been cancelled
A second front-end next to bmoe-cli. bmoe-server keeps one model loaded in one Session and serves a browser UI on 127.0.0.1:8765: chat with the reasoning channel, live engine metrics per token, model management and every engine setting. The settings form is rendered from the parameter table alone (GET /api/params), so a knob added to the engine appears in the UI with no UI change, and the UI names no engine parameter. The same server speaks an OpenAI-compatible /v1/chat/completions, streaming, so any OpenAI client can drive the engine. The API is in docs/server-api.md. Layout. server/ holds HTTP, files and the platform, none of it in core/: conversation, settings_store, event_bus and json_util are pure policy (bmoe_server_core, unit-tested without a model); engine_host owns the one Session; model_library the models folder, the catalog and resumable downloads; planner_adapter offers /api/plan; platform.cpp is the only file with platform conditionals. ui/ is Svelte and TypeScript. The model catalog is data, catalog/models.json. Behaviour. Settings live in layers (defaults, plan, launch flags, user) and only the user's are saved. A chat continues the engine's KV cache when the request extends the previous conversation by one message. With the planner built in every load plans first, with the previous model unloaded so the probes measure the machine and not the model in it. Security model, stated in docs/server-api.md. The server binds to loopback unless --host says otherwise and answers only to loopback host names. A request to /api/* or /v1/* from another site is refused on every method, whether it carries a foreign Origin or is marked Sec-Fetch-Site: cross-site. Loads, unloads and plan measurements are serialised, request bodies are capped at 8 MiB, and a parameter that makes the engine write a file is refused over the API. It does not protect against other local processes or users, and --host on a network has neither authentication nor TLS. Build and release. BMOE_BUILD_SERVER (on by default, off for Android) builds bmoe-server and stages ui/dist and the catalog beside it. CI formats server/ and compile-checks the server on macOS; release-host builds the UI and attaches a bmoe-desktop archive per host target next to bmoe-cli. Tool calling and conversations. /v1/chat/completions takes tools, tool_choice and tool messages and returns tool_calls, streamed or not; the model's own chat template writes the tools and llama.cpp's chat layer parses the calls back, so the server knows no tool syntax. The conversation is the client's: every request carries the whole history and the engine reuses the leading tokens its KV cache holds. Requests that arrive while one runs wait their turn. A coding agent (opencode) fixes a file through it. Hardening. Every POST, PUT and DELETE to /api/* or /v1/* must declare Content-Type: application/json, checked ahead of routing, and the chat view drops form controls and styles from model output: together they stop markup in an answer from posting to the API from the UI's own origin. max_tokens is clamped to the n-predict bound. CI runs the UI's unit tests. The plan's arguments reach /api/plan as the list the plan renders from the parameter table, not as text parsed back. A split model's storage is probed where its experts are.
54 lines
2.9 KiB
Bash
54 lines
2.9 KiB
Bash
#!/usr/bin/env bash
|
|
# Build bmoe-cli with the Hexagon NPU backend, inside upstream's Snapdragon toolchain container
|
|
# (ghcr.io/snapdragon-toolchain/arm64-android, which carries the NDK and the Hexagon SDK; no
|
|
# Qualcomm account needed). This is the release recipe: .github/workflows/release-apk.yml runs it.
|
|
# From the host:
|
|
#
|
|
# docker run --rm -v <repo>:/workspace ghcr.io/snapdragon-toolchain/arm64-android:v0.7 \
|
|
# bash /workspace/scripts/build-hexagon-android.sh
|
|
#
|
|
# From a git worktree whose third_party/llama.cpp is a Windows junction, the junction does not
|
|
# resolve inside the container: mount the real submodule over it with a second
|
|
# `-v <checkout>/third_party/llama.cpp:/workspace/third_party/llama.cpp`.
|
|
#
|
|
# The CPU side is built exactly as the CPU-only release (scripts/build-android.ps1): same API level,
|
|
# same ARM target, so the decode, which stays on the CPU, is the same binary code on every phone,
|
|
# and the app still installs where it did. The Hexagon backend only registers where the phone has
|
|
# the fastrpc driver, and the engine only opens it when asked for the prefill.
|
|
#
|
|
# Output in $OUT (default /workspace/build-hexagon): the CLI, the ggml/llama shared libs,
|
|
# libggml-hexagon.so, one DSP-side skel libggml-htp-v<arch>.so per NPU generation (the backend
|
|
# loads the one matching the phone), and this NDK's libc++_shared.so (the one the libs were built
|
|
# against). scripts/stage-hexagon-jnilibs.ps1 puts them in the app.
|
|
#
|
|
# HTP_ARCHS skels to build (default: every generation the backend supports, v73..v81)
|
|
# CPU_ARCH GGML_CPU_ARM_ARCH (default: the release's armv8.2-a+dotprod+fp16)
|
|
set -euo pipefail
|
|
|
|
HTP_ARCHS="${HTP_ARCHS:-v73 v75 v79 v81}"
|
|
CPU_ARCH="${CPU_ARCH:-armv8.2-a+dotprod+fp16}"
|
|
OUT="${OUT:-/workspace/build-hexagon}"
|
|
|
|
cmake -S /workspace -B "$OUT" \
|
|
-G Ninja \
|
|
-DCMAKE_TOOLCHAIN_FILE="$ANDROID_NDK_ROOT/build/cmake/android.toolchain.cmake" \
|
|
-DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-29 \
|
|
-DCMAKE_BUILD_TYPE=Release \
|
|
-DCMAKE_C_FLAGS="-D_GNU_SOURCE" -DCMAKE_CXX_FLAGS="-D_GNU_SOURCE" \
|
|
-DBMOE_BUILD_TESTS=OFF -DBMOE_BUILD_SERVER=OFF \
|
|
-DGGML_NATIVE=OFF -DGGML_OPENMP=OFF -DGGML_OPENCL=OFF -DGGML_LLAMAFILE=OFF \
|
|
-DGGML_CPU_ARM_ARCH="$CPU_ARCH" \
|
|
-DGGML_HEXAGON=ON \
|
|
-DHEXAGON_SDK_ROOT="$HEXAGON_SDK_ROOT" -DHEXAGON_TOOLS_ROOT="$HEXAGON_TOOLS_ROOT" \
|
|
-DPREBUILT_LIB_DIR=android_aarch64 \
|
|
-DLLAMA_CURL=OFF -DLLAMA_OPENSSL=OFF
|
|
|
|
# The DSP skels are ExternalProject targets, and the llama.cpp subdirectory is EXCLUDE_FROM_ALL here,
|
|
# so they are only built when asked for by name.
|
|
targets=(bmoe-cli)
|
|
for v in $HTP_ARCHS; do targets+=("htp-$v"); done
|
|
cmake --build "$OUT" --target "${targets[@]}" -j "$(nproc)"
|
|
|
|
cp -f "$ANDROID_NDK_ROOT/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/libc++_shared.so" "$OUT/"
|
|
echo "built:"
|
|
find "$OUT" -maxdepth 6 \( -name 'bmoe-cli' -o -name 'lib*.so' \) -newer "$OUT/CMakeCache.txt" | sort
|