--- summary: "Run GGUF chat with managed or existing llama.cpp servers and managed local embeddings" read_when: - You want OpenClaw to install and manage a local llama.cpp server - You want a local model recommendation for your Gateway hardware - You want OpenClaw to connect to an existing llama-server - You want memory search embeddings from a local GGUF model - You are configuring memory.search.provider = "local" title: "llama.cpp Provider" sidebarTitle: "llama.cpp Provider" --- The `llama-cpp` plugin provides one `llama-cpp` model provider. OpenClaw can manage a local `llama-server` or connect to one that you operate. Both choices use `llama-cpp/` references and the OpenAI-compatible transport. ```bash openclaw plugins install @openclaw/llama-cpp-provider openclaw onboard ``` ## Choose server ownership | Setup choice | Process owner | Local embeddings | | --------------------- | ----------------------------- | ---------------- | | Managed local server | OpenClaw | Yes | | Existing llama-server | You or an external supervisor | No | `models.providers.llama-cpp.localService` is the ownership discriminator. If it exists, OpenClaw manages the process. Without it, `baseUrl` identifies an existing endpoint. Switching choices rewrites ownership-specific state on the same provider. It never creates another provider namespace. Managed llama.cpp uses the plugin's non-secret authentication marker. An older `apiKey` SecretRef stays in your configuration but is inactive for this managed transport. Explicit headers and request credentials still resolve normally. Other managed providers retain their own authentication rules; `localService` alone does not make an authored API key optional. ## Managed local server The managed loopback server does not need an API key. Fresh setup leaves `apiKey` unset; the provider supplies its non-secret authentication marker at runtime. Existing configured credentials are preserved for Doctor and explicit secret migration, and external servers retain their own authentication requirements. Choose **Managed local server** when OpenClaw should install, start, and stop the server. Setup reads the **Gateway host's** hardware and recommends a model from its available memory, GPU capability, and free disk space. A browser connected to a remote Gateway installs and runs the model on that Gateway, not on the browser's computer. Review the named host, execution backend, model, and download size, then confirm the download. Setup verifies pinned model files and the llama.cpp build, prepares a loopback endpoint, and checks inference before saving the new default. Guided activation also asks the model to read a temporary file through an OpenClaw tool and return its contents. The tool check uses an isolated workspace without your agent's bootstrap instructions. A plain text reply alone does not pass that check. Each verification check has a 90-second deadline. Changing `agents.defaults.timeoutSeconds` does not extend setup verification. Failures identify whether the response check or tool-use check timed out. If the configured server executable is missing from OpenClaw's current managed installation, chat and local embeddings restore the same verified build before starting it. Custom server paths and installations from older releases still require manual repair or rerunning setup. Managed local models automatically use structured [Tool Search](/tools/tool-search) unless you have explicitly configured it. Optional capabilities remain available. Their schemas load as needed, reducing the input the model must process before replying. Setup does not enable lean mode. Normal chats still include your agent's instructions. On CPU-only hosts, the first reply can take several minutes even after setup verification succeeds. ### Model recommendations Setup prefers these text-only recipes as memory permits. Each uses a 65,536-token context and supports tools: | Model | Chat download | Minimum host memory | | ----------------------- | ------------- | --------------------------- | | Qwen3.5 4B Q4_K_M | About 2.7 GB | 8 GiB | | Qwen3.5 9B Q4_K_M | About 5.7 GB | 16 GiB | | Gemma 4 12B IT Q4_K_M | About 7.1 GB | 24 GiB and GPU acceleration | | Muse Glimmer 30B Q4_K_M | About 16.8 GB | 32 GiB and GPU acceleration | | Qwen3.8 27B UD-Q4_K_M | About 16.5 GB | 32 GiB and GPU acceleration | Qwen3.8 is the first recommendation when its memory budget fits. Muse has a smaller context-cache budget and can fit a 24 GiB NVIDIA card where Qwen3.8 does not. CPU recommendations stop at Qwen3.5 9B. Gemma 4 E2B, E4B, and 26B A4B remain in the catalog for existing routes and cached downloads. The recommendation order is a product default, not a claim that one model wins every task. These are selection floors, not guaranteed fit or speed. Setup reserves memory for the operating system, context cache, runtime, and default embedding model. It accounts for current memory pressure and container memory limits, and may recommend a smaller model when RAM, GPU memory, or disk space is limited. Separate NVIDIA cards are not added together to assume a model will fit. Existing Gemma 4 E4B configurations and cached custom models remain supported. The chat download also includes your configured local embedding model, or EmbeddingGemma by default (about 0.3 GB). Leave additional disk space for the runtime and download staging. Setup checks this before offering a recommended model. When the cache and runtime use independent volumes, setup checks each volume's free space separately. Shared storage pools and volumes whose independence cannot be established use a combined reserve. Custom embedding models may need more memory and disk space than these budgets. ### Execution backends | Gateway host | Managed backend | | ---------------------------------------------------- | ------------------------- | | macOS on Apple silicon | Metal with unified memory | | macOS on Intel | CPU | | Linux x64 or arm64 | CPU | | Windows x64 with a supported NVIDIA GPU | CUDA 12.4 | | Windows x64 without supported CUDA, or Windows arm64 | CPU | The verified Windows CUDA build requires NVIDIA driver 551.78 or newer and compute capability 5.0 or newer. Setup checks the driver and installed runtime's device discovery. When an NVIDIA GPU has no compatible managed CUDA build, setup explains the limitation and names CPU execution in the confirmation. For other acceleration backends, run a compatible server yourself and choose **Existing llama-server**. Runtime validation allows up to two minutes per executable version check so macOS security assessment and Metal initialization can finish after installation or when reusing a runtime. Setup remains cancellable during validation. The verified macOS builds require macOS 13.3 or later, and setup stops before downloading on older releases. To keep managed chat and local embeddings there, build `llama-server` on that Mac and set `models.providers.llama-cpp.localService.command` to its absolute path; see [Local model services](/gateway/local-model-services). If no recommendation fits, setup explains whether to free memory, free disk space, or fix cache-directory permissions. Cancelling or failing guided verification leaves the previous default model selected. A setup candidate has its own server preset, so verification does not rewrite an existing managed server's preset. Downloaded files may remain cached for a retry. Setup verifies and reuses cached recommendations, and charges disk space only for missing model and runtime files. Managed router presets retain configured chat models in a fixed order and remove model sections outside that inventory. Chat and embedding preparation update their owned settings while preserving the header, `[*]` defaults, comments, and additional options on retained models. Embedding-only setup uses a fresh preset. Managed embeddings default to one server slot (`parallel = 1`), a 2,048-token context, and a 2,048-token physical batch (`ubatch-size`). Memory indexing reserves four positions for tokenizer overhead and the server's context boundary. These defaults also apply to custom GGUFs and legacy EmbeddingGemma cache paths. One slot bounds llama.cpp's large embedding output buffer to one input; the context bound avoids reserving a custom model's full training context for short memory chunks. `memory.search.outputDimensionality` truncates local embedding vectors before L2 normalization, preserving the requested size in the index identity. Use a GGUF model that supports dimensionality reduction. Without this setting, vectors retain their native size; requesting a larger size than the model returns fails. If an existing index contains native-size vectors despite a smaller configured size, rebuild it with `openclaw memory index --force --agent `. Existing context, batch, and slot settings in the embedding section, `[*]`, or the managed service's arguments or environment remain authoritative. OpenClaw adds the context bound only when no slot count or per-slot context is configured, because llama.cpp can divide total context across slots. If you choose a different slot count, size its context too. Long queries or direct embedding requests may need larger native capacity settings on models that support them. Missing defaults are added on the next managed preset preparation. ### Set up only local embeddings When `memory.search.provider` is `local` and chat setup cannot proceed or is declined, OpenClaw offers a separate embedding-only setup. It installs only the managed server and the configured embedding model after explicit consent. It does not add a llama.cpp chat model or change the current chat model. Setup discovery remains read-only and never installs or downloads anything. If no recommended chat model fits your memory budget, enable local memory search and retry setup to get the embedding-only offer (about 0.3 GB for the default embedding model): ```bash openclaw config set memory.search.provider local openclaw models auth login --provider llama-cpp --method local ``` If you use `--profile`, use the same profile for both commands. If the llama.cpp provider has any configured chat models, embedding-only setup leaves it unchanged. Move any chat routes to another provider and remove those model entries before retrying. An existing external llama.cpp server config must also be removed before OpenClaw can manage embeddings. ### Use another managed GGUF Add a model under `models.providers.llama-cpp.models`, select its `llama-cpp/` reference, and run managed setup again: ```json5 { id: "my-local-model", name: "My local GGUF", reasoning: false, input: ["text"], cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, contextWindow: 65536, maxTokens: 2048, params: { modelPath: "~/Models/my-model.Q4_K_M.gguf", contextSize: 65536, }, compat: { supportsTools: true }, } ``` `modelPath` accepts local paths, cache-relative filenames, full `hf:` file URIs, and HTTPS GGUF URLs that publish a SHA-256 response digest. The default cache is `~/.openclaw/models/llama.cpp`. A configured `modelCacheDir` remains authoritative for managed setup. Setup resolves model aliases and account-qualified references to the selected authored model, and keeps that model even when it is not cached. It shows the download size from the pinned recipe or source metadata when available and asks before downloading. Invalid sources produce a model-specific error instead of a different recommendation. Declining leaves the configured routes unchanged. The hardware recommendation applies when no managed chat models are configured; custom models remain subject to the source's checksum and runtime verification. ## Managed local OCR and vision Run this on the Gateway host: ```bash openclaw models auth login --provider llama-cpp --method local-media ``` Setup recommends separate OCR and vision recipes from the Gateway's hardware, current available memory, supported runtime backend, and free disk. It shows the host, backend, model and projector downloads, memory budget, rejected candidates, and degraded modes before asking permission. No download occurs before consent. The initial automatic media setup scope is **Linux x64 with CPU execution**. Other managed backends remain available for chat and embeddings, but automatic OCR and vision activation on those hosts waits for matching end-to-end verification. Use a Linux x64 Gateway for this setup, or configure another image provider. Updating OpenClaw never activates these routes; setup requires explicit consent. The curated recipes use pinned repository revisions, sizes and SHA-256 checksums for **both** the model and multimodal projector. The existing managed installer verifies the llama.cpp runtime. Setup then runs two real images through the registered image provider: transcription of `OPENCLAW OCR 4827`, and a question about a red square above a blue circle, with no text in the second image. Both checks must pass before configuration is saved. Cancellation or failure retains the previous configuration and removes the candidate preset. Verified downloads remain cached for retry. | Role | Recipe | Model + projector download | Model memory budget | | ------ | -------------------- | -------------------------- | ------------------- | | OCR | GLM-OCR Q8_0 | About 1.43 GB | 4 GiB | | Vision | SmolVLM2 2.2B Q4_K_M | About 1.71 GB | 5 GiB | A complete pair needs at least 6 GiB host RAM, sufficient **available** memory after headroom, and disk for missing files plus runtime staging. These are eligibility floors, not guarantees of speed or accuracy. Only one router model is resident at a time, including existing chat and embedding models. Switching tasks can incur model-loading latency. No additional model process manager is installed. ### Choose the image task The `local_image` tool accepts a local path or `media://` reference and an explicit task: ```json { "path": "./receipt.png", "task": "ocr" } ``` ```json { "path": "./diagram.png", "task": "vision", "prompt": "Explain the relationship between these shapes." } ``` Omitting `task` selects vision. OCR transcribes visible text; it does not replace visual interpretation of photographs, charts, maps or interfaces. The tool maps the typed task to the installed model and uses the same registered provider as normal image understanding. Prompt keywords never select the route. Local file permissions, input limits, cancellation and output limits still apply. The tool is unavailable in sandboxes because it requires the host's local file access. Setup records these selections in `models.providers.llama-cpp.params.mediaModels`, adds the two image-capable models to the existing inventory, sets an explicit image route in `tools.media.models`, and selects local vision as `agents.defaults.imageModel` with no fallbacks. Existing chat defaults, embedding configuration, model inventory, provider settings and retained preset comments remain intact. Existing media entries need explicit `capabilities` so setup can preserve audio/video routes while replacing image routes. External servers and managed direct-model commands must first be configured as managed routers. Router command-line limits override model presets. Setup rejects image-token, context, or parallelism arguments that exceed the media recipes' budgets; lower those limits or place them in the existing models' individual preset sections. Compatible lower limits and unrelated router options are preserved. The CLI can also select the installed OCR route explicitly: ```bash openclaw infer image describe --file ./receipt.png \ --model llama-cpp/glm-ocr-q8_0 --timeout-ms 600000 --json ``` ### Privacy and limits The managed image routes use the loopback server without remote fallback. They reject explicit request proxies and environment proxy routing for that address. If your host uses an HTTP proxy, include the managed server address in `NO_PROXY` (for example, `127.0.0.1,::1`) before setup and inference. Inference never downloads missing models or projectors; rerun setup to repair the cache. Original media remains available to the normal pipeline. **An existing cloud chat model can still receive original images under its native-vision policy.** Local image preprocessing does not change the main chat provider. The initial models process one image per request, with a 10 MiB input ceiling and bounded image encoding and output. Dense text may need separate crops. GLM-OCR does not install document layout detection or PDF rendering. Small vision models can misinterpret complex diagrams and spatial relationships; setup checks basic functionality, not general accuracy. Each verification request has a deadline of up to ten minutes to accommodate CPU inference; a shorter configured image timeout still applies. ## Existing llama-server Choose **Existing llama-server** when another terminal, container, service manager, or machine owns the process. Give the model a stable alias: ```bash llama-server \ --model /path/to/model.gguf \ --alias my-model \ --host 127.0.0.1 \ --port 8080 ``` Run `openclaw onboard`, choose **Existing llama-server**, and enter the endpoint. Enable API-key authentication only when the server or proxy requires it. The URL prompt accepts HTTP or HTTPS endpoints and host shorthand such as `localhost:8080`. Invalid URLs and embedded credentials are rejected inline so you can correct the endpoint without restarting setup. ```bash openclaw models list --provider llama-cpp openclaw models set llama-cpp/my-model ``` OpenClaw reads `/health`, `/models` (falling back to `/v1/models`), and `/props`. Router property checks use `autoload=false`. Discovery never loads, wakes, unloads, downloads, or reloads models. Discovery and inference share the provider's private-network request policy. `models.providers.llama-cpp.request.allowPrivateNetwork: true` permits both to reach an operator-controlled link-local server, such as a Podman host gateway. The same opt-in also permits addresses otherwise blocked by the inference policy, including metadata addresses; use it only for a trusted endpoint. Setting it to `false` disables private-network and configured-origin trust for both discovery and inference. Discovery has no separate retry or redirect exception. For discovered models, OpenClaw advertises reasoning and effort controls only when `/props` sets `chat_template_caps.supports_reasoning_effort` to `true`. Missing or false values leave those capabilities unadvertised. Explicit configured context and tool values remain authoritative over discovery. Setup saves context limits and tool support only when live `/props` verifies them. Unloaded router models remain in the configured inventory with those fields unset. Runtime discovery uses a 128,000-token context fallback and keeps tools disabled while those facts are unknown; it does not save these defaults as model overrides. After loading a model yourself, refresh its live facts: ```bash openclaw models list --provider llama-cpp --refresh ``` Refresh fills unset context and tool fields in the runtime catalog from live `/props`, without changing your explicit overrides or loading other models. Older setup versions may have saved `contextWindow: 128000`, `contextTokens: 128000`, and `compat.supportsTools: false` for unloaded models. If those were generated by setup rather than values you intended to pin, remove the affected model row from `models.providers.llama-cpp.models`, then refresh or rerun **Existing llama-server** setup. Load the model first to discover its actual limits. Rerunning setup alone preserves existing overrides. Refreshing a configured external server reports authentication rejection or unavailability when discovery fails. Previously discovered models remain visible only while their endpoint and credentials are unchanged. A successful empty list removes discovered rows. Explicit configured models remain. Restore the server or correct its credentials, then refresh again to recover the live inventory. ### Authentication and endpoint replacement Existing endpoints support no auth, API keys, SecretRefs, auth profiles, and explicit authorization headers. An explicit `Authorization` header wins over ambient API-key discovery unless setup receives a new key. Choosing no API key removes the default llama.cpp auth profile and stale inline key fields while preserving an explicit `Authorization` header and unrelated headers. Endpoint URLs containing a username or password are rejected. ```bash export LLAMA_SERVER_API_KEY="" openclaw onboard ``` When the endpoint changes, setup does not send the old endpoint's environment, profile, configured key, or header credentials to the replacement. Switching from managed mode also removes `localService`, managed model/cache parameters, and the managed request timeout before discovery. For non-interactive setup: ```bash openclaw onboard \ --non-interactive \ --accept-risk \ --auth-choice llama-cpp-existing-server \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id my-model ``` Use `--llama-server-api-key ` when a replacement endpoint requires a new credential. `LLAMA_SERVER_API_KEY` remains available for initial setup and unchanged endpoints. ### Manual configuration Guided setup is recommended because it verifies discovery. The minimal manual shape is: ```json5 { models: { mode: "merge", providers: { "llama-cpp": { baseUrl: "http://127.0.0.1:8080/v1", api: "openai-completions", request: { allowPrivateNetwork: true }, models: [], }, }, }, } ``` Custom provider IDs may also point at llama-server through the generic OpenAI-compatible path. They remain custom providers and should declare the `llamacpp` tool-schema profile explicitly. See [custom provider capability declarations](/gateway/config-tools#custom-provider-capability-declarations). ## Requests and local embeddings Both ownership choices use OpenClaw's normal chat, image, streaming, and tool transport. The llama.cpp compatibility family cleans unsupported tool-schema constraints. Agent turns and standalone completions also map thinking-off requests to the server's chat-template flag and adapt JSON Schema requests for older llama-server builds. For tool continuations with an unchanged prompt prefix, OpenClaw uses the server's measured context usage plus newly appended content to budget the next response. This avoids charging historical reasoning that the chat template discards. The first request of an attempt, a request with a changed prefix, or a request with a response schema keeps the conservative payload estimate. Replayed reasoning is preserved for templates that need it, and genuinely full contexts still trigger output limits or compaction. Local memory embeddings require managed mode: For automated embedding-only setup, configure local memory and explicitly consent to downloading the embedding model and verified llama.cpp runtime: This unattended option is not available in v2026.10.5. If your installation rejects `--llama-cpp-embedding-only`, use the [interactive embedding-only setup](/plugins/llama-cpp#set-up-only-local-embeddings) instead. ```bash openclaw config set memory.search.provider local openclaw onboard --non-interactive --accept-risk \ --auth-choice llama-cpp --llama-cpp-embedding-only openclaw memory index --force openclaw memory status --deep ``` `--llama-cpp-embedding-only` installs only the embedding model (about 0.3 GB by default) and server runtime. It uses the same setup path as interactive embedding-only setup and preserves your chat model. It honors an enabled agent's configured embedding model; agents using local memory must agree on that model. An existing external server or managed chat route cannot be replaced by this embedding-only path. Without the consent flag, non-interactive managed setup fails before downloading anything. To choose the embedding model explicitly: ```json5 { memory: { search: { provider: "local", local: { modelPath: "hf:ggml-org/embeddinggemma-300m-qat-q8_0-GGUF/embeddinggemma-300m-qat-Q8_0.gguf", }, }, }, } ``` The default EmbeddingGemma model automatically receives different task prefixes for search queries and indexed documents. Existing indexes built without these prefixes rebuild once on the next search or sync, without `--force`. Cached raw embeddings are not reused, and keyword search remains available during recovery. Run `openclaw memory status --index` after intentionally changing the embedding model. ## Troubleshooting - Managed setup: run `openclaw doctor` and `openclaw memory status --deep`. - Managed runtime status requires healthy `/health`, `/models`, and `/props` endpoints and no model-load error. Optional `/metrics` can be `unavailable` when a custom server omits `--metrics`; this does not make the runtime failed. - Existing server: inspect `/health`, `/models`, and `/props`. HTTP 503 means the model is still loading. - Missing tools: verify both tool capability flags in `/props` and use a tool-capable Jinja chat template. - Managed Linux builds require glibc 2.34 on x64 or 2.38 on arm64. Windows builds require the Microsoft Visual C++ 2015-2022 Redistributable. - If a Linux upgrade removes a shared library, setup and embedding diagnostics name the missing library and its runtime package. For `libssl.so.3` or `libcrypto.so.3`, install OpenSSL 3 (`libssl3t64` or `libssl3` on Debian/Ubuntu); for `libgomp.so.1`, install OpenMP (`libgomp1` on Debian/Ubuntu). OpenSSL 4 does not replace OpenSSL 3 for the pinned build. Restore the missing runtime, rerun llama.cpp setup, then verify with `openclaw memory status --deep` and retry `openclaw memory index --force`. Re-downloading the same build does not supply system libraries; existing memory indexes are preserved while embeddings are unavailable. - A model that replies to a simple prompt but fails the setup tool check is not selected as the default. Retry after reviewing its tool support or choose another model. - Platforms without a verified managed build should use an existing server. - Managed embedding server keeps exiting or is OOM-killed: serve embeddings from an external [llmman](/providers/llmman#advanced-configuration) daemon instead. It relaunches a dead backend on the next request, and the Gateway manages no child process. OpenClaw does not auto-select ROCm, SYCL, OpenVINO, or Vulkan archives. ## Related - [Local model services](/gateway/local-model-services) - [Model providers](/concepts/model-providers) - [LM Studio](/providers/lmstudio) - [Llama Cpp plugin reference](/plugins/reference/llama-cpp) — manifest and config reference for the managed and external llama.cpp servers