openclaw/docs/plugins/llama-cpp.md

summary	read_when	title	sidebarTitle
Run GGUF chat with managed or existing llama.cpp servers and managed local embeddings	You want OpenClaw to install and manage a local llama.cpp server You want a local model recommendation for your Gateway hardware You want OpenClaw to connect to an existing llama-server You want memory search embeddings from a local GGUF model You are configuring memory.search.provider = "local"	llama.cpp Provider	llama.cpp Provider

The llama-cpp plugin provides one llama-cpp model provider. OpenClaw can manage a local llama-server or connect to one that you operate. Both choices use llama-cpp/<model> references and the OpenAI-compatible transport.

openclaw plugins install @openclaw/llama-cpp-provider
openclaw onboard

Choose server ownership

Setup choice	Process owner	Local embeddings
Managed local server	OpenClaw	Yes
Existing llama-server	You or an external supervisor	No

models.providers.llama-cpp.localService is the ownership discriminator. If it exists, OpenClaw manages the process. Without it, baseUrl identifies an existing endpoint. Switching choices rewrites ownership-specific state on the same provider. It never creates another provider namespace.

Managed llama.cpp uses the plugin's non-secret authentication marker. An older apiKey SecretRef stays in your configuration but is inactive for this managed transport. Explicit headers and request credentials still resolve normally. Other managed providers retain their own authentication rules; localService alone does not make an authored API key optional.

Managed local server

The managed loopback server does not need an API key. Fresh setup leaves apiKey unset; the provider supplies its non-secret authentication marker at runtime. Existing configured credentials are preserved for Doctor and explicit secret migration, and external servers retain their own authentication requirements.

Choose Managed local server when OpenClaw should install, start, and stop the server. Setup reads the Gateway host's hardware and recommends a model from its available memory, GPU capability, and free disk space. A browser connected to a remote Gateway installs and runs the model on that Gateway, not on the browser's computer.

Review the named host, execution backend, model, and download size, then confirm the download. Setup verifies pinned model files and the llama.cpp build, prepares a loopback endpoint, and checks inference before saving the new default. Guided activation also asks the model to read a temporary file through an OpenClaw tool and return its contents. The tool check uses an isolated workspace without your agent's bootstrap instructions. A plain text reply alone does not pass that check. Each verification check has a 90-second deadline. Changing agents.defaults.timeoutSeconds does not extend setup verification. Failures identify whether the response check or tool-use check timed out.

If the configured server executable is missing from OpenClaw's current managed installation, chat and local embeddings restore the same verified build before starting it. Custom server paths and installations from older releases still require manual repair or rerunning setup.

Managed local models automatically use structured Tool Search unless you have explicitly configured it. Optional capabilities remain available. Their schemas load as needed, reducing the input the model must process before replying. Setup does not enable lean mode. Normal chats still include your agent's instructions. On CPU-only hosts, the first reply can take several minutes even after setup verification succeeds.

Model recommendations

Setup prefers these text-only recipes as memory permits. Each uses a 65,536-token context and supports tools:

Model	Chat download	Minimum host memory
Qwen3.5 4B Q4_K_M	About 2.7 GB	8 GiB
Qwen3.5 9B Q4_K_M	About 5.7 GB	16 GiB
Gemma 4 12B IT Q4_K_M	About 7.1 GB	24 GiB and GPU acceleration
Muse Glimmer 30B Q4_K_M	About 16.8 GB	32 GiB and GPU acceleration
Qwen3.8 27B UD-Q4_K_M	About 16.5 GB	32 GiB and GPU acceleration

Qwen3.8 is the first recommendation when its memory budget fits. Muse has a smaller context-cache budget and can fit a 24 GiB NVIDIA card where Qwen3.8 does not. CPU recommendations stop at Qwen3.5 9B. Gemma 4 E2B, E4B, and 26B A4B remain in the catalog for existing routes and cached downloads. The recommendation order is a product default, not a claim that one model wins every task.

These are selection floors, not guaranteed fit or speed. Setup reserves memory for the operating system, context cache, runtime, and default embedding model. It accounts for current memory pressure and container memory limits, and may recommend a smaller model when RAM, GPU memory, or disk space is limited. Separate NVIDIA cards are not added together to assume a model will fit. Existing Gemma 4 E4B configurations and cached custom models remain supported.

The chat download also includes your configured local embedding model, or EmbeddingGemma by default (about 0.3 GB). Leave additional disk space for the runtime and download staging. Setup checks this before offering a recommended model. When the cache and runtime use independent volumes, setup checks each volume's free space separately. Shared storage pools and volumes whose independence cannot be established use a combined reserve. Custom embedding models may need more memory and disk space than these budgets.

Execution backends

Gateway host	Managed backend
macOS on Apple silicon	Metal with unified memory
macOS on Intel	CPU
Linux x64 or arm64	CPU
Windows x64 with a supported NVIDIA GPU	CUDA 12.4
Windows x64 without supported CUDA, or Windows arm64	CPU

The verified Windows CUDA build requires NVIDIA driver 551.78 or newer and compute capability 5.0 or newer. Setup checks the driver and installed runtime's device discovery. When an NVIDIA GPU has no compatible managed CUDA build, setup explains the limitation and names CPU execution in the confirmation. For other acceleration backends, run a compatible server yourself and choose Existing llama-server.

Runtime validation allows up to two minutes per executable version check so macOS security assessment and Metal initialization can finish after installation or when reusing a runtime. Setup remains cancellable during validation.

The verified macOS builds require macOS 13.3 or later, and setup stops before downloading on older releases. To keep managed chat and local embeddings there, build llama-server on that Mac and set models.providers.llama-cpp.localService.command to its absolute path; see Local model services.

If no recommendation fits, setup explains whether to free memory, free disk space, or fix cache-directory permissions. Cancelling or failing guided verification leaves the previous default model selected. A setup candidate has its own server preset, so verification does not rewrite an existing managed server's preset. Downloaded files may remain cached for a retry. Setup verifies and reuses cached recommendations, and charges disk space only for missing model and runtime files.

Managed router presets retain configured chat models in a fixed order and remove model sections outside that inventory. Chat and embedding preparation update their owned settings while preserving the header, [*] defaults, comments, and additional options on retained models. Embedding-only setup uses a fresh preset.

Managed embeddings default to one server slot (parallel = 1), a 2,048-token context, and a 2,048-token physical batch (ubatch-size). Memory indexing reserves four positions for tokenizer overhead and the server's context boundary. These defaults also apply to custom GGUFs and legacy EmbeddingGemma cache paths. One slot bounds llama.cpp's large embedding output buffer to one input; the context bound avoids reserving a custom model's full training context for short memory chunks.

memory.search.outputDimensionality truncates local embedding vectors before L2 normalization, preserving the requested size in the index identity. Use a GGUF model that supports dimensionality reduction. Without this setting, vectors retain their native size; requesting a larger size than the model returns fails. If an existing index contains native-size vectors despite a smaller configured size, rebuild it with openclaw memory index --force --agent <id>.

Existing context, batch, and slot settings in the embedding section, [*], or the managed service's arguments or environment remain authoritative. OpenClaw adds the context bound only when no slot count or per-slot context is configured, because llama.cpp can divide total context across slots. If you choose a different slot count, size its context too. Long queries or direct embedding requests may need larger native capacity settings on models that support them. Missing defaults are added on the next managed preset preparation.

Set up only local embeddings

When memory.search.provider is local and chat setup cannot proceed or is declined, OpenClaw offers a separate embedding-only setup. It installs only the managed server and the configured embedding model after explicit consent. It does not add a llama.cpp chat model or change the current chat model. Setup discovery remains read-only and never installs or downloads anything.

If no recommended chat model fits your memory budget, enable local memory search and retry setup to get the embedding-only offer (about 0.3 GB for the default embedding model):

openclaw config set memory.search.provider local
openclaw models auth login --provider llama-cpp --method local

If you use --profile, use the same profile for both commands.

If the llama.cpp provider has any configured chat models, embedding-only setup leaves it unchanged. Move any chat routes to another provider and remove those model entries before retrying. An existing external llama.cpp server config must also be removed before OpenClaw can manage embeddings.

Use another managed GGUF

Add a model under models.providers.llama-cpp.models, select its llama-cpp/<id> reference, and run managed setup again:

{
  id: "my-local-model",
  name: "My local GGUF",
  reasoning: false,
  input: ["text"],
  cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
  contextWindow: 65536,
  maxTokens: 2048,
  params: {
    modelPath: "~/Models/my-model.Q4_K_M.gguf",
    contextSize: 65536,
  },
  compat: { supportsTools: true },
}

modelPath accepts local paths, cache-relative filenames, full hf: file URIs, and HTTPS GGUF URLs that publish a SHA-256 response digest. The default cache is ~/.openclaw/models/llama.cpp. A configured modelCacheDir remains authoritative for managed setup.

Setup resolves model aliases and account-qualified references to the selected authored model, and keeps that model even when it is not cached. It shows the download size from the pinned recipe or source metadata when available and asks before downloading. Invalid sources produce a model-specific error instead of a different recommendation. Declining leaves the configured routes unchanged. The hardware recommendation applies when no managed chat models are configured; custom models remain subject to the source's checksum and runtime verification.

Managed local OCR and vision

Run this on the Gateway host:

openclaw models auth login --provider llama-cpp --method local-media

Setup recommends separate OCR and vision recipes from the Gateway's hardware, current available memory, supported runtime backend, and free disk. It shows the host, backend, model and projector downloads, memory budget, rejected candidates, and degraded modes before asking permission. No download occurs before consent.

The initial automatic media setup scope is Linux x64 with CPU execution. Other managed backends remain available for chat and embeddings, but automatic OCR and vision activation on those hosts waits for matching end-to-end verification. Use a Linux x64 Gateway for this setup, or configure another image provider. Updating OpenClaw never activates these routes; setup requires explicit consent.

The curated recipes use pinned repository revisions, sizes and SHA-256 checksums for both the model and multimodal projector. The existing managed installer verifies the llama.cpp runtime. Setup then runs two real images through the registered image provider: transcription of OPENCLAW OCR 4827, and a question about a red square above a blue circle, with no text in the second image. Both checks must pass before configuration is saved. Cancellation or failure retains the previous configuration and removes the candidate preset. Verified downloads remain cached for retry.

Role	Recipe	Model + projector download	Model memory budget
OCR	GLM-OCR Q8_0	About 1.43 GB	4 GiB
Vision	SmolVLM2 2.2B Q4_K_M	About 1.71 GB	5 GiB

A complete pair needs at least 6 GiB host RAM, sufficient available memory after headroom, and disk for missing files plus runtime staging. These are eligibility floors, not guarantees of speed or accuracy. Only one router model is resident at a time, including existing chat and embedding models. Switching tasks can incur model-loading latency. No additional model process manager is installed.

Choose the image task

The local_image tool accepts a local path or media:// reference and an explicit task:

{ "path": "./receipt.png", "task": "ocr" }

{
  "path": "./diagram.png",
  "task": "vision",
  "prompt": "Explain the relationship between these shapes."
}

Omitting task selects vision. OCR transcribes visible text; it does not replace visual interpretation of photographs, charts, maps or interfaces. The tool maps the typed task to the installed model and uses the same registered provider as normal image understanding. Prompt keywords never select the route. Local file permissions, input limits, cancellation and output limits still apply. The tool is unavailable in sandboxes because it requires the host's local file access.

Setup records these selections in models.providers.llama-cpp.params.mediaModels, adds the two image-capable models to the existing inventory, sets an explicit image route in tools.media.models, and selects local vision as agents.defaults.imageModel with no fallbacks. Existing chat defaults, embedding configuration, model inventory, provider settings and retained preset comments remain intact. Existing media entries need explicit capabilities so setup can preserve audio/video routes while replacing image routes. External servers and managed direct-model commands must first be configured as managed routers. Router command-line limits override model presets. Setup rejects image-token, context, or parallelism arguments that exceed the media recipes' budgets; lower those limits or place them in the existing models' individual preset sections. Compatible lower limits and unrelated router options are preserved.

The CLI can also select the installed OCR route explicitly:

openclaw infer image describe --file ./receipt.png \
  --model llama-cpp/glm-ocr-q8_0 --timeout-ms 600000 --json

Privacy and limits

The managed image routes use the loopback server without remote fallback. They reject explicit request proxies and environment proxy routing for that address. If your host uses an HTTP proxy, include the managed server address in NO_PROXY (for example, 127.0.0.1,::1) before setup and inference. Inference never downloads missing models or projectors; rerun setup to repair the cache. Original media remains available to the normal pipeline. An existing cloud chat model can still receive original images under its native-vision policy. Local image preprocessing does not change the main chat provider.

The initial models process one image per request, with a 10 MiB input ceiling and bounded image encoding and output. Dense text may need separate crops. GLM-OCR does not install document layout detection or PDF rendering. Small vision models can misinterpret complex diagrams and spatial relationships; setup checks basic functionality, not general accuracy. Each verification request has a deadline of up to ten minutes to accommodate CPU inference; a shorter configured image timeout still applies.

Existing llama-server

Choose Existing llama-server when another terminal, container, service manager, or machine owns the process.

Give the model a stable alias:

```bash
llama-server \
  --model /path/to/model.gguf \
  --alias my-model \
  --host 127.0.0.1 \
  --port 8080
```

Run `openclaw onboard`, choose **Existing llama-server**, and enter the endpoint. Enable API-key authentication only when the server or proxy requires it.

The URL prompt accepts HTTP or HTTPS endpoints and host shorthand such as
`localhost:8080`. Invalid URLs and embedded credentials are rejected inline
so you can correct the endpoint without restarting setup.

```bash openclaw models list --provider llama-cpp openclaw models set llama-cpp/my-model ```

OpenClaw reads /health, /models (falling back to /v1/models), and /props. Router property checks use autoload=false. Discovery never loads, wakes, unloads, downloads, or reloads models.

Discovery and inference share the provider's private-network request policy. models.providers.llama-cpp.request.allowPrivateNetwork: true permits both to reach an operator-controlled link-local server, such as a Podman host gateway. The same opt-in also permits addresses otherwise blocked by the inference policy, including metadata addresses; use it only for a trusted endpoint. Setting it to false disables private-network and configured-origin trust for both discovery and inference. Discovery has no separate retry or redirect exception.

For discovered models, OpenClaw advertises reasoning and effort controls only when /props sets chat_template_caps.supports_reasoning_effort to true. Missing or false values leave those capabilities unadvertised. Explicit configured context and tool values remain authoritative over discovery.

Setup saves context limits and tool support only when live /props verifies them. Unloaded router models remain in the configured inventory with those fields unset. Runtime discovery uses a 128,000-token context fallback and keeps tools disabled while those facts are unknown; it does not save these defaults as model overrides. After loading a model yourself, refresh its live facts:

openclaw models list --provider llama-cpp --refresh

Refresh fills unset context and tool fields in the runtime catalog from live /props, without changing your explicit overrides or loading other models. Older setup versions may have saved contextWindow: 128000, contextTokens: 128000, and compat.supportsTools: false for unloaded models. If those were generated by setup rather than values you intended to pin, remove the affected model row from models.providers.llama-cpp.models, then refresh or rerun Existing llama-server setup. Load the model first to discover its actual limits. Rerunning setup alone preserves existing overrides.

Refreshing a configured external server reports authentication rejection or unavailability when discovery fails. Previously discovered models remain visible only while their endpoint and credentials are unchanged. A successful empty list removes discovered rows. Explicit configured models remain. Restore the server or correct its credentials, then refresh again to recover the live inventory.

Authentication and endpoint replacement

Existing endpoints support no auth, API keys, SecretRefs, auth profiles, and explicit authorization headers. An explicit Authorization header wins over ambient API-key discovery unless setup receives a new key. Choosing no API key removes the default llama.cpp auth profile and stale inline key fields while preserving an explicit Authorization header and unrelated headers. Endpoint URLs containing a username or password are rejected.

export LLAMA_SERVER_API_KEY="<API_KEY>"
openclaw onboard

When the endpoint changes, setup does not send the old endpoint's environment, profile, configured key, or header credentials to the replacement. Switching from managed mode also removes localService, managed model/cache parameters, and the managed request timeout before discovery.

For non-interactive setup:

openclaw onboard \
  --non-interactive \
  --accept-risk \
  --auth-choice llama-cpp-existing-server \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id my-model

Use --llama-server-api-key <API_KEY> when a replacement endpoint requires a new credential. LLAMA_SERVER_API_KEY remains available for initial setup and unchanged endpoints.

Manual configuration

Guided setup is recommended because it verifies discovery. The minimal manual shape is:

{
  models: {
    mode: "merge",
    providers: {
      "llama-cpp": {
        baseUrl: "http://127.0.0.1:8080/v1",
        api: "openai-completions",
        request: { allowPrivateNetwork: true },
        models: [],
      },
    },
  },
}

Custom provider IDs may also point at llama-server through the generic OpenAI-compatible path. They remain custom providers and should declare the llamacpp tool-schema profile explicitly. See custom provider capability declarations.

Requests and local embeddings

Both ownership choices use OpenClaw's normal chat, image, streaming, and tool transport. The llama.cpp compatibility family cleans unsupported tool-schema constraints. Agent turns and standalone completions also map thinking-off requests to the server's chat-template flag and adapt JSON Schema requests for older llama-server builds.

For tool continuations with an unchanged prompt prefix, OpenClaw uses the server's measured context usage plus newly appended content to budget the next response. This avoids charging historical reasoning that the chat template discards. The first request of an attempt, a request with a changed prefix, or a request with a response schema keeps the conservative payload estimate. Replayed reasoning is preserved for templates that need it, and genuinely full contexts still trigger output limits or compaction.

Local memory embeddings require managed mode:

For automated embedding-only setup, configure local memory and explicitly consent to downloading the embedding model and verified llama.cpp runtime:

This unattended option is not available in v2026.10.5. If your installation rejects --llama-cpp-embedding-only, use the interactive embedding-only setup instead.

openclaw config set memory.search.provider local
openclaw onboard --non-interactive --accept-risk \
  --auth-choice llama-cpp --llama-cpp-embedding-only
openclaw memory index --force
openclaw memory status --deep

--llama-cpp-embedding-only installs only the embedding model (about 0.3 GB by default) and server runtime. It uses the same setup path as interactive embedding-only setup and preserves your chat model. It honors an enabled agent's configured embedding model; agents using local memory must agree on that model. An existing external server or managed chat route cannot be replaced by this embedding-only path. Without the consent flag, non-interactive managed setup fails before downloading anything.

To choose the embedding model explicitly:

{
  memory: {
    search: {
      provider: "local",
      local: {
        modelPath: "hf:ggml-org/embeddinggemma-300m-qat-q8_0-GGUF/embeddinggemma-300m-qat-Q8_0.gguf",
      },
    },
  },
}

The default EmbeddingGemma model automatically receives different task prefixes for search queries and indexed documents. Existing indexes built without these prefixes rebuild once on the next search or sync, without --force. Cached raw embeddings are not reused, and keyword search remains available during recovery. Run openclaw memory status --index after intentionally changing the embedding model.

Troubleshooting

• Managed setup: run openclaw doctor and openclaw memory status --deep.
• Managed runtime status requires healthy /health, /models, and /props endpoints and no model-load error. Optional /metrics can be unavailable when a custom server omits --metrics; this does not make the runtime failed.
• Existing server: inspect /health, /models, and /props. HTTP 503 means the model is still loading.
• Missing tools: verify both tool capability flags in /props and use a tool-capable Jinja chat template.
• Managed Linux builds require glibc 2.34 on x64 or 2.38 on arm64. Windows builds require the Microsoft Visual C++ 2015-2022 Redistributable.
• If a Linux upgrade removes a shared library, setup and embedding diagnostics name the missing library and its runtime package. For libssl.so.3 or libcrypto.so.3, install OpenSSL 3 (libssl3t64 or libssl3 on Debian/Ubuntu); for libgomp.so.1, install OpenMP (libgomp1 on Debian/Ubuntu). OpenSSL 4 does not replace OpenSSL 3 for the pinned build. Restore the missing runtime, rerun llama.cpp setup, then verify with openclaw memory status --deep and retry openclaw memory index --force. Re-downloading the same build does not supply system libraries; existing memory indexes are preserved while embeddings are unavailable.
• A model that replies to a simple prompt but fails the setup tool check is not selected as the default. Retry after reviewing its tool support or choose another model.
• Platforms without a verified managed build should use an existing server.
• Managed embedding server keeps exiting or is OOM-killed: serve embeddings from an external llmman daemon instead. It relaunches a dead backend on the next request, and the Gateway manages no child process.

OpenClaw does not auto-select ROCm, SYCL, OpenVINO, or Vulkan archives.

Related

• Local model services
• Model providers
• LM Studio
• Llama Cpp plugin reference — manifest and config reference for the managed and external llama.cpp servers