Concedo
4f78d791c5
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/behavior-check.yml
# .github/workflows/build-apple.yml
# CMakeLists.txt
# docs/backend/SYCL.md
# docs/ops.md
# docs/ops/OpenCL.csv
# docs/speculative.md
# ggml/CMakeLists.txt
# ggml/src/ggml-cann/ggml-cann.cpp
# ggml/src/ggml-cpu/amx/amx.cpp
# ggml/src/ggml-cuda/common.cuh
# ggml/src/ggml-cuda/fattn.cu
# ggml/src/ggml-et/ggml-et.cpp
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp-kparams.cpp
# ggml/src/ggml-hexagon/htp-kparams.h
# ggml/src/ggml-hexagon/htp-opnode.cpp
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/concat-ops.c
# ggml/src/ggml-hexagon/htp/concat-ops.h
# ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c
# ggml/src/ggml-hexagon/htp/gated-delta-net-ops.h
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/im2col-ops.c
# ggml/src/ggml-hexagon/htp/im2col-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/ssm-conv.c
# ggml/src/ggml-hexagon/htp/ssm-conv.h
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# ggml/src/ggml-openvino/ggml-openvino.cpp
# ggml/src/ggml-openvino/openvino/op/scale.cpp
# ggml/src/ggml-sycl/CMakeLists.txt
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/convert.cpp
# ggml/src/ggml-sycl/dequantize.hpp
# ggml/src/ggml-sycl/fattn-mkl.cpp
# ggml/src/ggml-sycl/fattn-onednn.cpp
# ggml/src/ggml-sycl/fattn.cpp
# ggml/src/ggml-sycl/getrows.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/mmq.cpp
# ggml/src/ggml-virtgpu/ggml-backend-buffer.cpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-zdnn/ggml-zdnn.cpp
# scripts/hip/gcn-cdna-vgpr-check.py
# scripts/sync_vendor.py
# skills/code-review/SKILL.md
# tests/CMakeLists.txt
# tests/fusion/MTL.csv
# tests/llama-context/cases-kv.cpp
# tests/llama-context/cases-rs.cpp
# tests/llama-context/test-util.cpp
# tests/test-backend-ops.cpp
# tests/test-llama-archs.cpp
# tests/test-quantize-fns.cpp
# tests/test-reasoning-budget.cpp
2026-10-30 23:09:16 +08:00
Adrien Gallouët
aa588a3463
common : drop the Windows argv reconstruction ( #31417 )
...
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
The manifest from #31329 already makes argv UTF-8.
Fixes #26416
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-30 14:02:00 +01:00
Concedo
9792f6b4cb
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/ci-self-hosted-webgpu.yml
# .github/workflows/release.yml
# CMakeLists.txt
# common/fit.h
# scripts/snapdragon/qdc/run_qdc_jobs.py
# tests/fusion/MTL.csv
# tests/llama-context/cases-rs.cpp
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tests/test-jinja.cpp
# tests/test-llama-archs.cpp
# tests/test-llama-context.cpp
# tests/test-server-component.cpp
# tools/mtmd/CMakeLists.txt
# tools/server/CMakeLists.txt
# tools/server/README.md
2026-10-28 22:25:38 +08:00
Tarek Dakhran
19ff817fb2
model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B ( #31257 )
...
* model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* convert : read the LFM3-Audio config.json as released
A transformers that knows lfm3_audio rewrites the config into another layout (no lfm/encoder keys).
Assisted-by: Claude Opus 5.5
* mtmd : disable flash attention for the LFM3-Audio depthformer
Assisted-by: Claude Opus 5.5
* tts : add a voice option for LFM3-Audio and LFM3-Omni
The model card offers a female and a male voice; --tts-voice for llama-tts, "voice" for the server /tts.
Assisted-by: Claude Opus 5.5
* clean up template handling
* convert : detect LFM3-Audio and LFM3-Omni by model_type
Assisted-by: Claude Opus 5.5
* move banned tokens --> logits bias --> mtmd_helper
* leave mtmd-cli unchanged
* no more template_asr_system
* left over
* llama : keep the LFM3-Audio encoder convs and codec unquantized
Assisted-by: Claude Opus 5.5
* common, mtmd : shorten comments
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-10-27 18:47:14 +01:00
Ruben Ortlam
85cc432e2c
server: add router device memory margin parameter for dynamic unloading ( #21231 )
...
* server: add --models-memory-max parameter to allow dynamically unloading models when they exceed a memory size threshold
estimate with to-be-loaded model size included
use no_alloc to get memory requirements for model load
only set model memory_mb if not previously calculated
use memory margin instead of total size limit, apply to each device separately
add server memory debug logging
move llama_context_device_memory function to llama-ext.h
fix model count exceeded check
improve memory_per_device map naming
improve variable naming, fix style
also strip models memory margin from child processes
cont : clean-up
replace device memory map with buft memory map. Use llama_get_memory_breakdown
extract duplicated check into helper function
move model memory estimation to subprocess
precompute name->buft map, map GPU host types to CPU buft
cleanup unused variable
remove duplicated init calls
* adapt to upstream changes
* wip
* common_memory_reports internface
* SERVER_ROLE_MEASURE
* server-models-sched
* fix build
* server : fix memory accounting of measure children and claim spin
A measure child runs no_alloc, so the free memory it reports does not
include its estimate: subtracting self there erased the memory held by
other processes, down to zero with --fit on, and the next load could
overshoot the margin. Subtract self only for instance reports.
A queued load only checked --models-max before claiming, so when the
memory margin blocked it behind a busy model, load() threw and the
waiter retried in a tight loop. Refuse the claim while the limits are
exceeded and no idle model can be unloaded, so the waiter sleeps on
the condition variable until one goes idle.
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
2026-10-27 12:14:42 +01:00
Julien Chaumond
74a1fd3d03
common : --model or -hf is required ( #31242 )
...
Assisted-by: Codex
2026-10-27 11:55:48 +01:00
Concedo
329f6a5b31
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-cpu.yml
# ci/run.sh
# common/CMakeLists.txt
# docs/backend/SYCL.md
# docs/ops.md
# docs/ops/Hexagon.csv
# docs/ops/SYCL.csv
# ggml/src/ggml-cann/ggml-cann.cpp
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-cpu/kleidiai/kernels.cpp
# ggml/src/ggml-cpu/kleidiai/kernels.h
# ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
# ggml/src/ggml-hexagon/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp-opnode.h
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/allreduce-ops.h
# ggml/src/ggml-hexagon/htp/argsort-ops.c
# ggml/src/ggml-hexagon/htp/argsort-ops.h
# ggml/src/ggml-hexagon/htp/binary-ops.c
# ggml/src/ggml-hexagon/htp/binary-ops.h
# ggml/src/ggml-hexagon/htp/concat-ops.c
# ggml/src/ggml-hexagon/htp/concat-ops.h
# ggml/src/ggml-hexagon/htp/cpy-ops.c
# ggml/src/ggml-hexagon/htp/cpy-ops.h
# ggml/src/ggml-hexagon/htp/cumsum-ops.c
# ggml/src/ggml-hexagon/htp/diag-ops.c
# ggml/src/ggml-hexagon/htp/dma-queue.h
# ggml/src/ggml-hexagon/htp/fill-ops.c
# ggml/src/ggml-hexagon/htp/flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/flash-attn-ops.h
# ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c
# ggml/src/ggml-hexagon/htp/gated-delta-net-ops.h
# ggml/src/ggml-hexagon/htp/gather-rows.c
# ggml/src/ggml-hexagon/htp/gather-rows.h
# ggml/src/ggml-hexagon/htp/get-rows-ops.c
# ggml/src/ggml-hexagon/htp/get-rows-ops.h
# ggml/src/ggml-hexagon/htp/glu-ops.c
# ggml/src/ggml-hexagon/htp/hex-common.h
# ggml/src/ggml-hexagon/htp/hex-profile.h
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/hvx-arith.h
# ggml/src/ggml-hexagon/htp/hvx-base.h
# ggml/src/ggml-hexagon/htp/hvx-floor.h
# ggml/src/ggml-hexagon/htp/hvx-mm-kernels-tiled.h
# ggml/src/ggml-hexagon/htp/hvx-norm.h
# ggml/src/ggml-hexagon/htp/hvx-sigmoid.h
# ggml/src/ggml-hexagon/htp/im2col-ops.c
# ggml/src/ggml-hexagon/htp/im2col-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-hexagon/htp/matmul-ops.h
# ggml/src/ggml-hexagon/htp/pad-ops.c
# ggml/src/ggml-hexagon/htp/pool-ops.c
# ggml/src/ggml-hexagon/htp/pool-ops.h
# ggml/src/ggml-hexagon/htp/repeat-ops.c
# ggml/src/ggml-hexagon/htp/roll-ops.c
# ggml/src/ggml-hexagon/htp/rope-ops.c
# ggml/src/ggml-hexagon/htp/rope-ops.h
# ggml/src/ggml-hexagon/htp/set-rows-ops.c
# ggml/src/ggml-hexagon/htp/set-rows-ops.h
# ggml/src/ggml-hexagon/htp/softmax-ops.c
# ggml/src/ggml-hexagon/htp/softmax-ops.h
# ggml/src/ggml-hexagon/htp/solve-tri-ops.c
# ggml/src/ggml-hexagon/htp/ssm-conv.c
# ggml/src/ggml-hexagon/htp/ssm-conv.h
# ggml/src/ggml-hexagon/htp/sum-rows-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.h
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/conv2d.cpp
# ggml/src/ggml-sycl/convert.cpp
# ggml/src/ggml-sycl/esimd.hpp
# ggml/src/ggml-sycl/fattn-mkl.cpp
# ggml/src/ggml-sycl/fattn-sparse.cpp
# ggml/src/ggml-sycl/fattn.cpp
# ggml/src/ggml-sycl/fattn.hpp
# ggml/src/ggml-sycl/fusion.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/mmvq.cpp
# ggml/src/ggml-sycl/mmvq.hpp
# ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/get_rows.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_decls.tmpl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec_acc.tmpl
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-mtmd-impl.cpp
# tests/test-server-component.cpp
# tests/test-tokenizer-0.cpp
# tools/mtmd/CMakeLists.txt
# tools/server/README.md
2026-10-24 12:10:10 +08:00
Concedo
4224e52453
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .flake8
# .github/workflows/build-and-test-snapdragon.yml
# .github/workflows/build-webgpu.yml
# .github/workflows/release.yml
# CMakeLists.txt
# cmake/common.cmake
# docs/backend/snapdragon/README.md
# docs/backend/snapdragon/linux.md
# docs/backend/snapdragon/windows.md
# docs/backend/zDNN.md
# docs/build.md
# ggml/cmake/common.cmake
# ggml/src/ggml-hexagon/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp-repack.cpp
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/libggml-htp.inf
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/gemm_noshuffle_q4_0_q8_1_dp4a.cl
# ggml/src/ggml-opencl/kernels/gemm_noshuffle_q4_k_q8_1_dp4a.cl
# ggml/src/ggml-opencl/kernels/gemm_noshuffle_q5_k_q8_1_dp4a.cl
# ggml/src/ggml-opencl/kernels/gemm_noshuffle_q6_k_q8_1_dp4a.cl
# ggml/src/ggml-opencl/kernels/gemm_noshuffle_q8_0_q8_1_dp4a.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q1_0_f32.cl
# ggml/src/ggml-opencl/kernels/mul_mm_f16_f32_l4_lm.cl
# ggml/src/ggml-opencl/kernels/mul_mv_mxfp4_f32_flat.cl
# ggml/src/ggml-opencl/kernels/rope.cl
# ggml/src/ggml-sycl/CMakeLists.txt
# ggml/src/ggml-sycl/concat.cpp
# ggml/src/ggml-sycl/convert.cpp
# ggml/src/ggml-sycl/count-equal.cpp
# ggml/src/ggml-sycl/dequantize.hpp
# ggml/src/ggml-sycl/dpct/helper.hpp
# ggml/src/ggml-sycl/element_wise.cpp
# ggml/src/ggml-sycl/fattn-common.hpp
# ggml/src/ggml-sycl/fattn-sparse.cpp
# ggml/src/ggml-sycl/fattn-tile.cpp
# ggml/src/ggml-sycl/fattn.cpp
# ggml/src/ggml-sycl/fused-gemm.cpp
# ggml/src/ggml-sycl/fused-gemm.hpp
# ggml/src/ggml-sycl/getrows.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/gla.cpp
# ggml/src/ggml-sycl/mem.cpp
# ggml/src/ggml-sycl/mmq.cpp
# ggml/src/ggml-sycl/topk-moe.cpp
# ggml/src/ggml-sycl/upscale.cpp
# ggml/src/ggml-sycl/vecdotq.hpp
# ggml/src/ggml-zdnn/common.hpp
# ggml/src/ggml-zdnn/ggml-zdnn.cpp
# ggml/src/ggml-zdnn/mmq.cpp
# scripts/check-apiabi-compat.sh
# scripts/snapdragon/build.py
# scripts/snapdragon/ggml-hexagon-inspect.py
# scripts/snapdragon/run.py
# scripts/snapdragon/sdk.py
# skills/slice-model/slice_model.py
# src/llama-vocab.cpp
# tests/CMakeLists.txt
# tests/fusion/MTL.csv
# tests/test-backend-ops.cpp
# tests/test-jinja.cpp
# tests/test-llama-archs.cpp
# tests/test-mtmd-impl.cpp
# tools/CMakeLists.txt
# tools/cli/README.md
# tools/completion/README.md
# tools/mtmd/CMakeLists.txt
# tools/server/CMakeLists.txt
# tools/server/README.md
# ty.toml
2026-10-24 12:02:00 +08:00
Adrien Gallouët
21ec307146
common : allow concurrent downloads across processes ( #29312 )
...
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-23 23:52:56 +02:00
Xuan-Son Nguyen
7adf160c41
server: add --connect, allow p2p access to server outside LAN ( #28277 )
...
* init
* wip
* wire it up
* make it clear that it's experimental
* add props.connect_code
* nits
* fix windows build
* server: do not advertise a connect code without the tunnel
--connect-code on its own had no effect other than being served in
/props, so the Web UI showed a remote access code that no tunnel was
serving. Clear it when --connect is off, and say so once at startup.
The teardown comment claimed that terminate() closes the child stdout.
It does not: the log thread leaves fgets() only once every writer of
that pipe is gone, so a descendant of llama-connect would block the
join forever. Describe the real contract instead.
Also restore the --reasoning-preserve default in the generated docs.
* Update tools/connect/CMakeLists.txt
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
* server: dial llama-connect on the bound port and a reachable host
Pass ctx_http.port so --port 0 and the GCP override work, pick the
dial host from --host (wildcards map to loopback, IPv6 is bracketed)
and refuse --connect when only unix sockets are bound.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
2026-10-20 00:23:14 +02:00
Concedo
8826b9c532
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/cann.Dockerfile
# .devops/cpu.Dockerfile
# .devops/cuda.Dockerfile
# .devops/intel.Dockerfile
# .devops/musa.Dockerfile
# .devops/rocm.Dockerfile
# .devops/vulkan.Dockerfile
# .devops/zendnn.Dockerfile
# .github/workflows/build-sycl.yml
# .github/workflows/python-type-check.yml
# .github/workflows/release.yml
# .pi/gg/SYSTEM.md
# CMakeLists.txt
# docs/backend/snapdragon/developer.md
# docs/backend/zDNN.md
# examples/sycl/build.sh
# examples/sycl/win-build-sycl.bat
# ggml/CMakeLists.txt
# ggml/src/ggml-hexagon/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp-opnode.h
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/get-rows-ops.c
# ggml/src/ggml-hexagon/htp/get-rows-ops.h
# ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-dcache.c
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/htp-tensor.h
# ggml/src/ggml-hexagon/htp/hvx-mm-kernels-float.h
# ggml/src/ggml-hexagon/htp/hvx-mm-kernels-tiled.h
# ggml/src/ggml-hexagon/htp/hvx-quant.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-hexagon/htp/matmul-ops.h
# ggml/src/ggml-metal/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/clamp.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f16.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f32.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f32_f16.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f32_q4_0.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f32_q8_0.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_iq4_nl_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q1_0_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_0_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_0_f32_spec.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_1_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_k_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_k_f32_o4.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q5_0_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q5_1_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q5_k_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q6_k_f32.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q6_k_f32_o4.cl
# ggml/src/ggml-opencl/kernels/gemv_noshuffle_q8_0_f32.cl
# ggml/src/ggml-openvino/openvino/op/glu_swiglu.cpp
# ggml/src/ggml-sycl/CMakeLists.txt
# ggml/src/ggml-sycl/convert.cpp
# ggml/src/ggml-sycl/dequantize.hpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/mmvq.cpp
# ggml/src/ggml-sycl/quants.hpp
# ggml/src/ggml-sycl/vecdotq.hpp
# scripts/server-bench.py
# scripts/snapdragon/run.py
# scripts/sync-ggml.last
# src/llama-vocab.cpp
# tests/llama-context/cases-kv.cpp
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tests/test-jinja.cpp
# tests/test-llama-archs.cpp
# tests/test-mtmd-impl.cpp
# tests/test-thread-safety.cpp
# tests/test-unicode.cpp
# tools/cli/README.md
# tools/mtmd/CMakeLists.txt
# tools/server/README.md
2026-10-16 13:43:20 +08:00
Concedo
c1ecdb2a78
Merge commit ' 5cdadfc326' into concedo_experimental
...
# Conflicts:
# docs/backend/SYCL.md
# docs/backend/snapdragon/README.md
# docs/ops.md
# docs/ops/zDNN.csv
# docs/speculative.md
# examples/diffusion/diffusion-cli.cpp
# examples/sycl/build.sh
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-cpu/kleidiai/kernels.cpp
# ggml/src/ggml-cpu/kleidiai/kernels.h
# ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/htp_iface.idl
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/gemm_moe_q8_1_dp4a.cl
# ggml/src/ggml-opencl/kernels/moe_sort_by_expert.cl
# ggml/src/ggml-rpc/CMakeLists.txt
# ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_id.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_id_gather.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_id_vec.wgsl
# scripts/gen-unicode-data.py
# scripts/snapdragon/run.py
# scripts/sync_vendor.py
# skills/code-review/SKILL.md
# tests/CMakeLists.txt
# tests/test-arg-parser.cpp
# tests/test-backend-ops.cpp
# tests/test-chat-auto-parser.cpp
# tests/test-chat-template.cpp
# tests/test-chat.cpp
# tests/test-jinja.cpp
# tests/test-server-component.cpp
# tests/test-unicode.cpp
# tools/cli/README.md
# tools/server/README.md
2026-10-16 13:26:05 +08:00
Concedo
b776b7a136
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-apple.yml
# .github/workflows/build-cache.yml
# .github/workflows/build-openvino.yml
# .github/workflows/ci-self-hosted-openvino.yml
# .github/workflows/release-publish.yml
# .github/workflows/release.yml
# CODEOWNERS
# ci/run.sh
# docs/backend/zDNN.md
# examples/speculative/speculative.cpp
# ggml/src/CMakeLists.txt
# ggml/src/ggml-cuda/ggml-cuda.cu
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/binary-ops.c
# ggml/src/ggml-hexagon/htp/binary-ops.h
# ggml/src/ggml-hexagon/htp/concat-ops.c
# ggml/src/ggml-hexagon/htp/concat-ops.h
# ggml/src/ggml-hexagon/htp/cpy-ops.c
# ggml/src/ggml-hexagon/htp/cpy-ops.h
# ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/hvx-mm-kernels-tiled.h
# ggml/src/ggml-hexagon/htp/im2col-ops.c
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-hexagon/htp/matmul-ops.h
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/flash_attn_f32_f16.cl
# ggml/src/ggml-sycl/dmmv.cpp
# ggml/src/ggml-vulkan/ggml-vulkan-types.h
# ggml/src/ggml-zdnn/common.hpp
# ggml/src/ggml-zdnn/ggml-zdnn.cpp
# ggml/src/ggml-zdnn/mmf.cpp
# ggml/src/ggml-zdnn/utils.cpp
# ggml/src/ggml-zdnn/utils.hpp
# scripts/snapdragon/build.py
# skills/code-review/SKILL.md
# src/llama-context.cpp
# tests/.gitignore
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-llama-archs.cpp
# tests/testing.h
# tools/rpc/README.md
# tools/server/README.md
2026-10-16 13:13:44 +08:00
Concedo
9be8c136ae
Merge commit ' 23b0202a18' into concedo_experimental
...
# Conflicts:
# ggml/src/CMakeLists.txt
# ggml/src/ggml-cpu/CMakeLists.txt
# tests/test-batch-alloc.cpp
# tools/server/README.md
2026-10-16 00:15:23 +08:00
Pascal
2639a3b4e0
server: save only restorable slot states to the prompt cache ( #30498 )
...
* server: save only restorable slot states to the prompt cache
The idle-slot save loop stores every idle slot whenever a task starts,
while only completion tasks restore from the prompt cache. Rerank,
embedding, decision and infill states fill the cache up to its limit
without ever being read back, and slots already cached are offered
again at every task start.
Each slot now tracks with cache_pending whether its state is restorable
and missing from the cache: a completion release or an explicit slot
restore sets it, prompt_save consumes it, prompt_clear resets it. Both
save paths share this rule, and the LRU path restores only for
completion tasks.
* server: test that decision slots skip the prompt cache
Decision tasks never mark their slot as cacheable, so starting new
tasks must not save their idle slots to the RAM prompt cache, with or
without a unified KV cache.
* server: restore any reusing task from the prompt cache
Embedding and rerank tasks never reuse their state, so their slots are
cleared on release like child slots and the cache only holds states
that a later task can restore. Infill and decision tasks now restore
from the cache too, which replaces the cache_pending flag.
Decision models disable the prompt cache unless --cache-ram is given,
since their states seldom repeat.
2026-10-15 06:30:17 +02:00
Georgi Gerganov
3fb47677ea
common : add --spec-draft-n-ubatch ( #30459 )
...
* common : add --spec-draft-n-ubatch
Allow configuring the physical batch size of the draft context. The
drafters feed it at most the drafted tokens per sequence and chunk larger
batches by its own n_ubatch, so it does not need the ubatch of the target.
A smaller value shrinks the draft compute buffer and leaves more VRAM for
the target KV cache (the draft is fitted together with the target).
The default 0 keeps the current behavior of following --n-ubatch.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : remove comment
[no ci]
2026-10-14 13:58:05 +03:00
Daniel Bevenius
77757a3467
examples : fix draft sampling and rejection handling ( #30095 )
...
* examples : fix draft sampling and rejection handling
This commit fixes two issues in the speculative example which are
described in detail in the linked issue.
Resolves: https://github.com/ggml-org/llama.cpp/issues/29975
* examples : add kv-unified requirement for parallel > 1
2026-10-12 12:02:24 +02:00
Mikolaj Kucharski
aa94f20861
args: add LLAMA_ARG_SLOT_SAVE_PATH env for --slot-save-path ( #30272 )
...
Allow configuring --slot-save-path via LLAMA_ARG_SLOT_SAVE_PATH so
it can be set from an EnvironmentFile in a systemd unit file.
2026-10-10 15:34:40 +02:00
Concedo
b3284ea444
Merge commit ' b42b7e6d30' into concedo_experimental
...
# Conflicts:
# .devops/cann.Dockerfile
# .devops/cpu.Dockerfile
# .devops/cuda.Dockerfile
# .devops/intel.Dockerfile
# .devops/musa.Dockerfile
# .devops/openvino.Dockerfile
# .devops/rocm.Dockerfile
# .devops/s390x.Dockerfile
# .devops/vulkan.Dockerfile
# .devops/zendnn.Dockerfile
# docs/backend/ZenDNN.md
# docs/function-calling.md
# examples/json_schema_pydantic_example.py
# examples/model-conversion/scripts/causal/modelcard.template
# examples/model-conversion/scripts/embedding/modelcard.template
# examples/model-conversion/scripts/utils/curl-embedding-server.sh
# examples/pydantic_models_to_grammar_examples.py
# ggml/src/ggml-hexagon/htp/im2col-ops.c
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/mmvq.cpp
# ggml/src/ggml-sycl/mmvq.hpp
# ggml/src/ggml-sycl/vecdotq.hpp
# scripts/compare-logprobs.py
# scripts/hip/gcn-cdna-vgpr-check.py
# scripts/server-bench.py
# scripts/server-test-function-call.py
# scripts/server-test-model.py
# scripts/server-test-parallel-tc.py
# scripts/server-test-structured.py
# tests/test-backend-ops.cpp
# tests/test-llama-archs.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/server/README.md
# tools/server/bench/README.md
2026-10-10 13:37:53 +08:00
Concedo
0890a5ee77
Merge commit ' 033df86b69' into concedo_experimental
...
# Conflicts:
# docs/backend/SYCL.md
# docs/development/parsing.md
# examples/speculative-simple/speculative-simple.cpp
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/get-rows-ops.c
# ggml/src/ggml-hexagon/htp/get-rows-ops.h
# ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/hvx-erf.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.h
# ggml/src/ggml-sycl/CMakeLists.txt
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/element_wise.cpp
# ggml/src/ggml-sycl/element_wise.hpp
# ggml/src/ggml-sycl/fusion.cpp
# ggml/src/ggml-sycl/fwht.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/im2col.hpp
# ggml/src/ggml-sycl/pad.hpp
# ggml/src/ggml-sycl/rope.hpp
# ggml/src/ggml-sycl/upscale.hpp
# scripts/snapdragon/run.py
# scripts/sync_vendor.py
# src/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-backend-sampler.cpp
# tests/test-chat.cpp
# tests/test-llama-archs.cpp
# tools/imatrix/imatrix.cpp
# tools/server/README.md
2026-10-10 13:19:02 +08:00
Xuan-Son Nguyen
b42b7e6d30
server: use port 9931 by default ( #30159 )
...
* server: use port 9931 by default
* revert unrelated changes
2026-10-09 10:40:03 +02:00
Concedo
14cf69fa7a
Merge commit ' 63bef2728d' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-apple.yml
# .github/workflows/docker.yml
# .github/workflows/make-release.yml
# .github/workflows/models-check.yml
# .github/workflows/winget.yml
# CMakeLists.txt
# docs/backend/OPENVINO.md
# docs/build-riscv64-spacemit.md
# ggml/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/dma-queue.c
# ggml/src/ggml-hexagon/htp/dma-queue.h
# ggml/src/ggml-hexagon/htp/flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/flash-attn-ops.h
# ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h
# ggml/src/ggml-hexagon/htp/hmx-utils.h
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-hexagon/htp/matmul-ops.h
# ggml/src/ggml-hexagon/htp/ssm-conv.c
# ggml/src/ggml-hexagon/htp/ssm-conv.h
# ggml/src/ggml-openvino/ggml-decoder.cpp
# ggml/src/ggml-openvino/ggml-decoder.h
# ggml/src/ggml-openvino/ggml-openvino-extra.cpp
# ggml/src/ggml-openvino/ggml-openvino.cpp
# ggml/src/ggml-openvino/openvino/op/fill.cpp
# ggml/src/ggml-openvino/openvino/op_table.cpp
# ggml/src/ggml-openvino/openvino/pass/fuse_moe_compressed.cpp
# ggml/src/ggml-openvino/openvino/translate_session.cpp
# ggml/src/ggml-openvino/utils.cpp
# ggml/src/ggml-openvino/utils.h
# scripts/make-release-checks.sh
# scripts/make-release-summary.txt
# scripts/snapdragon/run.py
# scripts/sync-ggml.last
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-recurrent-state-rollback.cpp
# tests/test-save-load-state.cpp
# tests/test-state-restore-fragmented.cpp
# tools/server/README.md
2026-10-08 23:43:45 +08:00
Aman Gupta
c811cb8f0a
llama: support MoE cache over multiple GPUs ( #30112 )
2026-10-08 15:50:49 +05:30
Aman Gupta
d6cf9acb25
llama : add a GPU cache for MoE experts kept in host memory ( #29887 )
...
* llama : add a GPU cache for MoE experts kept in host memory
Assisted-by: Claude
* use llama_moe_cache_ptr
2026-10-07 21:07:45 +03:00
Concedo
acef3c226f
Merge commit ' ebe18bee5a' into concedo_experimental
...
# Conflicts:
# .devops/openvino.Dockerfile
# .github/workflows/ai-issues.yml
# .github/workflows/build-3rd-party.yml
# .github/workflows/build-and-test-snapdragon.yml
# .github/workflows/build-android.yml
# .github/workflows/build-apple.yml
# .github/workflows/build-cache.yml
# .github/workflows/build-cann.yml
# .github/workflows/build-cmake-pkg.yml
# .github/workflows/build-cpu.yml
# .github/workflows/build-cross.yml
# .github/workflows/build-cuda-ubuntu.yml
# .github/workflows/build-cuda-windows.yml
# .github/workflows/build-ibm.yml
# .github/workflows/build-msys.yml
# .github/workflows/build-opencl.yml
# .github/workflows/build-openvino.yml
# .github/workflows/build-riscv.yml
# .github/workflows/build-sanitize.yml
# .github/workflows/build-sycl.yml
# .github/workflows/build-virtgpu.yml
# .github/workflows/build-vulkan.yml
# .github/workflows/build-wasm.yml
# .github/workflows/build-webgpu.yml
# .github/workflows/check-vendor.yml
# .github/workflows/ci-self-hosted-cpu.yml
# .github/workflows/ci-self-hosted-cuda.yml
# .github/workflows/ci-self-hosted-kleidiai.yml
# .github/workflows/ci-self-hosted-metal.yml
# .github/workflows/ci-self-hosted-openvino.yml
# .github/workflows/ci-self-hosted-vulkan.yml
# .github/workflows/ci-self-hosted-webgpu.yml
# .github/workflows/close-issue.yml
# .github/workflows/code-style.yml
# .github/workflows/docker.yml
# .github/workflows/editorconfig.yml
# .github/workflows/gguf-publish.yml
# .github/workflows/hip-quality-check.yml
# .github/workflows/labeler.yml
# .github/workflows/make-release.yml
# .github/workflows/models-check.yml
# .github/workflows/pr-draft-label.yml
# .github/workflows/pre-tokenizer-hashes.yml
# .github/workflows/python-check-requirements.yml
# .github/workflows/python-lint.yml
# .github/workflows/python-type-check.yml
# .github/workflows/release.yml
# .github/workflows/server-sanitize.yml
# .github/workflows/server-self-hosted.yml
# .github/workflows/server.yml
# .github/workflows/ui-build-self-hosted.yml
# .github/workflows/ui-build.yml
# .github/workflows/ui-publish.yml
# .github/workflows/ui-self-hosted.yml
# .github/workflows/ui.yml
# .github/workflows/update-ops-docs.yml
# .github/workflows/winget.yml
# AGENTS.md
# docs/backend/OPENVINO.md
# ggml/src/ggml-metal/CMakeLists.txt
# ggml/src/ggml-openvino/CMakeLists.txt
# ggml/src/ggml-openvino/ggml-decoder.cpp
# ggml/src/ggml-openvino/ggml-decoder.h
# ggml/src/ggml-openvino/ggml-openvino-extra.cpp
# ggml/src/ggml-openvino/ggml-openvino-extra.h
# ggml/src/ggml-openvino/ggml-openvino.cpp
# ggml/src/ggml-openvino/model-cache.cpp
# ggml/src/ggml-openvino/model-cache.h
# ggml/src/ggml-openvino/openvino/node_context.h
# ggml/src/ggml-openvino/openvino/op/add.cpp
# ggml/src/ggml-openvino/openvino/op/argsort.cpp
# ggml/src/ggml-openvino/openvino/op/cpy.cpp
# ggml/src/ggml-openvino/openvino/op/flash_attn_ext.cpp
# ggml/src/ggml-openvino/openvino/op/gated_delta_net.cpp
# ggml/src/ggml-openvino/openvino/op/get_rows.cpp
# ggml/src/ggml-openvino/openvino/op/im2col.cpp
# ggml/src/ggml-openvino/openvino/op/l2_norm.cpp
# ggml/src/ggml-openvino/openvino/op/mul_mat_id.cpp
# ggml/src/ggml-openvino/openvino/op/reshape.cpp
# ggml/src/ggml-openvino/openvino/op/rms_norm.cpp
# ggml/src/ggml-openvino/openvino/op/rope.cpp
# ggml/src/ggml-openvino/openvino/op/scale.cpp
# ggml/src/ggml-openvino/openvino/op/view.cpp
# ggml/src/ggml-openvino/openvino/op_table.cpp
# ggml/src/ggml-openvino/openvino/op_table.h
# ggml/src/ggml-openvino/openvino/pass/fuse_moe_compressed.cpp
# ggml/src/ggml-openvino/openvino/pass/fuse_moe_compressed.h
# ggml/src/ggml-openvino/openvino/pass/fuse_to_conv.cpp
# ggml/src/ggml-openvino/openvino/translate_session.cpp
# ggml/src/ggml-openvino/openvino/utils.cpp
# ggml/src/ggml-openvino/openvino/utils.h
# ggml/src/ggml-openvino/utils.cpp
# ggml/src/ggml-openvino/utils.h
# ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec_q_acc.tmpl
# ggml/src/ggml-webgpu/wgsl-shaders/quantize_q8.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/set_rows.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/set_rows_quant.wgsl
# scripts/sync_vendor.py
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-batch-alloc.cpp
# tests/test-chat.cpp
# tests/test-llama-archs.cpp
# tools/imatrix/README.md
# tools/imatrix/imatrix.cpp
# tools/server/README.md
2026-10-07 23:18:17 +08:00
Adrien Gallouët
4d60b4d087
common, server : report model input/output modalities in GET /models ( #29987 )
...
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-05 17:08:29 +02:00
Ed Addario
0504396140
imatrix: calculate activation-based statistics for new format (GGUF) imatrices ( #14891 )
...
* Use activations to calculate the stats
* Determine calculation mode
* Compute entropy for activations
* Compute cosine similarity based on activations
* Compute l2 norm
* Add compute_layer_statistics() function
* Update aggregated statistic report layout
* Fix printing l2 norm when calc_mode = 1
* Refactor variable name
* Compute aggregated (per layer) l2 norm
* Update aggregated sum of squared activations per layer
* Make ZD Score two-tailed
* Update report layout
* Reverse conditional logic to match convention
* Rename report heading
* Add --activation-statistics parameter
* Add Euclidean–Cosine Score (ECS)
* Add --activation-statistics logic to avoid doubling the imatrix size by default
* Update stats output sort based on imatrix type
* Process external NextN draft files (-md / --model-draft)
* Refactor to use new llama_batch_ext
Co-authored-by: compilade <git@compilade.net>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-10-04 11:24:53 +02:00
Concedo
4d27749c93
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-cuda-ubuntu.yml
# .github/workflows/build-cuda-windows.yml
# .github/workflows/build-ibm.yml
# .github/workflows/ci-self-hosted-cuda.yml
# .github/workflows/release.yml
# .github/workflows/server-self-hosted.yml
# AGENTS.md
# README.md
# ci/run.sh
# docs/autoparser.md
# docs/backend/SYCL.md
# docs/build.md
# docs/development/HOWTO-add-model.md
# docs/function-calling.md
# docs/ops.md
# docs/ops/CPU.csv
# ggml/CMakeLists.txt
# ggml/src/ggml-blas/CMakeLists.txt
# ggml/src/ggml-cann/ggml-cann.cpp
# ggml/src/ggml-cpu/amx/amx.cpp
# ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
# ggml/src/ggml-cuda/CMakeLists.txt
# ggml/src/ggml-et/ggml-et.cpp
# ggml/src/ggml-hexagon/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/allreduce-ops.c
# ggml/src/ggml-hexagon/htp/allreduce-ops.h
# ggml/src/ggml-hexagon/htp/concat-ops.c
# ggml/src/ggml-hexagon/htp/cpy-ops.c
# ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/hvx-mm-kernels-tiled.h
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-hexagon/htp/matmul-ops.h
# ggml/src/ggml-hexagon/htp/work-queue.c
# ggml/src/ggml-metal/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-openvino/ggml-openvino.cpp
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/dmmv.cpp
# ggml/src/ggml-sycl/fattn-common.hpp
# ggml/src/ggml-sycl/fattn-onednn.cpp
# ggml/src/ggml-sycl/fattn-vec.hpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/mmvq.cpp
# ggml/src/ggml-sycl/vecdotq.hpp
# ggml/src/ggml-virtgpu/ggml-backend-buffer-type.cpp
# ggml/src/ggml-vulkan/ggml-vulkan.cpp
# ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/common_decls.tmpl
# ggml/src/ggml-webgpu/wgsl-shaders/get_rows.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_decls.tmpl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec_acc.tmpl
# ggml/src/ggml-webgpu/wgsl-shaders/ssm_scan.wgsl
# ggml/src/ggml-zdnn/ggml-zdnn.cpp
# pyproject.toml
# scripts/snapdragon/run.py
# skills/code-review/SKILL.md
# tests/fusion/MTL.csv
# tests/test-alloc.cpp
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tests/test-llama-archs.cpp
# tests/test-model-resolution.cpp
# tests/test-quantize-fns.cpp
# tests/test-recurrent-state-rollback.cpp
# tools/llama-bench/README.md
# tools/llama-bench/llama-bench.cpp
# tools/server/CMakeLists.txt
# tools/server/README.md
2026-10-03 17:59:14 +08:00
Concedo
2374751a07
Merge commit ' 19e28a2770' into concedo_experimental
...
# Conflicts:
# .devops/intel.Dockerfile
# .devops/musa.Dockerfile
# .github/workflows/build-cuda-ubuntu.yml
# .github/workflows/build-ibm.yml
# .github/workflows/build-sycl.yml
# .github/workflows/release.yml
# ci/README-MUSA.md
# ci/run.sh
# docs/backend/SYCL.md
# examples/simple-chat/simple-chat.cpp
# examples/simple/simple.cpp
# examples/speculative-simple/speculative-simple.cpp
# ggml/include/ggml-zdnn.h
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/act-ops.c
# ggml/src/ggml-hexagon/htp/concat-ops.c
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/hvx-arith.h
# ggml/src/ggml-hexagon/htp/hvx-base.h
# ggml/src/ggml-hexagon/htp/hvx-sigmoid.h
# ggml/src/ggml-hexagon/htp/hvx-utils.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.h
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-openvino/ggml-openvino.cpp
# ggml/src/ggml-zdnn/ggml-zdnn.cpp
# scripts/hip/gcn-cdna-vgpr-check.py
# scripts/snapdragon/ggml-hexagon-trace.py
# src/llama-context.cpp
# tests/gguf-model-data.cpp
# tests/test-chat.cpp
# tests/test-fusion.cpp
# tests/test-gguf.cpp
# tools/cli/cli.cpp
# tools/server/README.md
# vendor/stb/CMakeLists.txt
2026-10-03 00:31:12 +08:00
Pranesh Gonegandla
1fb7ef3e33
spec : add probabilistic sampling for simple draft and MTP ( #27694 )
...
* Make the drafter probabilistic and the target verify by rejection sampling
* Drop stale spec_draft_q before drafting
* Fallback to argmax sampling for grammar-constrained requests and adding flag for enabling probabilistic draft sampling. Default flag value is greedy.
* Support grammar-constrained requests in rejection sampling
* Fix - renormalize distribution after masking
* copy rng on sampler copy and re-accept drafted tokens on replay
* Fix draft sampler sharing the target's rng stream
* Simplify the rejection sampler's inputs and move replay to the server
* Truncate the draft candidates along with the draft
---------
Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
2026-10-02 17:52:54 +03:00
pr3pony
feb9a3d6de
args: fix cli download mmproj arg ( #28977 )
...
* tests: add tests for cli download arg parsing
* args: fix cli download mmproj arg
2026-09-30 20:45:39 +02:00
Concedo
3cee526a68
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/musa.Dockerfile
# .github/workflows/build-apple.yml
# .github/workflows/build-cpu.yml
# .github/workflows/build-vulkan.yml
# .github/workflows/python-type-check.yml
# CODEOWNERS
# docs/backend/snapdragon/CMakeUserPresets.json
# docs/build.md
# docs/docker.md
# docs/ops.md
# docs/ops/Hexagon.csv
# ggml/cmake/common.cmake
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-cpu/ggml-cpu.c
# ggml/src/ggml-cuda/CMakeLists.txt
# ggml/src/ggml-hexagon/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp-opnode.h
# ggml/src/ggml-hexagon/htp/argsort-ops.c
# ggml/src/ggml-hexagon/htp/binary-ops.c
# ggml/src/ggml-hexagon/htp/binary-ops.h
# ggml/src/ggml-hexagon/htp/cpy-ops.c
# ggml/src/ggml-hexagon/htp/dma-queue.h
# ggml/src/ggml-hexagon/htp/get-rows-ops.c
# ggml/src/ggml-hexagon/htp/get-rows-ops.h
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/hvx-arith.h
# ggml/src/ggml-hexagon/htp/hvx-base.h
# ggml/src/ggml-hexagon/htp/hvx-reduce.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/sum-rows-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.h
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-sycl/fwht.cpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# scripts/hip/gcn-cdna-vgpr-check.py
# scripts/snapdragon/ggml-hexagon-inspect.py
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-jinja.cpp
# tests/test-recurrent-state-rollback.cpp
# tests/test-save-load-state.cpp
# tests/test-state-restore-fragmented.cpp
# tools/llama-bench/README.md
# tools/llama-bench/llama-bench.cpp
# tools/mtmd/CMakeLists.txt
2026-09-29 23:24:34 +08:00
Adrien Gallouët
00af63567a
common : use fs::path for config dir ( #29649 )
...
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-09-29 17:22:49 +02:00
Adrien Gallouët
76a5bc86d1
common : use fs::path for cache dirs ( #29595 )
...
- Avoid useless string conversions on Windows.
- No need for BSD or emscripten special cases.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-09-29 07:24:03 +02:00
Georgi Gerganov
a97cce86a8
common : avoid side effects around params parsing ( #29537 )
...
- register --rpc unconditionally and call llama_supports_rpc() only from its handler
- print server "initialization ..." log after args are parsed
Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
2026-09-27 20:18:56 +03:00
Concedo
084d797f2d
Merge commit ' 4ceb171910' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-sycl.yml
# .github/workflows/docker.yml
# .github/workflows/release.yml
# cmake/llama-config.cmake.in
# docs/backend/snapdragon/README.md
# docs/backend/snapdragon/developer.md
# examples/simple-cmake-pkg/CMakeLists.txt
# ggml/include/ggml-sycl.h
# ggml/src/ggml-cpu/repack.cpp
# ggml/src/ggml-cpu/repack.h
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp-opnode.h
# ggml/src/ggml-hexagon/htp/act-ops.c
# ggml/src/ggml-hexagon/htp/allreduce-ops.c
# ggml/src/ggml-hexagon/htp/argsort-ops.c
# ggml/src/ggml-hexagon/htp/binary-ops.c
# ggml/src/ggml-hexagon/htp/concat-ops.c
# ggml/src/ggml-hexagon/htp/cpy-ops.c
# ggml/src/ggml-hexagon/htp/cumsum-ops.c
# ggml/src/ggml-hexagon/htp/diag-ops.c
# ggml/src/ggml-hexagon/htp/dma-queue.c
# ggml/src/ggml-hexagon/htp/dma-queue.h
# ggml/src/ggml-hexagon/htp/fill-ops.c
# ggml/src/ggml-hexagon/htp/flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/flash-attn-ops.h
# ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c
# ggml/src/ggml-hexagon/htp/get-rows-ops.c
# ggml/src/ggml-hexagon/htp/hmx-fa-kernels.h
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/htp-tensor.c
# ggml/src/ggml-hexagon/htp/htp-tensor.h
# ggml/src/ggml-hexagon/htp/htp_iface.idl
# ggml/src/ggml-hexagon/htp/hvx-exp.h
# ggml/src/ggml-hexagon/htp/hvx-mm-kernels-tiled.h
# ggml/src/ggml-hexagon/htp/im2col-ops.c
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-hexagon/htp/matmul-ops.h
# ggml/src/ggml-hexagon/htp/pad-ops.c
# ggml/src/ggml-hexagon/htp/repeat-ops.c
# ggml/src/ggml-hexagon/htp/roll-ops.c
# ggml/src/ggml-hexagon/htp/rope-ops.c
# ggml/src/ggml-hexagon/htp/rope-ops.h
# ggml/src/ggml-hexagon/htp/set-rows-ops.c
# ggml/src/ggml-hexagon/htp/softmax-ops.c
# ggml/src/ggml-hexagon/htp/solve-tri-ops.c
# ggml/src/ggml-hexagon/htp/ssm-conv.c
# ggml/src/ggml-hexagon/htp/sum-rows-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-sycl/dsv4-hc.cpp
# ggml/src/ggml-sycl/fattn-mkl.cpp
# ggml/src/ggml-sycl/fattn-tile.hpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# scripts/snapdragon/ggml-hexagon-profile.py
# scripts/snapdragon/ggml-hexagon-trace.py
# scripts/snapdragon/run.py
# scripts/sync_vendor.py
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tests/test-llama-archs.cpp
# tests/test-recurrent-state-rollback.cpp
# tests/test-save-load-state.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/server/README.md
2026-09-24 16:37:57 +08:00
Emanuil Rusev
217f81c266
server: Add support for binding to multiple addresses ( #28690 )
...
* Add support for binding llama-server to multiple addresses
Assisted-by: Codex
* remove redundant thread handler
* make it clear about overlapping addr
* reject --port 0 with multiple tcp addr
* improve arg handler
* nits
* fix test
* nits 2
* nits
* nits 2
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-09-22 15:16:40 +02:00
Mikolaj Kucharski
e0dff58475
args: add env vars for temperature, top-p, min-p and penalties ( #27380 )
...
Allow configuring --temp, --top-p, --min-p, --repeat-penalty,
--presence-penalty and --frequency-penalty via LLAMA_ARG_* so
llama-server can be fully controlled from an EnvironmentFile
(e.g. systemd on Debian).
Use `llama-gen-docs` to regenerate the readme files.
2026-09-21 12:47:38 +02:00
Concedo
c43b36bcb5
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-ibm.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/code-style.yml
# .github/workflows/editorconfig.yml
# .github/workflows/fusion.yml
# .github/workflows/release.yml
# .github/workflows/server-sanitize.yml
# .pi/gg/SYSTEM.md
# CMakeLists.txt
# ci/run.sh
# common/CMakeLists.txt
# docs/backend/SYCL.md
# ggml/CMakeLists.txt
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-sycl/CMakeLists.txt
# ggml/src/ggml-sycl/backend.hpp
# ggml/src/ggml-sycl/base.hpp
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/gemm.hpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/mem.cpp
# scripts/sync-ggml.last
# src/CMakeLists.txt
# tests/CMakeLists.txt
# tests/snapshots/nvidia-nemotron-3-nano-30b-a3b.schema
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tests/test-llama-archs.cpp
# tests/test-quant-type-selection.cpp
# tools/mtmd/CMakeLists.txt
# tools/server/CMakeLists.txt
2026-09-14 22:44:04 +08:00
Concedo
a4f7e70b9f
Merge commit ' acecd56032' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-cpu.yml
# common/CMakeLists.txt
# common/chat-auto-parser-generator.cpp
# docs/development/parsing.md
# grammars/README.md
# json_to_gbnf.py
# tests/CMakeLists.txt
# tests/test-chat-peg-parser.cpp
# tests/test-chat.cpp
# tests/test-grammar-integration.cpp
# tests/test-json-schema-to-grammar.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/server/README.md
2026-09-14 22:35:27 +08:00
Concedo
483a456fd0
Merge commit ' 52d4268656' into concedo_experimental
...
# Conflicts:
# .github/actions/windows-setup-cuda/action.yml
# .github/workflows/build-riscv.yml
# .github/workflows/build-sanitize.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/release.yml
# .github/workflows/server-sanitize.yml
# docs/build-s390x.md
# docs/build.md
# ggml/CMakeLists.txt
# ggml/cmake/common.cmake
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-cpu/repack.cpp
# ggml/src/ggml-cuda/CMakeLists.txt
# ggml/src/ggml-cuda/fattn.cu
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/hvx-sin-cos.h
# ggml/src/ggml-hexagon/htp/rope-ops.c
# ggml/src/ggml-hip/CMakeLists.txt
# ggml/src/ggml-musa/CMakeLists.txt
# ggml/src/ggml-vulkan/ggml-vulkan.cpp
# pyproject.toml
# requirements/requirements-convert_legacy_llama.txt
# requirements/requirements-gguf_editor_gui.txt
# requirements/requirements-server-bench.txt
# requirements/requirements-tool_bench.txt
# scripts/make-release-checks.sh
# tests/test-backend-ops.cpp
# tests/test-jinja.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/llama-bench/llama-bench.cpp
# tools/server/README.md
2026-09-13 23:12:41 +08:00
Xuan-Son Nguyen
8e330954ad
common: add LOG_JSON macro to log structured data ( #28586 )
...
* add LOG_JSON macro
* fit: add demo LOG_JSON
2026-09-13 01:36:34 +02:00
Aldehir Rojas
acecd56032
common : implement common_schema internal representation for JSON schemas ( #28736 )
...
* common : implement common_schema types
* common : implement a json schema optimizer
* common : reduce optimizations
* common : refactor json-schema-to-grammar to use common_schema
* common : use common_trie
* common/schema : implement type/kind resolution
* cont : cleanup
* cont : remove common_chat_tool_parameters
* cont : simplify schema resolution
* cont : pass common_schema through the json-schema-to-grammar builder
* cont : cleanup
* cont : move enums under common_schema and add type enum
* cont : reduce test cases
* cont : clean up
* cont : clean up
* refactor : rename common_schema_parse to common_schema_from_json
* tests : fix gcc dangling-reference warning in test-json-schema
* tests : take the schema label as const char * to satisfy gcc dangling-reference
* refactor : rename common_schema_builder parse_* methods to build_*
* cont : fix may_be_string
* cont : properly handle empty tool parameters
* cont : add tests for empty $ref
* cont : remove dead code
* cont : update docs
* cont : make "{}" mean any object for json_object as well
* cont : restore (min|max)Length to imply string type
* cont : rename common_schema to common_chat_schema
2026-09-12 16:14:50 -05:00
Aaron Teo
14a9d09f75
args: officially deprecate --mmap|mlock|dio ( #28334 )
...
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
2026-09-09 18:36:27 +08:00
Concedo
ad1eee0c5f
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/server-self-hosted.yml
# .github/workflows/ui-build-self-hosted.yml
# .github/workflows/ui-build.yml
# .github/workflows/ui.yml
# ggml/src/ggml-sycl/mmvq.cpp
# ggml/src/ggml-sycl/vecdotq.hpp
# tests/test-backend-ops.cpp
# tests/test-sampling.cpp
# tools/cli/README.md
# tools/server/README.md
2026-09-09 11:47:23 +08:00
Aman Gupta
415e909d84
spec: single device drafter should create meta backend wrapper ( #28390 )
2026-09-08 20:44:33 +08:00
Concedo
2d357d8359
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/nix/package.nix
# .github/ISSUE_TEMPLATE/config.yml
# .github/workflows/make-release.yml
# docs/autoparser.md
# flake.nix
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-openvino/ggml-openvino.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/norm.cpp
# ggml/src/ggml-sycl/norm.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# models/templates/README.md
# scripts/make-release-checks.sh
# scripts/ui-assets.cmake
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tests/test-llama-archs.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/server/CMakeLists.txt
# tools/server/README.md
2026-09-08 11:59:02 +08:00
Xuan-Son Nguyen
7620399f58
common: add --log-jsonl ( #28437 )
...
* common: add --log-jsonl
* rename unknown to none
2026-09-06 08:21:22 +02:00
Concedo
a3e6006530
Merge commit ' d646c9d155' into concedo_experimental
...
# Conflicts:
# .github/actions/windows-setup-rocm/action.yml
# .github/workflows/build-apple.yml
# .github/workflows/release.yml
# .github/workflows/server-self-hosted.yml
# examples/training/README.md
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/hvx-arith.h
# ggml/src/ggml-hexagon/htp/hvx-log.h
# ggml/src/ggml-hexagon/htp/hvx-norm.h
# ggml/src/ggml-hexagon/htp/hvx-scale.h
# ggml/src/ggml-hexagon/htp/hvx-sqrt.h
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.h
# ggml/src/ggml-sycl/mmvq.cpp
# ggml/src/ggml-sycl/vecdotq.hpp
# tests/test-backend-ops.cpp
# tests/test-mtmd-c-api.c
# tests/test-mtmd-impl.cpp
2026-09-04 16:05:40 +08:00
Georgi Gerganov
e750b887a8
common, server : enable preserve_reasoning kwarg by default, log its effective state ( #28174 )
...
* common, server : enable preserve_reasoning kwarg by default, log its effective state
If the preserve_reasoning chat template kwarg is not specified explicitly
via --reasoning-preserve / --no-reasoning-preserve, it is enabled by
default after argument processing. The server logs the effective state of
the kwarg, warns that it is enabled by default when the template supports
it, and only warns "has no effect" when it was enabled explicitly on a
template that does not support it. Setting the kwarg via
--chat-template-kwargs is deprecated.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : update comment
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
---------
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
2026-09-02 19:19:54 +03:00
