vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-11-01 03:11:22 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
15950 commits 3 branches 259 tags 871 MiB
Commit graph

364 commits

Author SHA1 Message Date
Concedo
bd39978ced Merge commit '171e8846b4' into concedo_experimental 
# Conflicts:
#	.github/workflows/build-cuda-ubuntu.yml
#	.github/workflows/server-sanitize.yml
#	.pi/gg/SYSTEM.md
#	ci/README-MUSA.md
#	ci/run.sh
#	docs/backend/SYCL.md
#	docs/build.md
#	ggml/CMakeLists.txt
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/concat-ops.c
#	ggml/src/ggml-hexagon/htp/cpy-ops.c
#	ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/hvx-mm-kernels-tiled.h
#	ggml/src/ggml-hexagon/htp/matmul-ops.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.h
#	ggml/src/ggml-metal/CMakeLists.txt
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-sycl/fattn.cpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	scripts/sync-ggml.last
#	scripts/sync_vendor.py
#	tests/test-backend-ops.cpp
#	tests/test-batch-alloc.cpp
#	tests/test-mtmd-impl.cpp
#	tests/test-rpc-multi-server.cpp
 2026-09-29 22:31:17 +08:00
Xuan-Son Nguyen
fc343a84bb
llama: add llama_batch_ext (#24669) 
* (wip) add llama_batch_ext

* wip

* updated design

* updated impl

* change signature

* unused var

* demo common_prompt_batch_decode

* fix pos

* tmp disable test-batch-alloc

* fix compat

* nits: add const

* no more pos_max

* add comment about llama_batch_ext_set_embd_state

* handle n_embd_out properly

* rename api --> embd_token

* llama_embd

* stub llama_batch_ext_set_embd_state

* support both token + embd + state in batch

* llama_batch_ext_add_embd

* upstream some changes

* nits

* fix test-batch-alloc

* add test for compat
 2026-09-24 16:25:07 +02:00
Concedo
8c5df0a015 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/hmx-fa-kernels.h
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/im2col-ops.c
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	tests/test-backend-ops.cpp
#	tests/test-llama-archs.cpp
 2026-09-19 17:00:05 +08:00
Georgi Gerganov
efa28e950e
test-llama-archs : generate dummy test vocab (#29084) 
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
 2026-09-19 11:27:46 +03:00
Concedo
dbf4704e7e Merge commit 'f172be756a' into concedo_experimental 
# Conflicts:
#	.github/workflows/make-release.yml
#	ggml/src/ggml-vulkan/CMakeLists.txt
#	ggml/src/ggml-vulkan/ggml-vulkan.cpp
#	scripts/check-apiabi-compat.sh
#	scripts/make-release-checks.sh
#	tests/test-gguf.cpp
 2026-09-19 10:48:29 +08:00
Yuri Khrustalev
81aeaeb74b
gguf : align the data section relative to the GGUF start, not the file (#28993) 
* gguf : align the data section relative to the GGUF start, not the file

gguf_init_from_file_ptr reads a GGUF from the current file position, but padded
the data section from file offset 0, so a GGUF embedded at an offset that is not
a multiple of the alignment loaded without error and returned wrong tensor data.

Also adds llama_adapter_lora_init_from_file_ptr, and disables mmap with a warning
when an embedded data section is not aligned, instead of asserting in ggml.

Assisted-by: Claude Opus 5

* llama : load lora from path through the FILE* variant

The test now checks that mmap is disabled only for an unaligned offset.

Assisted-by: Claude Fable 5.1

* Update ggml/src/gguf.cpp

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Update include/llama.h

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* llama : error on unaligned mmap of an embedded GGUF, drop test-load-file-ptr

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
 2026-09-17 09:19:44 +02:00
Concedo
483a456fd0 Merge commit '52d4268656' into concedo_experimental 
# Conflicts:
#	.github/actions/windows-setup-cuda/action.yml
#	.github/workflows/build-riscv.yml
#	.github/workflows/build-sanitize.yml
#	.github/workflows/build-self-hosted.yml
#	.github/workflows/release.yml
#	.github/workflows/server-sanitize.yml
#	docs/build-s390x.md
#	docs/build.md
#	ggml/CMakeLists.txt
#	ggml/cmake/common.cmake
#	ggml/src/ggml-cpu/CMakeLists.txt
#	ggml/src/ggml-cpu/repack.cpp
#	ggml/src/ggml-cuda/CMakeLists.txt
#	ggml/src/ggml-cuda/fattn.cu
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/hvx-sin-cos.h
#	ggml/src/ggml-hexagon/htp/rope-ops.c
#	ggml/src/ggml-hip/CMakeLists.txt
#	ggml/src/ggml-musa/CMakeLists.txt
#	ggml/src/ggml-vulkan/ggml-vulkan.cpp
#	pyproject.toml
#	requirements/requirements-convert_legacy_llama.txt
#	requirements/requirements-gguf_editor_gui.txt
#	requirements/requirements-server-bench.txt
#	requirements/requirements-tool_bench.txt
#	scripts/make-release-checks.sh
#	tests/test-backend-ops.cpp
#	tests/test-jinja.cpp
#	tools/cli/README.md
#	tools/completion/README.md
#	tools/llama-bench/llama-bench.cpp
#	tools/server/README.md
 2026-09-13 23:12:41 +08:00
linsen458-spec
4850c7727f
llama : use int32_t for llama_sampler_chain_n return type (#28631) 
Contributes to #4574

Co-authored-by: linsen <linsen@insta360.com>
 2026-09-09 15:27:25 +03:00
Concedo
4e3a25f3bc Merge commit '9e54e687cb' into concedo_experimental 
# Conflicts:
#	docs/backend/snapdragon/windows.md
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp-drv.cpp
#	ggml/src/ggml-hexagon/htp-drv.h
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-sycl/fattn-common.hpp
#	ggml/src/ggml-sycl/fattn-onednn.cpp
#	ggml/src/ggml-sycl/fattn-onednn.hpp
#	ggml/src/ggml-sycl/fattn.cpp
#	ggml/src/ggml-sycl/fattn.hpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-virtgpu/ggml-backend.cpp
#	scripts/snapdragon/build.py
#	tests/test-alloc.cpp
#	tests/test-backend-ops.cpp
#	tools/llama-bench/README.md
#	tools/llama-bench/llama-bench.cpp
 2026-08-31 18:05:39 +08:00
Xuan-Son Nguyen
50f068ffff
bench: add --tensor-read-lazy (#27881) 
* bench: add --tensor-read-lazy

* rm the alias

* rename to LLAMA_LAZY_MODE_*
 2026-08-28 20:51:05 +02:00
Concedo
8d223ab855 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/openvino.Dockerfile
#	.github/workflows/build-cache.yml
#	.github/workflows/build-openvino.yml
#	.github/workflows/build-self-hosted.yml
#	.github/workflows/release.yml
#	ci/run.sh
#	docs/backend/OPENVINO.md
#	docs/speculative.md
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/hvx-arith.h
#	ggml/src/ggml-hexagon/htp/hvx-log.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/unary-ops.c
#	ggml/src/ggml-hexagon/htp/unary-ops.h
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-openvino/CMakeLists.txt
#	ggml/src/ggml-openvino/ggml-decoder.cpp
#	ggml/src/ggml-openvino/ggml-decoder.h
#	ggml/src/ggml-openvino/ggml-openvino-extra.cpp
#	ggml/src/ggml-openvino/ggml-openvino.cpp
#	ggml/src/ggml-openvino/openvino/op/cpy.cpp
#	ggml/src/ggml-openvino/openvino/op/flash_attn_ext.cpp
#	ggml/src/ggml-openvino/openvino/op/gated_delta_net.cpp
#	ggml/src/ggml-openvino/openvino/op/view.cpp
#	ggml/src/ggml-openvino/openvino/op_table.cpp
#	ggml/src/ggml-openvino/openvino/op_table.h
#	ggml/src/ggml-openvino/openvino/translate_session.cpp
#	ggml/src/ggml-openvino/openvino/utils.cpp
#	ggml/src/ggml-openvino/utils.cpp
#	ggml/src/ggml-openvino/utils.h
#	ggml/src/ggml-sycl/fattn-onednn.cpp
#	ggml/src/ggml-sycl/fattn.cpp
#	scripts/pr2wt.sh
#	src/CMakeLists.txt
#	src/llama-mmap.cpp
#	src/llama-quant.cpp
#	tests/CMakeLists.txt
#	tests/test-arg-parser.cpp
#	tests/test-backend-ops.cpp
#	tests/test-llama-archs.cpp
#	tests/test-save-load-state.cpp
#	tools/cli/README.md
#	tools/completion/README.md
#	tools/server/README.md
 2026-08-28 22:29:03 +08:00
Concedo
ae60e1d6e0 Merge commit 'deae5ee133' into concedo_experimental 
# Conflicts:
#	.devops/openvino.Dockerfile
#	.github/workflows/build-apple.yml
#	.github/workflows/build-cuda-ubuntu.yml
#	.github/workflows/docker.yml
#	.github/workflows/release.yml
#	.github/workflows/server-sanitize.yml
#	.github/workflows/ui-build-self-hosted.yml
#	.github/workflows/ui-build.yml
#	.github/workflows/ui-publish.yml
#	.github/workflows/ui-self-hosted.yml
#	.github/workflows/ui.yml
#	CMakeLists.txt
#	docs/backend/snapdragon/CMakeUserPresets.json
#	docs/backend/snapdragon/README.md
#	docs/backend/snapdragon/developer.md
#	docs/backend/snapdragon/linux.md
#	docs/backend/snapdragon/windows.md
#	docs/ops.md
#	docs/ops/Vulkan.csv
#	ggml/cmake/ggml-config.cmake.in
#	ggml/src/ggml-cpu/CMakeLists.txt
#	ggml/src/ggml-cpu/kleidiai/kernels.cpp
#	ggml/src/ggml-cpu/kleidiai/kernels.h
#	ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp-opnode.h
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/act-ops.c
#	ggml/src/ggml-hexagon/htp/cpy-ops.c
#	ggml/src/ggml-hexagon/htp/dma-queue.h
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/get-rows-ops.c
#	ggml/src/ggml-hexagon/htp/hex-utils.h
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/htp-tensor.c
#	ggml/src/ggml-hexagon/htp/htp-tensor.h
#	ggml/src/ggml-hexagon/htp/hvx-arith.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.h
#	ggml/src/ggml-hexagon/htp/set-rows-ops.c
#	ggml/src/ggml-rpc/CMakeLists.txt
#	scripts/snapdragon/ggml-hexagon-profile.py
#	scripts/snapdragon/ggml-hexagon-trace.py
#	tests/test-backend-ops.cpp
#	tools/cli/README.md
#	tools/llama-bench/llama-bench.cpp
#	tools/rpc/README.md
#	tools/server/README.md
#	tools/ui/tests/stories/a11y/ChatScreenForm.a11y.stories.svelte
 2026-08-28 19:31:29 +08:00
Xuan-Son Nguyen
732707dff2
quantize: cap working memory size to avoid loading big tensors onto RAM (#27795) 2026-08-27 18:31:13 +02:00
Xuan-Son Nguyen
fac889fb38
llama: model_loader: add TENSOR_READ_LAZY (#27794) 
* llama: model_loader: add TENSOR_GET_ROW_LAZY

* add --tensor-read-lazy

* rename to TENSOR_READ_LAZY

* gen docs

* address comments
 2026-08-27 15:14:34 +02:00
Xuan-Son Nguyen
925e117994
llama: add token ID tracking to KV cell (#27762) 
* kv: track token id

* rm get_prev_tokens, move it to the main pr

* nits

* add get_prev_tokens
 2026-08-26 23:34:28 +02:00
Concedo
4447017602 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/actions/ccache-clear/action.yml
#	.github/workflows/build-apple.yml
#	.github/workflows/build-cpu.yml
#	.github/workflows/build-cuda-ubuntu.yml
#	.github/workflows/build-opencl.yml
#	.github/workflows/build-openvino.yml
#	.github/workflows/build-sycl.yml
#	.github/workflows/build-vulkan.yml
#	.github/workflows/build-wasm.yml
#	.github/workflows/build-webgpu.yml
#	.github/workflows/hip-quality-check.yml
#	.github/workflows/server.yml
#	CONTRIBUTING.md
#	README.md
#	ci/run.sh
#	common/CMakeLists.txt
#	common/chat.cpp
#	docs/autoparser.md
#	ggml/src/ggml-webgpu/wgsl-shaders/flash_attn.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/flash_attn_tile.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/flash_attn_vec_split.wgsl
#	scripts/sync_vendor.py
#	tests/CMakeLists.txt
#	tests/peg-parser/test-json-serialization.cpp
#	tests/peg-parser/tests.h
#	tests/test-chat-auto-parser.cpp
#	tests/test-chat-peg-parser.cpp
#	tests/test-chat-template.cpp
#	tests/test-chat.cpp
#	tests/test-grammar-integration.cpp
#	tests/test-jinja.cpp
#	tests/test-json-schema-to-grammar.cpp
#	tests/test-llama-archs.cpp
#	tests/test-model-resolution.cpp
#	tests/test-recurrent-state-rollback.cpp
#	tools/CMakeLists.txt
 2026-08-25 20:44:34 +08:00
Aman Gupta
b0539c43ed
DeepseekV4: fix rollback with multi-seq (#26756) 
* DeepseekV4: fix rollback with multi-seq

* fix model loading

* make pending rollback single use

* only clear cache for seq_id for full load

* add assert for compress ratio

* make graph topology static

* pass true instead of flags in clear_compressed

* cont : clean-up + TODOs

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-08-23 13:57:49 +03:00
Concedo
886a42d446 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/workflows/build-cache.yml
#	.github/workflows/build-cmake-pkg.yml
#	.github/workflows/build-cpu.yml
#	.github/workflows/build-cuda-windows.yml
#	.github/workflows/build-sanitize.yml
#	.github/workflows/release.yml
#	.github/workflows/winget.yml
#	CMakeLists.txt
#	README.md
#	app/llama.cpp
#	cmake/llama-config.cmake.in
#	cmake/llama.pc.in
#	common/CMakeLists.txt
#	common/build-info.h
#	docs/backend/OPENVINO.md
#	docs/backend/SYCL.md
#	docs/preset.md
#	ggml/CMakeLists.txt
#	ggml/cmake/ggml-config.cmake.in
#	ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
#	ggml/src/ggml-hip/CMakeLists.txt
#	ggml/src/ggml-openvino/ggml-decoder.cpp
#	ggml/src/ggml-openvino/ggml-decoder.h
#	ggml/src/ggml-openvino/ggml-openvino-extra.cpp
#	ggml/src/ggml-openvino/ggml-openvino-extra.h
#	ggml/src/ggml-openvino/ggml-openvino.cpp
#	ggml/src/ggml-openvino/ggml-quants.cpp
#	ggml/src/ggml-openvino/ggml-quants.h
#	ggml/src/ggml-openvino/openvino/decoder.h
#	ggml/src/ggml-openvino/openvino/node_context.h
#	ggml/src/ggml-openvino/openvino/op/cpy.cpp
#	ggml/src/ggml-openvino/openvino/op/gated_delta_net.cpp
#	ggml/src/ggml-openvino/openvino/op/get_rows.cpp
#	ggml/src/ggml-openvino/openvino/op/l2_norm.cpp
#	ggml/src/ggml-openvino/openvino/op/mul_mat_id.cpp
#	ggml/src/ggml-openvino/openvino/op/repeat.cpp
#	ggml/src/ggml-openvino/openvino/op/reshape.cpp
#	ggml/src/ggml-openvino/openvino/op/rms_norm.cpp
#	ggml/src/ggml-openvino/openvino/op/rope.cpp
#	ggml/src/ggml-openvino/openvino/op/scale.cpp
#	ggml/src/ggml-openvino/openvino/op/set_rows.cpp
#	ggml/src/ggml-openvino/openvino/op/ssm_conv.cpp
#	ggml/src/ggml-openvino/openvino/op/view.cpp
#	ggml/src/ggml-openvino/openvino/op_table.cpp
#	ggml/src/ggml-openvino/openvino/op_table.h
#	ggml/src/ggml-openvino/openvino/translate_session.cpp
#	ggml/src/ggml-openvino/openvino/utils.cpp
#	ggml/src/ggml-openvino/openvino/utils.h
#	ggml/src/ggml-openvino/utils.cpp
#	ggml/src/ggml-openvino/utils.h
#	ggml/src/ggml-sycl/common.hpp
#	ggml/src/ggml-sycl/concat.cpp
#	ggml/src/ggml-sycl/dmmv.cpp
#	ggml/src/ggml-sycl/element_wise.cpp
#	ggml/src/ggml-sycl/element_wise.hpp
#	ggml/src/ggml-sycl/fusion.cpp
#	ggml/src/ggml-sycl/fusion.hpp
#	ggml/src/ggml-sycl/gated_delta_net.cpp
#	ggml/src/ggml-sycl/gated_delta_net.hpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-sycl/mmvq.cpp
#	ggml/src/ggml-sycl/mmvq.hpp
#	scripts/sync-ggml.last
#	src/CMakeLists.txt
#	tests/test-backend-ops.cpp
#	tests/test-chat.cpp
#	tests/test-gguf.cpp
#	tests/test-quantize-stats.cpp
#	tools/cvector-generator/cvector-generator.cpp
#	tools/mtmd/CMakeLists.txt
#	tools/ui/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionModels.svelte
 2026-08-14 18:10:18 +08:00
Daniel Bevenius
680a9ae63d
cmake : introduce semantic versioning (#26839) 
* cmake : introduce semantic versioning (wip)

This commit introduces semantic versioning to llama.cpp.

* squash! cmake : introduce semantic versioning (wip)

* cmake : update test-cmake README notes [no ci]

* include libmtmd in output so show its semversioned

* ci : add make-release workflow

* ci : fix build number check in build-cmake-pkg.yml

* examples : remove trailing whitespace

* ci : abort if upstream ggml version does not exist

* ci : extract step contents into scripts

* ci : add GGML_NATIVE=OFF to ubuntu job

* examples : remove CI build information from test-cmake [no ci]

This commit removes the nightly/release information that I added
previously to keep this focused only on using building and installing
llama.cpp with cmake and being able to quickly verify changes or
troubleshoot issues.

* ci : merge scripts into single script

* remove -dev-build_number support

This commit removes the incremental build number (versioning) support
that I added. This was incorrect and we should only use the semver for
the version. Releases will be tag a nightly build and package
maintainers/managers that build from source can use the tag and it is
therefor important that the correct version is reported. So a
nightly-build will report the semver without the build number. The build
number and commit as availble via cmake and test-cmake has been updated
to include an example of using them:
```console
$ ./build.sh
[test-cmake] version: 0.1.0, build: 10360 (08c69e381)
...
```

Refs: https://github.com/ggml-org/llama.cpp/pull/26839#discussion_r3755836969

* docs: add initial release.md documentation

* cmake : clean-up and add LLAMA_BUILD_IS_DEV option

* ci : remove version input from make-release job

* ci : add LLAMA_BUILD_IS_DEV=OFF to build-cmake-pkg.yml

Refs: https://github.com/danbev/llama.cpp/actions/runs/31576801921/job/94050639145

* docs : update release notes with LLAMA_BUILD_IS_DEV info [no ci]

* ci : add TODO to winget workflow [no ci]

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-08-12 14:15:03 +02:00
Chipmunk
5d9e5ac30e
server : support slot save/restore with media inputs (#26640) 
* server : save serialized image chunks at the end of the llama state

* server : support multimodal slot state save/restore with packed payload

* server : refine image slot state serialization

* server : support media slot state and centralize media validation

* server : remove unnecessary comment

* server : remove defensive media checks and move the chunk type check to validate()
 2026-08-12 12:20:28 +02:00
Concedo
dcfe29317e Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/actions/windows-setup-rocm/action.yml
#	.github/workflows/build-cache.yml
#	.github/workflows/build-cuda-windows.yml
#	.github/workflows/release.yml
#	.github/workflows/server-sanitize.yml
#	.github/workflows/server-self-hosted.yml
#	.github/workflows/server.yml
#	ci/run.sh
#	docs/speculative.md
#	examples/lookup/lookup.cpp
#	examples/speculative-simple/speculative-simple.cpp
#	examples/speculative/speculative.cpp
#	ggml/src/ggml-cann/ggml-cann.cpp
#	ggml/src/ggml-et/ggml-et.cpp
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/flash_attn_f32_f16.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f32_q4_0.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f32_q8_0.cl
#	ggml/src/ggml-openvino/ggml-openvino.cpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-virtgpu/backend/backend-dispatched-device.cpp
#	ggml/src/ggml-virtgpu/backend/shared/api_remoting.h
#	ggml/src/ggml-virtgpu/ggml-backend-buffer-type.cpp
#	ggml/src/ggml-virtgpu/ggml-backend-device.cpp
#	ggml/src/ggml-virtgpu/virtgpu-forward-device.cpp
#	ggml/src/ggml-virtgpu/virtgpu-forward.gen.h
#	ggml/src/ggml-vulkan/ggml-vulkan.cpp
#	ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/cpy.wgsl
#	ggml/src/ggml-zdnn/ggml-zdnn.cpp
#	ggml/src/ggml-zendnn/ggml-zendnn.cpp
#	tests/peg-parser/test-json-parser.cpp
#	tests/test-arg-parser.cpp
#	tests/test-backend-sampler.cpp
#	tests/test-sampling.cpp
#	tools/cli/README.md
#	tools/completion/README.md
#	tools/llama-bench/llama-bench.cpp
#	tools/server/README.md
 2026-08-11 15:13:15 +08:00
Ruben Ortlam
153d324bcf
llama: add default load-mode auto, which avoids mmap on iGPUs (#26081)
Some checks failed
Python Type-Check / python type-check (push) Has been cancelled
Details 
* llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used

* Update ggml/src/ggml-hexagon/ggml-hexagon.cpp

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>

* set mmap_support to false on OpenCL backend

* fix order of load modes

* use -1 for auto

* resolve load mode auto earlier to correctly pick gpu host or cpu memory

* add load mode auto to llama-bench

* bump virtgpu api version, regenerate docs

---------

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-08-11 09:20:46 +03:00
Gaurav Garg
030ebb558a
Address review comment of PR 25532 (#26852) 2026-08-11 00:02:25 +05:30
Gaurav Garg
dd1ea52433
llama : support multi-output backend sampling (#25532) 
* Enable backend sampling with token speculation

* Clamp the mask sum before converting it into the sampled index

* Add a numeric context parameter declaring the maximum outputs one sequence

* More fixes

* Don't reuse memory for output views.

* Match dist between CPU and GPU

* Fix CPU and backend sampling mismatches

* Simpify some of the changes

* Fix tests on Vulkan

* More test fixes

* Rebase changes

* Rebase and address review comments

* Address review comments

* Address review comments

* Update src/llama-sampler.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-08-10 16:58:56 +03:00
Concedo
8a16f96307 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/workflows/build-apple.yml
#	.github/workflows/build-self-hosted.yml
#	.github/workflows/release.yml
#	SECURITY.md
#	build-xcframework.sh
#	ci/run.sh
#	docs/development/HOWTO-add-model.md
#	examples/model-conversion/scripts/causal/convert-model.sh
#	examples/model-conversion/scripts/embedding/convert-model.sh
#	scripts/sync_vendor.py
#	scripts/ui-assets.cmake
#	tests/test-arg-parser.cpp
#	tests/test-backend-sampler.cpp
#	tests/test-grammar-parser.cpp
#	tests/test-llama-archs.cpp
#	tests/test-sampling.cpp
#	tools/cli/README.md
#	tools/completion/README.md
#	tools/mtmd/CMakeLists.txt
#	tools/mtmd/mtmd.h
#	tools/mtmd/tests/test-deepseek-ocr.py
#	tools/server/README.md
#	tools/tts/CMakeLists.txt
#	tools/tts/convert_pt_to_hf.py
 2026-08-07 20:46:56 +08:00
Concedo
2580b4591d Merge commit '1c3c9674de' into concedo_experimental 
# Conflicts:
#	.github/workflows/build-apple.yml
#	.github/workflows/build-vulkan.yml
#	.github/workflows/release.yml
#	docs/ops.md
#	docs/ops/Vulkan.csv
#	examples/gen-docs/gen-docs.cpp
#	ggml/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-sycl/concat.cpp
#	ggml/src/ggml-sycl/fattn-onednn.cpp
#	ggml/src/ggml-sycl/fattn.cpp
#	models/templates/deepseek-ai-DeepSeek-V4.jinja
#	scripts/sync-ggml.last
#	scripts/sync_vendor.py
#	src/CMakeLists.txt
#	src/llama-model-loader.cpp
#	tests/CMakeLists.txt
#	tests/test-arg-parser.cpp
#	tests/test-backend-sampler.cpp
#	tests/test-chat.cpp
#	tests/test-sampling.cpp
#	tools/server/README.md
 2026-08-07 18:16:11 +08:00
Concedo
7098c9c53f Merge commit '0b14b87d7c' into concedo_experimental 
# Conflicts:
#	.github/workflows/build-cpu.yml
#	.github/workflows/release.yml
#	AGENTS.md
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-sycl/common.hpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/repeat.wgsl
#	src/models/qwen3next.cpp
#	tests/test-backend-ops.cpp
#	tests/test-chat-peg-parser.cpp
#	tests/test-chat.cpp
#	tests/test-llama-archs.cpp
#	tests/test-recurrent-state-rollback.cpp
 2026-08-07 18:04:33 +08:00
Oliver Simons
a6aa6f5450
sampler : remove "full-context windows" from history-based samplers (#26524) 
* Resolve -1 to 1024 instead of ctx-len for samplers

Because of backend-sampling we initialize samplers before the complete
llama_context is there. Therefore, we cannot infer the resolved context
length yet at the time we construct the samplers.

* Shared default of 64 for history-based samplers, remove context_size
 2026-08-04 21:28:55 +03:00
Oliver Simons
935cad6497
llama : move n_vocab from llama_sampler_data to penalty_sampler (#26520) 
This matches how it is done for logit_bias and mirostat samplers, see
https://github.com/ggml-org/llama.cpp/pull/25262#discussion_r3703951151
 2026-08-04 09:02:49 +03:00
Konrad Moren
96278e39fc
CUDA: Add backend sampler for penalties sampler (#25262) 
* sampling: enhance penalty handling in common_sampler_init

- Set default value for penalty_last_n based on model context if not specified.
- Ensure penalty_last_n and n_prev are non-negative.
- Update llama_sampler_penalties structure to inherit from llama_sampler_backend and add backend input handling for penalties.
- Implement backend initialization and application logic for penalties, including frequency and presence adjustments.

* tests: add backend penalties sampling tests and utility functions

- Introduced `accept_prompt` and `unique_prompt_tokens` functions to handle prompt acceptance and token uniqueness.
- Implemented `compare_penalties_logits` to compare logits from backend and CPU samplers with penalties.
- Added `test_backend_penalties_sampling` to validate backend penalties with various configurations.
- Enhanced the test suite for better coverage of penalty handling in sampling.

* sampling: add support for top-k penalties in backend sampling

* sampling: add fix to ensure  stable numerical results. Preserve masked logits as -Inf and no longer generate NaN.

* sampling: enhance penalty comparison tests with masking penalties logic

* add comments on padding

* sampling: add comments on modifications

* add the unit test to cover masked-out token as -INF

* validate repeat penalty to ensure it is finite and greater than 0; add tests for invalid values

* refactor: test functions to share logic and be less verbose

* add test to cover case where previously penalized token is not part of candidates

* remove comments

* remove redundant penalty_last_n initialization and validation in common_sampler_init

* add support for penalties in sampler chain with configurable positions

* add validation for penalty parameters and enhance tests for non-finite values

* add context parameter to common_sampler_init and set default for penalty_last_n

* add llama_n_ctx parameter to common_sampler_init for improved sampler initialization

* replace penalty_last_n x n_candidates comparison matrix with a vocabulary-sized count tensor

* add tests for backend penalties sampling without filler entries , token_count.size() == n_active == n_max == 64

* add test for backend penalties sampling  after top-p with large history window

* remove as unused

* add is_disabled method, tensor logits reshape, add rest review suggestions

* clarify comment
 2026-08-03 14:26:09 +02:00
fairydreaming
82dbc4f017
llama : load MTP tensors only if they are really used (#26296) 
* llama : load MTP tensors only if they are really used

* llama : skip loading MTP (if not used) in remaining models that support MTP

---------

Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
 2026-07-31 14:57:02 +02:00
Concedo
4a69c10078 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	README.md
#	docs/backend/SYCL.md
#	ggml/CMakeLists.txt
#	ggml/src/ggml-cpu/repack.cpp
#	ggml/src/ggml-sycl/backend.hpp
#	ggml/src/ggml-sycl/common.hpp
#	ggml/src/ggml-sycl/convert.cpp
#	ggml/src/ggml-sycl/cpy.cpp
#	ggml/src/ggml-sycl/cpy.hpp
#	ggml/src/ggml-sycl/dequantize.hpp
#	ggml/src/ggml-sycl/fattn.cpp
#	ggml/src/ggml-sycl/fattn.hpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-sycl/mmvq.cpp
#	ggml/src/ggml-sycl/norm.cpp
#	ggml/src/ggml-sycl/norm.hpp
#	ggml/src/ggml-sycl/vecdotq.hpp
#	ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/common_decls.tmpl
#	ggml/src/ggml-webgpu/wgsl-shaders/flash_attn.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/flash_attn_quant_staging.tmpl
#	ggml/src/ggml-webgpu/wgsl-shaders/flash_attn_tile.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/flash_attn_vec_split.wgsl
#	ggml/src/ggml-zendnn/CMakeLists.txt
#	ggml/src/ggml-zendnn/ggml-zendnn.cpp
#	scripts/sync-ggml.last
#	tests/CMakeLists.txt
#	tests/snapshots/qwen3.6-27b.schema
#	tests/test-autorelease.cpp
#	tests/test-backend-ops.cpp
#	tests/test-backend-sampler.cpp
#	tests/test-model-load-cancel.cpp
#	tests/test-quant-type-selection.cpp
 2026-07-31 19:44:27 +08:00
Xuan-Son Nguyen
afeebe103b
llama: move suppress_tokens handling to common/sampling (#26276) 
* llama: move suppress_tokens handling to common/sampling

* address security issues

* rm has_logit_bias
 2026-07-29 18:02:30 +02:00
Concedo
fee0bf446c Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/workflows/build-webgpu.yml
#	CMakeLists.txt
#	common/CMakeLists.txt
#	docs/development/HOWTO-add-model.md
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-sycl/CMakeLists.txt
#	tests/test-arg-parser.cpp
#	tests/test-jinja.cpp
#	tests/test-llama-archs.cpp
#	tests/test-save-load-state.cpp
#	tools/cli/README.md
#	tools/completion/README.md
#	tools/llama-bench/llama-bench.cpp
#	tools/mtmd/CMakeLists.txt
#	tools/server/README.md
 2026-07-27 22:28:38 +08:00
Aaron Teo
ad256ded30
args: add -lm mlock where it mlocks but doesnt mmap (#26135) 
* arg: add `-lm mlock` where it mlocks but doesnt mmap

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: rm unwanted docs changes

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: revert auto-formatting

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* bench: fix automated review point 3

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* arg: revert the meaning of --mlock to non-mmap'ed mlock

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: update docs

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: remove extra changes from `llama-gen-docs`

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2026-07-27 16:44:08 +08:00
Concedo
49dbdaaab5 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	AGENTS.md
#	CODEOWNERS
#	CONTRIBUTING.md
#	docs/backend/OPENCL.md
#	docs/development/HOWTO-add-model.md
#	examples/training/finetune.cpp
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp-drv.cpp
#	ggml/src/ggml-hexagon/htp/act-ops.c
#	ggml/src/ggml-hexagon/htp/dma-queue.c
#	ggml/src/ggml-hexagon/htp/dma-queue.h
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.h
#	ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/htp-tensor.c
#	ggml/src/ggml-hexagon/htp/htp-tensor.h
#	ggml/src/ggml-hexagon/htp/hvx-fa-kernels.h
#	ggml/src/ggml-hexagon/htp/hvx-reduce.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.h
#	ggml/src/ggml-hexagon/htp/unary-ops.c
#	ggml/src/ggml-hexagon/htp/unary-ops.h
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	scripts/compare-llama-bench.py
#	scripts/snapdragon/ggml-hexagon-profile.py
#	scripts/snapdragon/ggml-hexagon-trace.py
#	scripts/sync_vendor.py
#	tests/test-arg-parser.cpp
#	tests/test-chat.cpp
#	tests/test-model-load-cancel.cpp
#	tests/test-quantize-stats.cpp
#	tools/cli/README.md
#	tools/completion/README.md
#	tools/llama-bench/llama-bench.cpp
#	tools/server/README.md
#	tools/ui/src/lib/constants/settings-registry.ts
 2026-07-25 12:20:51 +08:00
Aaron Teo
e6dd0e29a6
args: refactor mlock/mmap/directio into load-mode (#20834) 
* args: overhaul mmap/mlock/dio into single arg

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: update docs with llama-gen-docs

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* chore: satisfy code quality

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* args: make the `+` sign an actual modifier now

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* chore: general code clean up + comments

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* arg: fix deprecated flags support

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* arg: quick sanity check

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* bench: sync llama-bench argument parsing

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* fix: bugfix variable behaviour + llama-bench lm column size

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* arg: inverse commands should do the opposite instead of doing nothing

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* bench: fix incorrect dash

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* bench: fix missing modifiers for deprecated flags

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* llama: switch back to thread_local

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* arg: switch back to single enum

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: update arg docs

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* chore: fix missing `mlock` from llama_load_mode_from_str + cleanup llama-bench

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* llama: fix mlock not activating

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* arg: add deprecation warning when old and new flags are combined

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* arg: cont add comment for todo in the future

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: sync with upstream

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: re-sync with upstream again

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2026-07-23 20:32:56 +08:00
Concedo
0f0245161e Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	tests/test-backend-ops.cpp
#	tests/test-quantize-fns.cpp
#	tools/server/README.md
#	tools/ui/src/lib/constants/settings-registry.ts
 2026-07-11 09:20:40 +08:00
Pasha Khosravi
bec4772f6a
Add Q2_0 quantization: type definition and CPU backend (#24448) 2026-07-07 12:05:47 -07:00
Concedo
e944cca86f Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	scripts/sync_vendor.py
#	src/llama-model-loader.cpp
#	tests/test-backend-ops.cpp
#	tests/test-chat-auto-parser.cpp
#	tests/test-chat.cpp
#	tools/cli/cli.cpp
#	tools/server/README.md
 2026-07-05 11:30:12 +08:00
Adrien Gallouët
fdb1db877c
llama : add llama_model_ftype_name() (#25134) 
* llama : add llama_model_ftype_name()

Expose the model file type (quantization) name, e.g. "Q8_0" or
"Q4_K - Medium", through a new public C API. The returned pointer is
valid for the lifetime of the model and nullptr when the model is
invalid or the file type is unknown.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Export enum

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* s/llama_model_ftype_name/llama_ftype_name/

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Move "(guessed)" to the front in llama_ftype_name

Prepend the "(guessed)" label instead of appending it. This allows removing
the non-thread-safe static std::string, making the function allocation-free.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Add LLAMA_FTYPE_PREFIX

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Dont check for model

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-07-02 17:26:47 +02:00
Concedo
3090ae0bf7 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/s390x.Dockerfile
#	.dockerignore
#	.github/workflows/docker.yml
#	.github/workflows/release.yml
#	docs/android.md
#	ggml/src/ggml-cpu/amx/mmq.cpp
#	ggml/src/ggml-hexagon/htp/ssm-conv.c
#	tests/peg-parser/test-gbnf-generation.cpp
#	tests/test-arg-parser.cpp
#	tests/test-chat.cpp
#	tests/test-jinja.cpp
#	tests/test-json-schema-to-grammar.cpp
#	tools/server/README.md
 2026-06-22 18:23:59 +08:00
YiChen Lv
d789527482
spec : Support Step3.5/3.7 flash mtp3 (#24340) 
* add mtp_layer_offset + include nextn flags in graph reuse

* add llama_set_mtp_layer_offset + llama_model_n_nextn_layer API

* offset head select + require all MTP blocks

* speculative multi-head process()

* speculative multi-head draft()

* gather outputs via inp_out_ids

* cleanup

* fix core

* minor cleanup

* merged draft_multi_head into draft()

* mtp rename nextn

* Apply suggestions from code review

Co-authored-by: Aman Gupta <amangupta052@gmail.com>

* clean-up comments

* fix for multi seq

* apply suggestions && chain-heads comment

* add a reference for chain_heads discussion

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
 2026-06-21 11:33:18 +03:00
Concedo
dd2f966f4b Merge commit 'c2b1518fd4' into concedo_experimental 
# Conflicts:
#	.devops/intel.Dockerfile
#	ggml/CMakeLists.txt
#	scripts/sync-ggml.last
#	tests/test-backend-ops.cpp
#	tests/test-llama-archs.cpp
#	tools/cli/cli.cpp
 2026-06-10 00:56:44 +08:00
Aman Gupta
04eb4c446d
llama : add Gemma4 MTP (#23398) 2026-06-07 20:50:54 +08:00
Concedo
13eaa04269 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/nix/package.nix
#	.github/workflows/server.yml
#	docs/development/HOWTO-add-model.md
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/argsort-ops.c
#	ggml/src/ggml-hexagon/htp/concat-ops.c
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c
#	ggml/src/ggml-hexagon/htp/hmx-flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
#	ggml/src/ggml-hexagon/htp/hmx-ops.h
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/hvx-utils.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.c
#	ggml/src/ggml-hexagon/htp/pad-ops.c
#	ggml/src/ggml-hexagon/htp/unary-ops.c
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/cvt.cl
#	ggml/src/ggml-webgpu/wgsl-shaders/cpy.wgsl
#	scripts/sync_vendor.py
#	src/llama-context.cpp
#	tests/test-backend-sampler.cpp
#	tools/server/README.md
#	tools/ui/tests/stories/ChatScreenForm.a11y.stories.svelte
 2026-06-02 19:28:39 +08:00
Georgi Gerganov
4f3a4beb8d
llama : deprecate llama_set_warmup (#24009) 
* llama : deprecate `llama_set_warmup`

* cont : fix type

Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>

---------

Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>
 2026-06-02 10:30:38 +03:00
Aman Gupta
de6f727aae
llama: limit max outputs of llama_context (#23861) 
* llama: save more VRAM by reserving n_outputs == n_seqs when possible

* add n_outputs_per_seq

* move n_outputs_max to server-context

* change ubatch to batch everywhere
 2026-06-01 18:01:38 +03:00
Concedo
9204f78926 Merge commit 'ae251b5ff2' into concedo_experimental 
# Conflicts:
#	.github/actions/linux-setup-spacemit/action.yml
#	.github/actions/unarchive-tar/action.yml
#	.github/workflows/build-android.yml
#	.github/workflows/build-cmake-pkg.yml
#	.github/workflows/build-cross.yml
#	.github/workflows/build-self-hosted.yml
#	.github/workflows/build.yml
#	.github/workflows/check-vendor.yml
#	.github/workflows/code-style.yml
#	.github/workflows/editorconfig.yml
#	.github/workflows/pre-tokenizer-hashes.yml
#	.github/workflows/python-check-requirements.yml
#	.github/workflows/python-lint.yml
#	.github/workflows/python-type-check.yml
#	.github/workflows/server-self-hosted.yml
#	.github/workflows/ui-build.yml
#	.github/workflows/ui.yml
#	.github/workflows/update-ops-docs.yml
#	ci/run.sh
#	docs/build-riscv64-spacemit.md
#	examples/convert_legacy_llama.py
#	ggml/cmake/ggml-config.cmake.in
#	ggml/src/CMakeLists.txt
#	ggml/src/ggml-cpu/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	scripts/sync_vendor.py
#	tests/test-chat-auto-parser.cpp
#	tests/test-chat.cpp
#	tests/test-gguf.cpp
#	tools/cli/README.md
#	tools/perplexity/perplexity.cpp
#	tools/server/README.md
 2026-05-26 22:01:57 +08:00
Tim Neumann
314e729347
llama : document that only one on-device state can be saved per sequence (#23520) 2026-05-25 10:29:28 +03:00