mirror of
https://github.com/LostRuins/koboldcpp.git
synced 2026-10-24 15:20:02 +00:00
033df86b69
* server : preserve context checkpoints across slot save/restore Append the checkpoints after the packed server_tokens payload added in #26640 and count them in n_written / n_read, so a restored slot can still roll back to a checkpoint instead of re-processing the whole prompt. * server : drop draft checkpoint data that does not match the draft context Restoring a slot saved with a different draft KV cache type aborted in load_dft(). Test-load one draft checkpoint on restore and drop the draft data if it does not fit, instead of crashing. Adds a regression test. Co-authored-by: Igor Okulist <okigan@gmail.com> * server : harden the checkpoint appendix of slot save files Bound each blob size by the bytes left in the file before allocating, open the file with UTF-8 paths on Windows like the llama state payload, fall back to full prompt re-processing when a checkpoint restored from a slot file fails to load, and replace the 1024 count cap by keeping the last n_ctx_checkpoints while reading. * server : report an incomplete checkpoint appendix as a failed slot save Return an error to the client when the appendix cannot be written, like a failed payload write, and make the oversized-blob test declare a size that cannot be allocated, so an unbounded allocation fails the test. * server : reject an empty target state in the checkpoint appendix A saved checkpoint always holds a target state, an empty blob would roll back without restoring anything. Also log with the slot id, and load the draft test model from the HF cache instead of a second download. * common : return bool from checkpoint load_tgt / load_dft A checkpoint restored from a slot file falls back to full prompt re-processing when it fails to load, a checkpoint created in memory still aborts. --------- Co-authored-by: Igor Okulist <okigan@gmail.com> |
||
|---|---|---|
| .. | ||
| CMakeLists.txt | ||
| README.md | ||
| speculative-simple.cpp | ||
llama.cpp/examples/speculative-simple
Demonstration of basic greedy speculative decoding
# spec-type draft-simple
./bin/llama-speculative-simple \
-hf ggml-org/Qwen3-8B-Base-GGUF:Q8_0 \
-hfd ggml-org/Qwen3-0.6B-Base-GGUF \
-p "Here is a quick sort implementation in C++. Just code, no comments:\n\n#include" \
--spec-type draft-simple --spec-draft-n-max 7 -ngld 99 --color on \
-n 256 --temp 0 --top-k 1 --seed 42 -ngl 99 -lv 4
# spec-type draft-mtp
./bin/llama-speculative-simple \
-hf ggml-org/Qwen3.6-27B-GGUF:Q8_0 \
-p "Here is a quick sort implementation in C++. Just code, no comments:\n\n#include" \
--spec-type draft-mtp --spec-draft-n-max 3 -ngld 99 --color on \
-n 256 --temp 0 --top-k 1 --seed 42 -ngl 99 -lv 4
# spec-type draft-mtp (with shared KV cache)
# note: this model needs a <s> token at the start to somewhat work without the chat template
./bin/llama-speculative-simple \
-hf ggml-org/Gemma-4-31B-it-GGUF:Q8_0 \
-p "<s>Here is a quick sort implementation in C++. Just code, no comments:\n\n#include" \
--spec-type draft-mtp --spec-draft-n-max 3 -ngld 99 --color on \
-n 256 --temp 0 --top-k 1 --seed 42 -ngl 99 -lv 4
# spec-type draft-eagle3
./bin/llama-speculative-simple \
-hf ggml-org/gpt-oss-20b-GGUF \
-p "Here is a quick sort implementation in C++. Just code, no comments:\n\n#include" \
--spec-type draft-eagle3 --spec-draft-n-max 3 -ngld 99 --color on \
-n 256 --temp 0 --top-k 1 --seed 42 -ngl 99 -lv 4
# spec-type draft-dflash
./bin/llama-speculative-simple \
-hf ggml-org/Qwen3-8B-GGUF \
-p "Here is a quick sort implementation in C++. Just code, no comments:\n\n#include" \
--spec-type draft-dflash --spec-draft-n-max 7 -ngld 99 --color on \
-n 256 --temp 0 --top-k 1 --seed 42 -ngl 99 -lv 4
# spec-type draft-dspark
./bin/llama-speculative-simple \
-hf ggml-org/Qwen3-8B-GGUF \
-p "Here is a quick sort implementation in C++. Just code, no comments:\n\n#include" \
--spec-type draft-dspark --spec-draft-n-max 7 -ngld 99 --color on \
-n 256 --temp 0 --top-k 1 --seed 42 -ngl 99 -lv 4