vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-10-26 08:09:57 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
koboldcpp/examples/parallel
History
Xuan-Son Nguyen 60e9cf7a7b
batch: migrate the rest of examples to llama_batch_ext (#29601) 
* migrate the rest

* test-thread-safety

* rm common_batch_staged
2026-09-30 18:08:43 +02:00
..
CMakeLists.txt libs : rename libcommon -> libllama-common (#21936) 2026-04-17 11:11:46 +03:00
parallel.cpp batch: migrate the rest of examples to llama_batch_ext (#29601) 2026-09-30 18:08:43 +02:00
README.md parallel : increase the variability of the prompt lengths (#13927) 2025-05-30 19:38:07 +03:00

README.md

llama.cpp/example/parallel

Simplified simulation of serving incoming requests in parallel

Example

Generate 128 client requests (-ns 128), simulating 8 concurrent clients (-np 8). The system prompt is shared (-pps), meaning that it is computed once at the start. The client requests consist of up to 10 junk questions (--junk 10) followed by the actual question.

llama-parallel -m model.gguf -np 8 -ns 128 --top-k 1 -pps --junk 10 -c 16384

Note

It's recommended to use base models with this example. Instruction tuned models might not be able to properly follow the custom chat template specified here, so the results might not be as expected.