mirror of
https://github.com/unslothai/unsloth.git
synced 2026-10-15 15:27:15 +00:00
16ec45112b
Refs https://github.com/unslothai/unsloth/issues/6488 `unsloth run --engine vllm` / `--engine sglang` already serve safetensors models, and Studio's run settings already have a "Multi-GPU mode" with "Replicas (data parallel)". The headless one-liner had no way to reach either: an engine load from the CLI always landed on the first GPU only, so a 6-GPU box ran one GPU. This adds two flags to `unsloth studio run` (and `unsloth run`): | Flag | Sends | Notes | |---|---|---| | `--engine-parallelism tensor\|pipeline\|data` | `engine_parallelism` | Same field the Studio "Multi-GPU mode" picker sends. Only valid with `--engine vllm/sglang`. `pipeline` and `data` default to every GPU Studio can see, since a single GPU makes them a no-op. | | `--gpus 0,1,...` | `gpu_ids` | Same field the Studio GPU picker sends. Works for GGUF and engine loads. | ``` # One vLLM replica per visible GPU behind one endpoint, requests spread across them unsloth run --model unsloth/Qwen3-8B --engine vllm --engine-parallelism data # Only GPUs 2-5 unsloth run --model unsloth/Qwen3-8B --engine vllm --engine-parallelism data --gpus 2,3,4,5 # Pin a GGUF server to one card unsloth run --model unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q6_K_XL --gpus 3 ``` ## Compatibility - Backend, frontend and the load API are untouched. The payload only gains `engine_parallelism` / `gpu_ids` when the flags are set, so default runs send exactly what they send today. - The studio-venv re-exec forwards the new flags only when set, so an older studio-venv CLI (which would hand an unknown flag to llama-server) still sees today's argv for default runs. - `--engine-parallelism` without `--engine` and malformed `--gpus` (`0,x`, `0,0`, `-1`) exit 2 before anything starts. ## Not in this PR GGUF replicas (several llama-server copies of one model on spare GPUs behind one endpoint) are still not implemented. That needs a router plus multi-process lifecycle in the GPU arbiter and is a separate change. For GGUF on one host today, `--parallel N --no-kv-unified` on a single GPU is usually the fastest option for MoE models (see the benchmark in the issue). ## Tests - New: payload fields, `--gpus` parsing, re-exec forwarding when set and omission when unset, both refusals. They fail on `main` (16 failures) and pass here. - `unsloth_cli/tests`: 1935 passed, 4 skipped. - Wiring check through the real backend code (CPU, two GPUs visible): CLI payload -> `LoadRequest` -> `managed_engine.validate_load` -> vLLM argv. | CLI | `gpu_ids` sent | `validate_load` | vLLM argv | |---|---|---|---| | `--engine vllm` (today's only option) | none | `[0]` | `--tensor-parallel-size 1` | | `--engine vllm --engine-parallelism data` | `[0, 1]` | `[0, 1]` | `--tensor-parallel-size 1 --data-parallel-size 2` | | `--engine vllm --engine-parallelism data --gpus 1` | `[1]` | `[1]` | `--data-parallel-size 1` | |
||
|---|---|---|
| .. | ||
| commands | ||
| tests | ||
| __init__.py | ||
| __main__.py | ||
| _inference.py | ||
| _install_kernels.py | ||
| _model_catalog.py | ||
| _ssl_keylog.py | ||
| _studio_deps.py | ||
| _studio_runtime_gate.py | ||
| _studio_stage.py | ||
| _system_dir_guard.py | ||
| _tool_policy.py | ||
| claude_subagent_mcp.py | ||
| codex_fallback_prompt.md | ||
| codex_subagent_mcp.py | ||
| config.py | ||
| options.py | ||
| pi_subagent.ts | ||