11394 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Daniel Han	0a96b9de31	Bump install.sh / install.ps1 pins to unsloth>=2026.10.7, unsloth-zoo>=2026.10.7 (#14276) ... ## Summary - PyPI release [unsloth 2026.10.7](https://pypi.org/project/unsloth/2026.10.7/) is now live, alongside [unsloth_zoo 2026.10.7](https://pypi.org/project/unsloth-zoo/2026.10.7/). - Bumps the pinned floors in `install.sh` and `install.ps1` from `unsloth>=2026.10.6` / `unsloth-zoo>=2026.10.6` to `unsloth>=2026.10.7` / `unsloth-zoo>=2026.10.7` so fresh installs resolve to the new wheels. Every install call site reads these two variables, so this is a two-line change per script. - `pyproject.toml` already carries `unsloth_zoo>=2026.10.7` on all three lines, so it is unchanged. - Tagged on main as `v0.1.909-beta`. ## Test plan - [x] `bash -n install.sh` - [x] `uv pip install --dry-run "unsloth>=2026.10.7" "unsloth-zoo>=2026.10.7"` resolves cleanly - [ ] Sanity-run `install.sh` on a clean Linux host - [ ] Sanity-run `install.ps1` on a clean Windows host	2026-10-19 09:19:44 -07:00
Daniel Han	e680e303ea	Tests: start every diffusion_segment test with no GPU arbiter resident (#14271) ... ## Summary Two tests in `studio/backend/tests/test_diffusion_segment.py` fail in the `(Python 3.13, a-k)` shard on PRs with different bases (#14229, #14245): ``` test_choose_device_needs_room_on_the_card test_choose_device_reads_bf16_on_the_chosen_card E AssertionError: assert ('cpu', torch.float32) == ('cuda:1', torch.bfloat16) ``` Both pass when the file runs on its own (64 passed). ## Cause `choose_device()` (from #14235, SAM 3 masks) puts SAM 3 on the CPU when another GPU arbiter owner holds the card (`_other_owner_on` -> `gpu_arbiter.residents()`). The arbiter's `_owner` / `_residents` are module globals. These two tests stub torch and the free-memory probe but not the arbiter. So when an earlier test in the same xdist worker leaves a CHAT owner behind, they see the card as taken and pick the CPU. The product is right to refuse a card another owner holds. The tests were order dependent. ## Change Test-only: an autouse fixture in `test_diffusion_segment.py` clears `gpu_arbiter._owner` and `_residents` before each test (with `monkeypatch`, so they are restored). Tests that exercise ownership already set their own owner, and still do. ## Testing - **`test_diffusion_segment.py`:** 64 passed. - **Sabotage:** I added a throwaway test that sets `gpu_arbiter._owner = gpu_arbiter.CHAT` and ran it first in the same process: - **Without the fixture:** `test_choose_device_needs_room_on_the_card` and `test_choose_device_reads_bf16_on_the_chosen_card` fail. These are the two CI failures. - **With it:** all pass. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-10-19 08:57:12 -07:00
Daniel Han	7b577a921c	Versioning	2026-10-19 08:56:41 -07:00
Daniel Han	87c6408455	Studio: a chat remembers the model it last used, not only the one it started on (#14259) ... Related to #9805. ## Problem A saved chat stores the model it was started on, and when you reopen it on a different model the header offers a "Switch back" chip for that model. The stored model was written once, at the chat's first send, and never again. So if you start a chat on one model, switch to a stronger one partway through and keep going, reopening that chat later offers the first model back, not the one you were actually using. For anyone who switches models per task, the offer is wrong exactly when it matters. ## Change - Every send now records the model (and GGUF quant) it ran on in the chat's row when it differs from what is stored. Sending again on the same model writes nothing, so repeated sends are a no-op. - This replaces the narrower queued-send backfill, which only filled in a row that had no model. That case is still covered: an empty row differs from the model the send resolved to. - Compare mode panes are left alone: their labels and pairing read the stored model. - An unknown quant never clears a saved one for the same model. - The write happens once the run has passed every check that can refuse the send, and is not awaited, so it never holds up the request. If it fails, only the reopen offer is affected. - The chip and popover now read "Last used model" instead of "Original model", since that is what they show. Nothing else moves: the chip still never loads a model on its own (a local load can be several GB), it still hides when the chat is already on its model or the model is gone, and the model picker stays global. The other asks in #9805 (a per-Project default model, a per-chat picker) need UI design and are left for a follow-up. ## Testing - `npm test` in `studio/frontend`: 12818 passed, 0 failed. - `node scripts/typecheck.mjs --tests`: clean. - `pytest tests/studio/test_multi_chat_prompt_queue_contract.py`: 16 passed. - New unit cases in `chat-remembers-its-model.test.ts` cover a model switch, a quant switch, a repeat send (no write), an unknown quant, a compare pane, and the empty row a queued send fills in. - ESLint on the touched files reports the same findings as `main`. ## Before / after One chat on three models served by a scripted stub: the first message on `model-alpha`, the second on `model-beta`, then the picker moved to `model-gamma` and the chat reopened. Before, the chip still offers `model-alpha`, which the chat stopped using; after, it offers `model-beta`, the model the last message ran on. The stored row reads `model-alpha` before and `model-beta` after. 	2026-10-19 08:48:12 -07:00
Daniel Han	d6a7ae8bd6	unsloth start vscode: print the group instead of writing a dead file inside a plain container (#14237) ... Fixes https://github.com/unslothai/unsloth/issues/10052 #13439 added `unsloth start vscode`, which puts the model Unsloth is serving into VS Code's Copilot Chat model picker. When there is no VS Code user dir on the machine it runs on, and the session is remote (SSH, a dev container or a Codespace), it writes nothing and prints the group to paste into a local VS Code window, since a file written there is never read. A plain container was missed. `docker run unsloth/unsloth` (Studio on :8000) sets none of `SSH_*`, `REMOTE_CONTAINERS` or `CODESPACES`, so running `unsloth start vscode` from a shell in the container wrote `~/.config/Code/User/chatLanguageModels.json` inside the container and printed `Added ... to VS Code`. VS Code on the host never reads that file, so nothing shows up in the model picker. ## Change - `_vscode_over_ssh` also treats a Docker (`/.dockerenv`) or Podman (`/run/.containerenv`) container as remote when the VS Code user dir does not exist there. - In a container, an existing user dir counts as residue only when all of these hold: this command's saved state names that path but was not saved by a write from this version (the released command saved it without a `written` mark), `chatLanguageModels.json` exists, and the dir holds nothing else but its backup. Such a container (one that got the dead file from the released command) prints the group too and leaves that file alone. A dir the command never targeted (fresh or bind-mounted), one this version already wrote, and one VS Code itself populated (for example `globalStorage`) keep the write, so VS Code running inside a container is unchanged. - The hint for a loopback server in a plain container says to publish the port with the detected engine (`docker run -p 8000:8000 ...`, or `podman run -p ...` under Podman) instead of `ssh -L`. With any of `SSH_*`, `REMOTE_CONTAINERS` or `CODESPACES` set (dev containers and Codespaces are usually Docker-backed too), the forwarding hint is unchanged. Nothing else changes: macOS, Windows, WSL and hosts with a VS Code user dir take the same branch as before. ## Tests - `test_vscode_in_a_plain_container_prints_the_group` (both markers): nothing is written, the group and the `docker run -p` / `podman run -p` hint are printed; a dir holding only a dead `chatLanguageModels.json` and its backup is left alone; once VS Code's own `globalStorage` exists the file is written. - `test_vscode_in_a_container_writes_a_user_dir_it_never_targeted`: a dir holding only `chatLanguageModels.json` with no saved state is written, and re-running after that write (another model) still writes. - `test_vscode_in_a_docker_backed_dev_container_keeps_the_forwarding_hint` (`REMOTE_CONTAINERS`, `CODESPACES`): the `ssh -L` hint stays. All fail with the `start.py` change reverted. `unsloth_cli/tests/test_start_app.py`: 58 passed on this host, and 58 passed inside a real Docker container (`python:3.12-slim`, real `/.dockerenv`), so the existing write tests keep passing when CI itself runs in a container. ## Live check in a real container `python:3.12-slim` under Docker 28.2.2 (real `/.dockerenv`, no env vars set), same test harness, no VS Code user dir: | | main | this PR | |---|---|---| | Output | `Added unsloth/gemma-4-E2B-it-GGUF from Unsloth at http://127.0.0.1:8888 to VS Code.` | `This is a remote session (SSH or a container) with no VS Code user dir here, so nothing was written. ... publish port 8888 to the same port on your machine (for example: docker run -p 8888:8888 ...)`, then the group to add | | `chatLanguageModels.json` written in the container | yes (never read by VS Code) | no | | VS Code tests in the container | 24 passed | 26 passed | --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-10-19 08:44:01 -07:00
Daniel Han	f210636c84	Format the two files #14247 left off the formatter fixed point (#14270) ... ## Summary `tests/test_formatter_fixed_point.py::test_every_tracked_python_file_is_already_formatted` fails in `Repo tests (CPU, rest)` on PRs based on current main (seen on #14229): ``` these tracked files are not a fixed point of the ruff-format-with-kwargs hook, so pre-commit.ci will fail the next PR that edits them ``` That run listed eight files. Most were reformatted on main since then. Running the hook (`scripts/run_ruff_format.py`, ruff 0.6.9 as pinned) over every tracked file inside the hook's scope, at main `6c159816c5`, leaves two: - `studio/backend/routes/settings.py`: one `logger.info(...)` call the hook joins onto one line. - `studio/backend/tests/test_video_cpu_developer_mode.py`: one `from utils.account_context import ...` the hook splits. Both came in with #14247, which merged without its pre-commit.ci format commit. ## Change The hook's own output for those two files, nothing else. No behavior change. ## Testing - **The hook over every tracked `.py`:** I ran it and then reverted everything the hook's `exclude` pattern skips (read from `.pre-commit-config.yaml` the same way the guard does). Exactly these two files remain changed. - **Running it again on this branch:** no further changes.	2026-10-19 08:43:23 -07:00
Daniel Han	f7ec00f5f7	Studio: timestamp every other sampled video frame for GGUF chat (#14254) ... Fixes #9962 When a video is attached to a GGUF chat model, llama-server samples it at the rate Studio sets (1 fps) but only writes a text timestamp into the prompt every 5 seconds. The model has to guess the time of the frames in between, and in practice it snaps every event to the nearest 5 second mark. That is the wrong timestamps reported in #9962 when captioning a clip. ## Change Studio now passes `--video-timestamp-interval` alongside `--video-fps`, set to two sampled frames (2000 ms at the default 1 fps). The interval follows the frame rate the user actually asked for, so `--video-fps 2` in the advanced llama-server arguments (or `LLAMA_ARG_VIDEO_FPS`) gives frames 0.5 s apart with a timestamp every second. That is the existing way to control frame spacing, and it now produces matching timestamps instead of a fixed 5 s grid. - A `--video-timestamp-interval` typed in the advanced arguments, or `LLAMA_ARG_VIDEO_TIMESTAMP_INTERVAL` in the environment, still wins and Studio adds nothing. - Builds that do not list the flag in `--help` are not sent it (same capability probe as `--video-fps`). - No UI change. Image, audio and text-only chats are unaffected; the flag only changes video prompts. Why two frames rather than every frame: llama.cpp writes each timestamp after its frame, and Qwen-VL models merge consecutive frame pairs into one temporal patch. A timestamp between every frame splits every pair, which raised Qwen3-VL prompt tokens by 1.7x. At two frames the pairs stay whole (prompt is slightly smaller than today), and Gemma 4 placed events most accurately at this spacing too. ## Results Synthetic 20 s, 640x360 clips: a gray screen that turns red for 2 s starting at a known second (14 clips, starts at 2, 3, 4, 6, 7, 8, 9, 11, 12, 13, 14, 16, 17, 18 s). Prompt: "At what time, in seconds from the start of the video, does the screen first turn red?" Temperature 0, `--video-fps 1`, llama.cpp prebuilt b11719, Q4_K_M with F16 projector. | Model | Timestamp every | Mean abs error (s) | Within 1 s | Prompt tokens | | --- | --- | ---: | ---: | ---: | | Gemma 4 E2B | 5 s (before) | 4.21 | 5/14 | 2197 | | Gemma 4 E2B | 2 frames (this PR) | 0.43 | 14/14 | 2254 | | Gemma 4 E2B | every frame | 1.00 | 14/14 | 2349 | | Gemma 4 E4B | 5 s (before) | 2.14 | 6/14 | 2197 | | Gemma 4 E4B | 2 frames (this PR) | 0.43 | 14/14 | 2254 | | Gemma 4 E4B | every frame | 0.86 | 14/14 | 2349 | | Qwen3-VL 2B | 5 s (before) | 1.86 | 6/14 | 2740 | | Qwen3-VL 2B | 2 frames (this PR) | 1.14 | 12/14 | 2575 | | Qwen3-VL 2B | every frame | 0.93 | 14/14 | 4668 | | Qwen3-VL 8B | 5 s (before) | 1.71 | 7/14 | 2740 | | Qwen3-VL 8B | 2 frames (this PR) | 0.71 | 12/14 | 2575 | | Qwen3-VL 8B | every frame | 0.79 | 14/14 | 4668 | Before the change, Gemma 4 E2B answered "5" for every clip starting at 2 through 8 s, and the Qwen models answered 5 or 10 for most clips. With no timestamps at all, Gemma answered numbers like 1000 and 1600, so the labels are what the model reads time from. For reference, typing `--video-fps 2 --video-timestamp-interval 500` in the advanced arguments brought Gemma 4 E4B to a mean error of 0.21 s (14/14 within 1 s) at 4659 prompt tokens, so users who need finer timing can still trade prompt length for it. ## Verification - `tests/test_llama_video_frame_budget.py`: 8 new tests (default interval, interval follows a typed or inherited frame rate, typed or inherited interval left alone, builds without the flag, labels land on every other frame, capability probe). 40 passed; 5 of the new tests fail with the `llama_cpp.py` change reverted. - Launch-related suites (`test_server_tuning_flags`, `test_llama_cpp_placement`, `test_load_mode_fit_matrix`, `test_metal_explicit_context_guard`, `test_model_memory_settings`, `test_llama_flag_catalog`, `test_video_chat_completions_behaviour`): 2852 passed, 32 skipped. - A text-only GGUF launched with `--video-fps 1 --video-timestamp-interval 1000` loads and answers normally.	2026-10-19 08:42:22 -07:00
Daniel Han	bf22118c98	Tests: make image-engine-choice and image-mask-from-text typecheck (#14272) ... ## Summary `Frontend build + bundle sanity` fails its `Typecheck` step (`npm run typecheck`, tests included) on PRs based on current main (seen on #14245): ``` tests/image-engine-choice.test.ts(305,48): error TS2339: Property 'baseRepo' does not exist on type '{ repoId: string; kind?: ...; filename?: ...; } | { repoId: string; kind: "gguf"; filename: string; displayRepoId?: string; baseRepo?: string; }'. tests/image-mask-from-text.test.ts(246,30): error TS2345: Argument of type '"drawImage"' is not assignable to parameter of type 'never'. tests/image-mask-from-text.test.ts(246,61): error TS2345: Argument of type '"putImageData"' is not assignable to parameter of type 'never'. ``` Reproduced on main `6c159816c5` with `node scripts/typecheck.mjs --tests` (TypeScript 7.0.2, as CI). The product code typechecks. Both errors are in tests: - **`image-engine-choice.test.ts` (from #14232):** `nativeReloadTarget(null, gguf)?.baseRepo`. With `null` as the last load, `T` falls back to its constraint, which has no `baseRepo`, so one branch of the return union lacks the property. - **`image-mask-from-text.test.ts` (from #14235):** `assert.deepEqual(calls, [])` from `node:assert/strict` is an assertion signature. It narrows `calls` to `never[]`, so the later `calls.includes("drawImage")` no longer typechecks. ## Change Test-only: - **`image-engine-choice.test.ts`:** `nativeReloadTarget<{ repoId: string; baseRepo?: string }>(null, gguf)?.baseRepo`. This names `T`, so the status-built branch, the one that carries `baseRepo`, is what is read. The assertion is unchanged. - **`image-mask-from-text.test.ts`:** `assert.equal(calls.length, 0)` in place of `assert.deepEqual(calls, [])`. It asserts the same thing without the narrowing, with a comment saying why. ## Testing - **`node scripts/typecheck.mjs --tests`:** clean (exit 0), from the three errors above on main. - **`node --experimental-strip-types --test` on both files:** 30 passed.	2026-10-19 08:37:01 -07:00
Daniel Han	6c159816c5	Tests: echo gguf_filename in the image text encoder Playwright status stub (#14269) ... ## Summary `Chat UI Tests (extra)` fails on main at `c708b1d727` (run 39268175533) and on every PR based on it, in the `Image text encoder precision regression (Playwright)` step: ``` File "tests/studio/playwright_image_text_encoder.py", line 233, in main with page.expect_request( playwright._impl._errors.TimeoutError: Timeout 30000ms exceeded while waiting for event "request" ``` Every later step in that job is skipped (25 steps), so the shard currently checks nothing past this point. ## Cause #14232 (per-load engine choice for GGUF picks) changed what Reapply reloads. `reapplyTarget` in `features/images/lib/engine-choice.ts` uses the page's last pick only while it still matches the resident model: same repo, kind and `gguf_filename`. Otherwise it falls back to the checkpoint that status reports, which for a GGUF needs a `gguf_filename`. The test's stub `/api/inference/images/status` reported `model_kind: "gguf"` with no `gguf_filename`. The real `status()` always carries it (`diffusion.py`, `"gguf_filename": state.gguf_filename`). So in the test, Reapply had no target and never sent `/api/inference/images/load`. The product behaves correctly against the real backend, and the stub had drifted from the contract. Bisected locally against a stubbed build (vite preview of `studio/frontend`, same script, same chromium): | commit | result | |---|---| | `f5e91f1ca5` (before #14232) | Passed | | `6ccf0e4e74` (#14232) | TimeoutError at line 233 | | main `4ea4c758dc` | TimeoutError at line 233 | ## Change Test-only, `tests/studio/playwright_image_text_encoder.py`: - **Stub status:** it now echoes the loaded file, `gguf_filename = loads[-1].get("gguf_filename")`, as the backend's status does. - **New check:** a GGUF load request without `gguf_filename` is recorded as a page error, and the script already asserts there are none. It records rather than raises inside the route handler, which would leave the request hanging. ## Testing Same local setup, all three variants the workflow runs: - `applied`: Passed: planned and pinned precision, four explicit modes, fallback reseeding and default omission - `PW_COMPLETION_FIRST=1`: Passed (same) - `PW_DECLINE=1`: Passed: a declined encoder precision reseeds to Dense (bf16) Not covered here: the 25 steps after this one that the failure skipped on main. This PR's `Chat UI Tests (extra)` run is the first chance they run again since #14232, and several need the real Studio backend.	2026-10-19 08:06:54 -07:00
Daniel Han	21c0205c6c	Tests: a grandchild reaped while its /proc stat is read counts as gone (#14268) ... ## Summary `studio/backend/tests/test_kernel_install.py::test_a_timed_out_build_stops_its_whole_process_tree` flakes in the `(Python 3.13, a-k)` shard (seen on #14240): ``` tests/test_kernel_install.py:1041: in test_a_timed_out_build_stops_its_whole_process_tree if stat.read().split()[2] == "Z": E ProcessLookupError: [Errno 3] No such process ``` ## Cause The test polls the killed grandchild until it is gone. That means `kill(pid, 0)` fails, or `/proc/<pid>/stat` shows a zombie, or that file is missing. The grandchild can be reaped between opening `/proc/<pid>/stat` and reading it. On Linux that read raises `ProcessLookupError` (ESRCH), not `FileNotFoundError`. So the test failed exactly when the process had already gone, which is the outcome it was waiting for. The product (`kernel_install.run_process_tree`) is fine. The poll came in with #13971. Reproduced directly: open `/proc/<pid>/stat` of a live child, kill and reap it, then read. The read raises `ProcessLookupError: [Errno 3] No such process`. ## Change Test-only: - **`_process_gone(pid)`:** the same three checks, with `(FileNotFoundError, ProcessLookupError)` both counted as gone. - **`test_a_process_reaped_while_its_stat_is_read_counts_as_gone`:** makes the race deterministic. `open` of the child's stat file kills and reaps the child right after opening, and `kill(pid, 0)` reports it alive, as at the start of the race. It is skipped where there is no procfs. ## Testing - **`studio/backend/tests/test_kernel_install.py`:** 119 passed. - **Sabotage:** with the helper back to catching only `FileNotFoundError`, the new test fails with `ProcessLookupError: [Errno 3] No such process`. That is the CI failure.	2026-10-19 08:06:50 -07:00
Leo Borcherding	7790e1b308	Unsloth Studio / Desktop: name chat exports after model, title and day, and reopen the last save folder (#12586) ... Closes #10103 ## What changes 1. **Export name.** A single-chat export (Markdown, JSONL, CSV, ShareGPT) is now saved as `<model> - <title> (YYYY-MM-DD)` instead of `conversation-<ISO timestamp>`. Whichever part the thread lacks is dropped, characters a filesystem refuses become `_`, invisible bidi controls become `_`, and the name is capped at 200 UTF-8 bytes on a character boundary so it stays under the 255-byte file name limit with its date and extension. A chat on an external provider is named after that provider's model (`gpt-4o`), not its internal id, and a chat on a local or cached model after the model (`Qwen3-4B-Q4_K_M`), not the folder path it loads from (a Windows path has no `/`, so a plain split would put `C:\Users\<name>\...` in the file name). Example: `Gemma-4-26B-A4B - Biology of Stag Beetles Explained (2026-09-01).md`. 2. **Last save folder (Unsloth Desktop).** The native save dialog was never given a starting folder. Windows and macOS fall back to their own per-app memory, but GTK on Linux does not, so every save started from the default location. The folder of the last successful save is now kept in the app config dir (`last-save-dir.txt`) and passed to the dialog. If that folder is gone, the OS default is used. It's one folder for every native save, the same as most desktop apps. Not in this PR: the `[AI Conversation]` tag and the `001`/`002` counter from the issue (the native dialog already asks before overwriting), and metadata inside the markdown. That last one touches the `unsloth-chat-v1` frame import depends on, so it gets its own change. ## Tests - `conversationExportBasename`: full name, missing model or title, separators, control characters and trailing dots, external provider ids, and Windows and Hugging Face cache paths. - Rust `the_last_save_folder_is_remembered_until_it_disappears`. - Frontend `npm test`: the 18 export tests pass. `typecheck` is clean. Rust `native_file_dialogs` tests: 32/32. Not yet run: driving the save dialog by hand in Unsloth Desktop on Linux, which is the case from the report. ## Before / after Same chat (model `unsloth/Gemma-4-26B-A4B`, title "Biology of Stag Beetles Explained") exported to Markdown from the composer's + > More > Export chat menu, on the merge base and on this branch. The caption at the top is drawn by the test harness to show the browser download's file name, which Studio itself never paints.  Before: `conversation-2026-10-08T08-17-35.md`. After: `Gemma-4-26B-A4B - Biology of Stag Beetles Explained (2026-10-08).md`. ### Linux save dialog (GTK3) The same GTK3 save dialog Studio's dialog plugin builds (`rfd` 0.16, `gtk3` backend), driven under Xvfb. One process saves into `~/Documents/AI-Chats/Beetles`, then opens the dialog again. Before: no starting folder is passed, and GTK opens in the home directory, so the second save lands at `~/Gemma-4-26B-A4B - ... .md`. After: the folder of the last save is passed, and the dialog reopens in `Beetles`, where the second save lands.  <!-- studio-regress:start --> ### Studio UI regression check Base `d8ad0b080` vs head `7b0fff498`. Functional: **NO_REGRESSION**, UI coverage 0%, 3 visual change(s), 0 text-only change(s). Only steps whose screenshots differ are shown. **desktop/s02_backend_ready**  **desktop/s06_route_studio**  **desktop/s07_route_export**  <!-- studio-regress:end --> The three Desktop pairs above are startup timing, not a change from this PR: this PR does not touch the splash screen, routing or backend start. Across three staging runs of the same head, `s06_route_studio` read `/studio` before and `/chat` after in one run and `/chat` on both sides in another, and `s02_backend_ready` passed on the head in two of three runs. The app's code changes here run only inside the save commands. --------- Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-10-19 08:01:00 -07:00
Nilay	4ea4c758dc	Studio: eject a model with a running chat from the Loaded models card (#14161) ... Clicking the minus on a chat model in the bottom-left Loaded models card failed whenever that model still had a chat generating, including one that was stuck. The card showed "Unloading this model would stop 1 chat that is still generating. Stop them first, or retry with force_cancel_active." and offered no way to retry. A chat started from an API client or another tab has no Stop button in this tab, so restarting Studio was the only way out from the card. The card now asks the same "Stop 1 running chat?" question the model picker already asks. "Stop and unload" stops that model's chats and unloads it. "Keep generating" leaves the model loaded and the chat running. A model with nothing generating still ejects straight away with no question. Fixes #14154 ## Before and after Studio built from the merge base (left) and from this branch (right), `unsloth/Qwen3-0.6B-GGUF` Q4_K_M loaded, one chat streaming from an API client so this tab has no Stop button, then the minus on the Loaded models row clicked. Clicking eject: the card used to surface the 409; it now asks the same "Stop 1 running chat?" question the model picker asks.  After the click settles: before, the model is still loaded; after "Stop and unload", the model is ejected and the card is empty (`/api/inference/status` reports nothing loaded).  --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>	2026-10-19 07:33:29 -07:00
Daniel Han	28a222f20c	Studio: let an approved rm run, never a recursive one (#14238) ... Fixes https://github.com/unslothai/unsloth/issues/9972 ## Problem In Ask and Approve for me, a terminal `rm` shows the approval prompt, but clicking Allow still returns `Blocked command(s) for safety: rm` and nothing is deleted. `rm` is both on the high risk list (which prompts) and on the hard blocklist, and only Full access skips the blocklist. The prompt could not change the outcome, so models fall back to Python (`os.remove`), which asks the same question and does delete. ## Change When the user answers Allow on a terminal call that is one plain `rm` naming its files, the `rm` block is lifted for that call. The check is an allowlist, not a scan for recursive flags: - The command word is exactly `rm`; options are only `-f`, `-v`, `-d` (in any cluster), `--force`, `--verbose`, `--dir`, `--interactive=never`; `--` ends options. Prompting modes (`-i`, `-I`, `--interactive`) are refused: stdin is closed, so rm would delete nothing and still exit 0. A trailing newline is fine. - Any shell syntax keeps the block: pipes, chains, redirects, `$`, backticks, braces, globs, backslashes, newlines. A glob or `xargs` can hand rm a file named `-rf`, and a line continuation can split a flag, so none of those are approvable. - Recursive deletion (`-r`, `-R`, `-rf`, `--recursive`, any quoting of them) is never approvable. The tool tells the model to name each file. - Only `rm` is lifted. `rm a && sudo b` stays blocked. - On the Windows cmd terminal profile an approved `rm` stays blocked: the check reads words the POSIX way and cmd deletes with `del`. - Off mode never prompts, so `rm` stays blocked there. Full access is unchanged. ## Tests `tests/test_approved_rm.py`: 9 plain forms approvable, 34 recursive, prompting, globbed, piped, expanded, escaped or substituted forms not, and end to end through `_bash_exec`: unapproved `rm` blocked with the file kept, approved `rm` deletes, approved `rm -rf` / `rm -r` / `rm --recursive` / `rm *` / `ls | xargs rm` refused with the tree kept even when a file named `-rf` sits beside it. The approved delete test fails on main (file still exists). Existing suites pass unchanged: `test_permission_mode.py` (2431), `test_sandbox_tools.py` (991), `test_bypass_permissions.py`, `test_tool_sandbox_wiring.py`, `test_windows_bash_shell.py`, `test_windows_cmd_terminal_profile.py`.	2026-10-19 07:25:52 -07:00
Daniel Han	9f78f5e580	MFU logging under fullgraph, NCCL logs under distributed debug, token-budget metrics across ranks (#14207) ... ## Summary Three follow-ups on opt-in features. Default runs are unchanged. 1. **MFU logging (#13885).** For packed batches that carry only `position_ids`, the token count branched on data, which broke `fullgraph = True` compilation and added a host sync per micro-batch. It is now computed without a data-dependent branch. Also adds an H200 NVL entry at 835.5 dense TFLOPS (half of NVIDIA's 1,671 with-sparsity figure); it was being measured against the H200 SXM peak. 2. **NCCL INFO logs under `UNSLOTH_DEBUG_DISTRIBUTED` (#13884).** Importing unsloth_zoo sets `NCCL_DEBUG=WARN` whenever a logging or compile-debug switch is on, so the debug flag lost the NCCL INFO logs it asks for. With `UNSLOTH_DEBUG_DISTRIBUTED` set, unsloth now saves `NCCL_DEBUG` before that import and restores it after. 3. **Token-budget split metrics across ranks (#13878).** Logged accuracy and entropy from split chunks were averaged per rank. Each chunk is now weighted by its token count summed across ranks. The decision to merge is folded into the existing chunk-count collective, so every rank merges or none do: no extra collective and no chance of one rank waiting on another. ## Tests | File | main | this PR | |---|---|---| | `tests/test_mfu_logging.py` | 2 failed | pass | | `tests/test_debug_distributed_env.py` | 2 failed (real zoo import) | pass | | `tests/test_token_budget_metrics_distributed.py` (2 gloo ranks) | 3 failed | pass | 53 passed in total. ## Not covered here - No real multi-rank NCCL run; the distributed tests use gloo. - The H200 NVL device-name string is assumed to contain "H200 NVL". - A `NCCL_DEBUG` you export yourself is still overwritten when the debug flag is off; that needs a change in unsloth_zoo.	2026-10-19 07:21:39 -07:00
Daniel Han	7e3e0d90c0	Decode: cover explicit positions past the lazy RoPE cache in Gemma2, Cohere, Granite and FalconH1 (#14199) ... ## Summary The RoPE cache is built lazily, with only the rows asked for. #13976 fixed Llama's decode loop for explicit `position_ids` past the KV length (for example after left padding or a prefill without `position_ids`). Gemma2, Cohere, Granite and FalconH1 have their own decode loops and still read cos / sin at the KV length, so a position past the cached rows hit a CUDA index-out-of-bounds assert. Each of the four decode loops now computes the rows one step needs once, with a new helper `_decode_rotary_seq_len` in `llama.py`: the larger of the KV length and the highest explicit position + 1. It then passes that value to every layer. Attention only extends the cache and reads the larger table when the position is past the KV length; otherwise it makes exactly the same calls as before. Cost: one `.item()` host sync per decode step, the same one Llama's loop already pays. It is skipped entirely (zero syncs) when the cache built at load already covers `max_position_embeddings`, which holds for any model with 32768 or fewer positions, such as Gemma2 (8192). Nothing is computed per layer. ## Tests `tests/test_decode_offset_rope_model_loops.py` uses tiny models with `max_position_embeddings = 40960`. It prefills 4 tokens without `position_ids`, then decodes at position 35000. | | Gemma2 | Cohere | Granite | FalconH1 | sync helper | |---|---|---|---|---|---| | main | index out of bounds | index out of bounds | index out of bounds | index out of bounds | fails | | this PR | pass | pass | pass | pass | pass | For normal decoding, logits are bitwise equal to main: - Gemma2: checked through `generate` at batch 1 and 2, on both the one-sync and the zero-sync path. - Cohere, Granite, FalconH1: checked with a direct decode loop. `tests/test_rope_cache_current_device.py` needs a one line update for the new argument; all 23 tests in the two files pass on GPU. ## Not covered here - In practice the crash reaches users only through Gemma2 configs with more than 32768 positions. `FastLanguageModel` does not route Cohere, Granite or FalconH1 to these decode loops, so they are only reached through their patchers directly. - No T4, ROCm, 4-bit or torch.compile graph-break run; torch 2.13 / transformers 5.17 only (FalconH1 also on 4.57.6).	2026-10-19 07:21:15 -07:00
Daniel Han	c708b1d727	Studio: a kept load short on its own GPU no longer unloads models on other GPUs or on the CPU (#14234) ... ## Summary With **Keep multiple models loaded** set to **When they fit**, a model pinned to one GPU that does not fully fit there unloads models that cannot make room for it: 1. Idle CPU-only kept models (a GGUF started with no GPU layers, for example a small LLM or TTS held in RAM) are picked as eviction victims, one per retry, though unloading them frees no VRAM on a discrete card. 2. When no kept model on the short GPUs can go, the active model is replaced even when it is pinned to other GPUs, or runs on the CPU. That frees nothing on the GPUs the new model is pinned to, so the new model lands on the same GPUs with the same `--fit on` offload it would have had anyway, and the user just loses the other models. The Model memory hint already promises the opposite: "When they fit and Always never unload a model on another GPU." This is the setup in #9257 (a big GPU model plus small CPU-only models in one Studio) and the "8 GPUs, two models split 4 + 4, sometimes it unloads the other model" report on #8488. ## Changes - `GpuMemoryShortError` gains `discrete`: true only when none of the short GPUs shares host RAM. It is false for a probe total of 0 (APU, iGPU), for the probe's own ROCm-unified and integrated-CUDA classifiers (DGX Spark, Jetson), for the Metal shortfall, and for a non-GGUF out-of-memory error. - `model_slots.eviction_victims(..., discrete)`: on a discrete shortfall, a kept llama-server with no VRAM plan and no GPU layers is not a victim. A GPU server that fell back to the CPU still has its plan and is evicted as before. - `load_model_gated`: when nothing kept can be evicted, the active model is kept and the load retries beside it with what fits (`force_alongside`) only when the active model's placement is one its child cannot leave and is disjoint from the short GPUs: an explicit `gpu_ids` pin (the recorded device mask, already widened for companion devices) or a CPU-only server. An unpinned, safetensors or otherwise unknown placement is replaced as before. Always and Off are unchanged. Without `gpu_ids`, the short GPUs include the active model's, so single-pool users keep today's replace. ## Testing Unit tests in `studio/backend/tests/test_multi_model_serving.py`: - `test_a_cpu_only_kept_model_is_never_evicted_for_gpu_room`, which also covers a shared pool and a GPU server that fell back to CPU. - `test_a_model_short_on_its_own_gpu_keeps_the_models_on_others`, which also covers a pinned `gpu_layers=0` active model. - `test_a_model_short_where_the_active_model_runs_still_replaces_it`. - `test_a_short_load_still_replaces_an_active_model_it_may_share_with`, parametrized over a shared pool, an unpinned active model, and a companion-widened pin. The Metal guard test now also asserts `discrete` is false. The multi-model, GPU arbiter, residency, Metal, training VRAM and shortfall suites pass: 1339 passed. The 3 failures in `test_ctx_checkpoint_host_budget.py` fail identically on the merge base. End to end on a Kaggle T4x2: one `install.sh --local` at merge base `3697c7d62`, then the same install with this patch applied, same models and steps: - Qwen3-0.6B Q4_K_M with `gpu_ids=[0]`, gemma-3-270m with `gpu_ids=[1]` kept alongside, and SmolLM2-135M kept alongside with `gpu_memory_mode="manual", gpu_layers=0`. - Each llama-server ran where asked (`CUDA_VISIBLE_DEVICES=0`, `1`, `-1`) and answered the requests naming it. Unloading gemma left the other two serving. This was the same on both sides. - GPU 1 was then filled to about 1 GB free, and Qwen3-1.7B was loaded alongside with `gpu_ids=[1]` under When they fit: | | merge base | this PR | |---|---|---| | serving before | Qwen3-0.6B (GPU 0), SmolLM2 (CPU) | Qwen3-0.6B (GPU 0), SmolLM2 (CPU) | | serving after | Qwen3-1.7B only | Qwen3-0.6B, SmolLM2, Qwen3-1.7B | | Qwen3-1.7B placement | GPU 1, `--fit on`, 518 MiB | GPU 1, `--fit on`, 518 MiB | Under Always, both sides refuse the same load with a 409 and unload nothing. Fixes #9257	2026-10-19 07:21:07 -07:00
Daniel Han	40e20017af	Context parallel: wrap trainer methods only once a trainer enables CP (torch 2.10 step time) (#14152) ... ## Summary `patch_sft_trainer()` (context parallelism, #4257) runs at `import unsloth` and replaces `train`, `training_step`, `prediction_step`, `evaluate`, `predict`, `evaluation_loop`, `save_model` and `get_total_train_batch_size` on `trl.SFTTrainer` with wrappers that pass straight through when `context_parallel_size = 1`. Even as pass-throughs they made LoRA training steps much slower on torch 2.10 with sm80+ GPUs (L4 / A100): gemma-4-E2B steady step went from ~300 ms to ~425 ms. Torch 2.14, T4 and torch 2.11 to 2.13 were not affected. Bisecting on Colab L4 / A100 with torch 2.10: - Reverting only the `patch_sft_trainer()` call restores base speed. - Keeping any single wrapper (`__init__`, `train`, `training_step`, `prediction_step`, `evaluation_loop`, or the four evaluation wrappers together) is as fast as base. - Only the full set is slow. The profile shows the extra time as CPU self time in AOTAutograd's `CompiledFunction` (413 ms vs 110 ms over 4 steps, same call count). This change keeps the `__init__` wrapper, which runs the CP option checks and installs the accelerator state. It now installs the other wrappers on the class only once a trainer actually enables context parallelism (`context_parallel_size > 1`, after every gate passes). Trainers with CP off run the original methods. CP behaviour is unchanged: the wrappers are the same functions, installed before `train()` can run. `patch_sft_trainer(install_cp_methods = True)` installs them up front, which the unit tests that call the wrappers directly use. ## Results (Colab L4, LoRA SFT, 15 steps, steady step ms) | arm | gemma-4-E2B torch 2.10 | Qwen3-0.6B torch 2.10 | gemma-4-E2B torch 2.14 | Qwen3-0.6B torch 2.14 | |---|---|---|---|---| | base (before #4257) | 297.0 | 304.2 | 254.8 | 285.3 | | main | 428.5 | 310.2 | 257.7 | 285.7 | | this PR | 303.2 | 304.5 | 255.2 | 285.5 | | this PR (repeat, last arm) | 305.2 | 302.0 | 259.6 | 292.4 | | main (repeat, last arm) | 421.6 | 313.8 | 255.2 | 287.5 | Peak memory is the same in every arm. The losses for the 6 logged steps match base exactly (torch 2.14: one identical loss trace across all arms), and there are 0 graph breaks in all arms. ## Tests - `tests/test_context_parallel.py`, `tests/test_ulysses.py`, `tests/test_ulysses_hybrid.py`, `tests/test_context_parallel_fsdp2_gloo.py`, `tests/test_context_parallel_fsdp2_save_gloo.py` pass on CPU / gloo. One test fails on this CPU-only host, `test_unsloth_training_arguments_keep_the_cp_options` (no bf16 on CPU), and it fails identically on main. - New `test_cp_methods_are_wrapped_only_once_a_trainer_enables_cp`: with CP off only `__init__` is wrapped, even after constructing a trainer. A trainer that enables CP installs the wrappers around the original methods.	2026-10-19 07:20:58 -07:00
Daniel Han	2c7c841f69	Studio: show the end of long prompts in the API monitor (#14241) ... Fixes https://github.com/unslothai/unsloth/issues/9898 (the API log part). ### Problem The request detail on the API page cuts the prompt to its first 12,000 characters. Agent clients (Roo Code, Cline, Kilo, opencode and so on) send a long system prompt first and the turn being answered last, and the monitor joins the messages in order (`system: ...`, then `user: ...`). So for almost every agent request the box showed only the agent's instructions and ended in `...` well before the user message. Copy prompt has copied the full text since #11340, but the visible part was still the least useful part. ### Change - The preview keeps the first 6,000 and the last 6,000 characters with a marker between them, for example `[... 48,213 characters omitted ...]`. SGLang's request log truncates the same way (`data[:half] + "..." + data[-half:]` in `request_logger.py`), so the latest turn and tool results are always on screen. - A `Show all` toggle in the block header (where `preview only` used to be) renders the full body, for bodies up to 1,000,000 characters. Bigger bodies keep the preview and Copy, so a huge prompt cannot stall the page. The toggle resets when a different request is selected. - Neither cut splits a surrogate pair. Nothing else moves: short prompts render exactly as before, `preview only` still shows when only the server's 360 character preview is loaded, Copy still copies the full body, and the backend is untouched. The same block also renders Reply and Error, which the backend already keeps under 12,000 characters, so they look the same as today. ### Performance `payloadPreview` is two slices and two anchored regex tests, so its cost does not grow with the prompt: about 1 to 4 microseconds a call for 12 K to 1 M character prompts, and under 1 microsecond for 64 M in Node 24. The rendered text stays about 12,000 characters, the same as before, unless the user asks for all of it. ### Tests - New `tests/api-monitor-payload-preview.test.ts`: short bodies unchanged, an agent shaped prompt keeps its system head and its final user turn, and no lone surrogate at either cut. - `npm test` (12,702 passed, 1 skipped), `npm run typecheck`, `npx tsc -b` and `npm run i18n:check:strict` pass. ESLint on the touched page reports the same findings as on main. ### Before and after The Prompt box for the same 63,100 character agent prompt, scrolled to the bottom, merge base `cefb057a4` on the left and this branch on the right, in two separate Studio installs (Chromium). The list and detail endpoints served the same payload produced by the backend `ApiMonitor` to both sides. Before, the box stops partway through the system prompt (`verbat...`) and the user turn never appears. After, the end of the prompt, including the user turn, is on screen. Show all renders all 63,100 characters exactly, then flips to Show less.  --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-10-19 07:18:03 -07:00
Daniel Han	527854709d	Studio installer: show the previous unsloth version next to the new one (#14248) ... Fixes https://github.com/unslothai/unsloth/issues/9910 When you re-run the Studio installer over an existing install, it never says which version you had. The only version line is `unsloth <version> installed` at the very end, so you cannot tell whether a run actually upgraded anything. This PR has both installers (`install.sh` and `install.ps1`) record the previous version just before the existing environment is moved aside for rollback, and report it in two places: ``` venv ... found unsloth 2026.9.3 preserving existing environment for rollback... ... unsloth 2026.10.7 installed (was 2026.9.3) ``` A re-run that lands on the same version prints `2026.10.7 installed (unchanged)`. A fresh install prints exactly what it prints today. ## Why not print the new version up front The installers request `unsloth>=<floor>` and uv picks the actual release partway through, so the real version is not known before installing. Printing the floor as the "new version" (as #9914 did) would be wrong on almost every upgrade. Printing the old version up front and the old/new pair once uv has resolved is accurate and needs no network lookup, which matches how rustup and pip report upgrades. ## How the previous version is read It is read from `<site-packages>/unsloth-*.dist-info/METADATA` on disk, without starting the old interpreter, the same way the installers already read the previous torch version from `torch/version.py`. A broken or wedged old environment therefore cannot stall or fail the install. If there is no record, or more than one, nothing extra is printed and the final line is unchanged. ## Scope - Output only. Nothing that installs, rolls back, or picks torch is touched, and the Desktop app's `[TAURI:*]` progress markers are unchanged. - Read-only and idempotent: every run reads the environment it is about to replace, so a second run reports the same thing. - Only the normal re-run path (an existing `unsloth_studio` environment) reports a previous version. The rarer legacy `.venv` migration keeps today's output. ## Tests `tests/test_installer_unsloth_version.py` adds POSIX `sh` and PowerShell cases for the disk reader (single record, CRLF metadata, `unsloth_zoo` not mistaken for `unsloth`, duplicate records, no record) and for the final line (no previous version, a different version, the same version). On the base installers 14 of the 18 tests fail; with this PR all 18 pass. I also ran the disk reader against a real uv-installed `unsloth==2026.9.1` environment, where it returned `2026.9.1`, matching `importlib.metadata`. The existing installer suites that read the edited blocks pass unchanged: `test_cross_platform_parity.py`, the Windows on ARM tests, `test_studio_install_workspace_guard.py`, `test_installer_profile_hardening.py` (1115 passed), plus `tests/sh/test_previous_torch_pin.sh`, `test_cu130_torch213_route.sh`, `test_install_python_guard.sh`, `test_mac_intel_compat.sh`, `test_torch_probe_isolation.sh`, `test_unsloth_torch_override.sh` and the two matching `.ps1` suites.	2026-10-19 07:17:03 -07:00
Daniel Han	bff0a25b4b	Studio: make inpaint and localized-edit masks from text with SAM 3 (#14235) ... Closes #7832. Adds "Mask from text" to the Images page. Type concepts such as `sofa, red lamp, pillow:2` under the inpaint mask painter (or the localized edit mask and paint modes) and Find fills the mask. The mask stays editable with the brush, and "Add to current mask" unions instead of replacing. ## How it works - `core/inference/diffusion_segment.py` runs the official `facebook/sam3` checkpoint through transformers `Sam3Model`: bf16 on the image model's card, CPU while training runs or the card has no room, CPU retry on OOM. - Settings follow ComfyUI's SAM3 Detect node and its Image Segmentation blueprint: comma-separated concepts, `:N` keeps up to N of a concept (default 1), 1008x1008 bilinear input, threshold 0.5, union of the kept masks at the source size. No dilation or feather, as in the blueprint. - The score multiplies in SAM 3's presence head, as Meta's and transformers' post-processing do. Without it a concept missing from the picture masked something else (an "elephant" in a living room masked the sofa at 0.72; 8 of 18 absent concepts produced a mask). With it all 8 are empty and no kept detection of a present object changes. - The segmenter is auxiliary, like the FLUX.3 prompt upsampler: a 120 s keep-alive, unloaded at once when the card is tight, and released by the GPU arbiter on any load or eviction and when training starts. Mask requests are refused (409) while an image renders or loads. - Routes: `POST /api/inference/images/segment`, plus `segment/status`, `segment/download` (gated repo, pinned revision, license hint on 401/403; owner only), `segment/cancel` and `segment/release`. ## Why SAM 3 and not SAM 3.1 SAM 3.1 is published only as an original-format multiplex `.pt` checkpoint with no transformers weights. Converted with transformers' own converter (which reproduces SAM 3's published `model.safetensors` exactly from its `.pt`), the 3.1 detector gives the same image masks as SAM 3: IoU of at least 0.993 on 15 prompts, scores within 0.008, identical kept counts. Its gains are in video tracking. The detector tensors differ only slightly (max abs 3.3e-3). ## Testing - Backend: 43 new tests with a tiny random SAM 3 running the real forward on CPU (prompt parsing, keep rule, mask size, keep-alive and release, arbiter hooks, render slot, CPU retry on OOM, route contract). With the arbiter and upsampler suites: 242 passed. - Frontend: 9 new tests; the image suites pass (421/421) and `tsc` is clean. - End to end on a real Studio server with `facebook/sam3` and Z-Image-Turbo on one B200: - "sofa, red lamp, pillow:2" found 4 objects at 0.97 to 0.98; "elephant" returned an empty mask. - First mask 17 to 27 s (load plus warm-up), then 70 to 180 ms per mask; 2.0 GB peak allocated; the release route freed 2.3 GB. - Loading Z-Image-Turbo released the segmenter; a mask requested mid-render got 409. - The sofa mask fed a real inpaint: mean pixel change 42.6 inside the mask, 3.0 outside.  Not covered: ComfyUI's optional tracker refine pass (it would load the tracker and roughly double memory), Apple Silicon / ROCm / Windows runs, and a browser run of the panel. ## How it is wired into Studio and Desktop Desktop (`studio/src-tauri`) bundles the same frontend and starts the same backend, so both get it with no platform code. - Frontend: `MaskFromTextPanel` sits under the Inpaint mask painter and under the localized edit painter in its mask and paint modes (not annotate). Find posts the source image and the text to `/api/inference/images/segment`; the returned mask is drawn into the painter's own mask layer (`mask-apply.ts`), replacing it or, with "Add to current mask", adding to it. The brush and Clear keep working on it, and Generate sends it as a painted mask. - Backend: `diffusion_segment.py` loads SAM 3 on the first Find, on the image model's card in bf16 (CPU while training runs or the card is short, MPS fp32 on Apple Silicon). The GPU arbiter releases it on any model load, eviction or training start, and a handover is refused while a GPU call it could not stop is still running. Closing the painter calls `/segment/release`. - First use: the panel shows a download notice for `facebook/sam3` (gated: the user's Hugging Face token must have accepted Meta's license; the error names the license page). `UNSLOTH_SAM3_PATH` points at a local copy instead. - Installs that cannot run it (Python below 3.10 with transformers 4.57, or `--no-torch`) show the panel disabled with the reason. ## Screenshots Inpaint with Z-Image-Turbo loaded, the same photo on both sides. BEFORE has no way to build the mask from text. AFTER: "sofa" then "pillow:2" with "Add to current mask" (mask coverage 25.4% then 29.7% of the image, segmenter status `facebook/sam3` supported and downloaded):  The flow end to end on the AFTER build: source, "sofa", Find, "pillow:2" added, then a real inpaint from the sofa mask (strength 0.6) that repaints only the sofa:  --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-10-19 07:08:11 -07:00
Daniel Han	80e2140071	Studio: raise the startup bundle budget for #11937 (#14264) ... ## Summary Main is over the startup bundle budget again, by 189 B raw, so `Frontend build + bundle sanity` fails on every PR based on it: ``` over the startup budget: raw 6682.2 KB > 6682.0 KB (main at 6ccf0e4e74) ``` #14258 set both limits to the measured build plus a 160 B raw / 189 B transfer margin. #11937 (a setting to hide the macOS menu bar icon) then added 349 B raw / 78 B transfer to the startup chunks, and merged without raising them. ## Cause #11937 adds strings to `i18n/locales/en.ts`, which ships in the startup chunks, and entries to the settings search index (`settings-search.ts`, `dialog-search-index.ts`). That is small real growth. I measured each commit since #14258 on its own: | tree | raw B | transfer B | |---|---|---| | `46e4f50bea` (#14258) | 6,842,183 | 2,060,321 | | `2b20770c91` (+ #11937) | 6,842,532 | 2,060,399 | | `f5e91f1ca5` (+ #14223) | 6,842,532 | 2,060,421 | | main `6ccf0e4e74` (+ #14232) | 6,842,532 | 2,060,400 | #14223 and #14232 add no raw bytes. #14021 and #14263 change no frontend code. ## Change - **`check-bundle-budget.ts`:** a BUDGET comment entry for #11937, and both limits moved to the measured build at main plus the same margin: - `rawBytes`: 6,842,343 -> 6,842,692. - `transferBytes`: 2,060,510 -> 2,060,589. ## Testing One machine and build, Node 24, exact byte totals from the budget script, 96 chunks. - **Main against the old limits:** `over the startup budget: raw 6682.2 KB > 6682.0 KB`. - **With this change:** `within budget (0.2 KB transfer, 0.2 KB raw to spare)`.	2026-10-19 07:00:35 -07:00
Daniel Han	d1343c4a83	Studio: wire in DeepSeek-V4.1-Flash-1017 (deepseek41) (#14260) ... Wires DeepSeek-V4.1-Flash-1017 (llama.cpp arch `deepseek41`, `unsloth/DeepSeek-V4.1-Flash-1017-GGUF`) into Studio. The model runs on the unslothai/llama.cpp prebuilt that pins unslothai/llama.cpp#298. ## What already matched - Sampling: temperature 1.0, top_p 1.0 come from the existing `deepseek-v4` family in `inference_defaults.json` (substring match), for every id shape (repo, `repo:variant`, HF cache snapshot, `.gguf` path). - Reasoning: the GGUF template gates thinking on `enable_thinking` / `thinking_mode` and maps `reasoning_effort` low / medium / high / max to a budget; `detect_reasoning_flags` exposes all four and tool calling. ## What this adds - **K == V cache rule**: llama.cpp rejects mismatched K/V cache types for `LLM_ARCH_DEEPSEEK41` (same condition as `LLM_ARCH_DEEPSEEK4`). `_requires_symmetric_kv` now matches `deepseek41` by name, since like deepseek4 it writes no `kv_lora_rank`. - **Context compute buffer**: deepseek41 has its own sparse-attention graph. Measured CUDA0 compute buffer on the pinned build (UD-IQ1_S, experts + Engram table on CPU, f16 cache): | ctx | ub=512 | ub=2048 | |---|---|---| | 16k | 2102 MiB | | | 64k | 2198 MiB | | | 256k | 3094 MiB | 9751 MiB | | 1M | 11197 MiB | 37143 MiB | The generic estimate gave 16 / 64 / 256 / 1024 MiB (under by up to ~10 GiB), and reusing deepseek4's rate would reserve ~74 GiB at 1M (6.6x over). New rate: 2 GiB flat + 9600 B/token at ub=512 (rate ub-scaled), plus the usual mask copies on a layer split. It sits above every measured point and within 1.2x at 1M. - **Default model lists**: `unsloth/DeepSeek-V4.1-Flash-1017-GGUF` next to `unsloth/DeepSeek-V4-Flash-GGUF`. ## Tests - New `tests/test_deepseek_v41_defaults.py`: sampling defaults per id shape, reasoning effort ladder, K == V, default lists. - `tests/test_compute_buffer.py`: `TestContextBufferDSV41` checks every measured point is covered, the 1M estimate stays within 1.2x, and a layer split adds exactly the mask copies. - `tests/test_mla_kv_cache_symmetry.py`: deepseek41 case. `pytest tests/test_compute_buffer.py tests/test_deepseek_v41_defaults.py tests/test_mla_kv_cache_symmetry.py tests/test_deepseek_v4_thinking_effort.py tests/test_mlx_bnb_substitution.py`: 354 passed, 1 skipped.	2026-10-19 06:59:25 -07:00
Daniel Han	6ccf0e4e74	Studio Images: per-load engine choice for GGUF picks (native sd.cpp for LoRA) (#14232) ... Closes #9295. ## Summary On a GPU host a GGUF image model always loads on diffusers, and a GGUF run as-is there cannot take a LoRA, so Studio hides the LoRA controls. The native sd.cpp engine applies LoRAs to the same GGUF, but a GPU host could only reach it through `UNSLOTH_DIFFUSION_ENGINE`. This adds an engine pick per load. ### Backend - `/images/load` and `/images/download-plan` take `engine`: `auto` (default, unchanged), `diffusers` or `sd_cpp`. It applies to that load only, and `UNSLOTH_DIFFUSION_ENGINE` still wins over it. - A pick the engine cannot serve is refused with a 400 before anything is unloaded, never swapped for the other engine: - native for a non-GGUF kind - native for a family without sd.cpp assets - native with no runnable binary - diffusers without the pipeline class - Switching engines goes through the existing safe unload. - Status adds `engine_forced`, `native_engine_eligible`, `lora_via_native`, and a `resolved.engine` provenance entry (Auto vs explicit, with the reason). ### Frontend - Images Advanced has an Engine select (Auto / Diffusers / Native (sd.cpp)), shown for a loaded GGUF the native engine can run, with the usual provenance badge. - The choice is remembered per checkpoint once a load lands with it, and recalled for that checkpoint's next load. A refused pick is never replayed. - When a GGUF runs as-is on diffusers (no LoRA) and native would apply LoRAs, a note under the prompt offers "Reload with native engine", which reloads the same pick with `engine=sd_cpp`. - The hint covers the tradeoffs: native applies LoRAs and usually needs less VRAM, but has no torch.compile or quantised fast paths and is often slower on CUDA. With precision on Auto a GPU host already loads the full base as dense int8, which bakes LoRAs in at load, so the note only appears when the GGUF runs as-is (precision Off). ## Screenshots FLUX.2 klein 4B GGUF loaded as-is on diffusers. The note under the prompt that offers the native reload:  The Engine select in Advanced, with its provenance badge:  ## Testing - Backend: `test_diffusion_engine_router.py` gains cases for the field, env precedence, refusals before unload, status eligibility and provenance. Router, MLX router, load-contract and routes suites: 579 passed. - Frontend: `tests/image-engine-choice.test.ts` (7 tests); image and diffusion suites 291/291; `tsc --noEmit -p .` and `scripts/typecheck.mjs --tests` clean. - End to end through the real `/images` routes: one CUDA GPU, FLUX.2 klein 4B `Q4_K_M` GGUF, 512x512, 4 steps, seed 7, sd.cpp `master-968` CUDA prebuilt, LoRA `adirik/pixel-art-lora-flux.2-klein-4B`. | Load | engine | supports_lora | lora_via_native | resolved.engine | load | gen (2nd) | LoRA gen | |---|---|---|---|---|---|---|---| | Auto, precision Off (GGUF as-is) | diffusers | false | true | auto | 50.6 s | 0.3 s | 400 (unsupported) | | engine=sd_cpp | sd_cpp | true | false | explicit | 12.0 s | 0.9 s | 200, 1.6 s | | Auto again | diffusers | false | true | auto | 37.2 s | | | | Auto, precision Auto (dense int8) | diffusers | true | false | auto | 63.4 s | 0.7 s | 400 (bake at load) | | engine=sd_cpp | sd_cpp | true | false | explicit | 41.5 s | 2.1 s | 200, 3.3 s | - The native LoRA applies: mean absolute pixel difference vs the same seed without it is 28.86 (0-255), and the next generation without the LoRA matches the no-LoRA image exactly. - Refusals: native for a pipeline pick returns 400 "The native sd.cpp engine loads GGUF models only."; an unknown engine returns 422. - Timings are single runs on a shared GPU, not a speed comparison. Not covered: the CPU sd.cpp bundle on a GPU host (one managed sd.cpp install per host, so a CPU pick would replace the CUDA build), and Mac / Windows / ROCm runs. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-10-19 06:49:32 -07:00
Daniel Han	f5e91f1ca5	Studio: delete selected recipe blocks with the Delete key (#14223) ... Fixes https://github.com/unslothai/unsloth/issues/9211 (the "couldn't find a way to remove a node" part). ## Problem The Data Recipes canvas only deletes a selected block with Backspace, which is React Flow's default `deleteKeyCode`. On Windows and most Linux keyboards people press Delete to remove things, and on the Data Recipes canvas Delete does nothing. There is also no visible delete button, so users conclude blocks cannot be removed at all (see the comment on #9211 from a Windows 10 / Edge user). ## Change Pass `deleteKeyCode={["Backspace", "Delete"]}` to the recipe canvas, as a module-level constant so React Flow's key listener is not re-subscribed on every render. - Backspace keeps working exactly as before (macOS users are unaffected). - Delete now removes the selected blocks and edges through the same React Flow path Backspace already uses, so node removal, edge cleanup and the store's removal helpers are unchanged. - Typing in inputs, textareas and editors is unaffected: React Flow ignores the delete key while focus is in an editable element, same as for Backspace today. - No UI changes, no new buttons, no flow changes. ## Tests - `tests/recipe-canvas-delete-keys.test.ts`: fails on main (constant missing) and when only the canvas wiring is reverted, passes with the change. - Typecheck clean; eslint / biome diagnostics on the touched source files are identical to main. ## Before / after Two fresh Studio installs (merge base `a33463f` vs this branch), the "Text to SQL" learning recipe opened fresh on each, one block selected, then the key pressed. Node counts are read from the canvas DOM. | | Chromium | Firefox | WebKit | |---|---|---|---| | Delete removes the selected block, before | no | no | no | | Delete removes the selected block, after | yes | yes | yes | | Backspace removes the selected block, before and after | yes | yes | yes | | Delete while typing in a block's inline field keeps the block, before and after | not exercised (no block selected) | yes | yes |  Before: the selected block on the right edge (and its dashed link) is still there after pressing Delete. After: it is removed.	2026-10-19 06:44:17 -07:00
Daniel Han	e632df1ca6	CI: keep one generation of prefix-restored caches, share Core's pip cache (#14263) ... The repo's Actions cache is at 50.2 of its 50 GiB, so GitHub is evicting entries by last access regardless of whether they are live. That is the slow outcome for CI: the entry that goes can be main's pip cache or a multi-GB GGUF model, and the next run re-downloads it. This PR removes duplicate copies only; every family keeps its newest entry. ## Where the space was | GiB | Entries | What | | --- | --- | --- | | 21.9 | 22 | `pip-v2-*`: 11 job families, most with 2 generations | | 12.1 | 6 | GGUF / HF model files (exact keys, untouched here) | | 6.3 | 6 | `uv-*`: 3 OSes x 2 generations | | 6.0 | 8 | `v0-rust-*`: 4 jobs x 2 generations | | 3.7 | 92 | ccache shards, Playwright, apt, fe-dist, misc | Seven pip families (`consolidated`, `consolidated-zoo`, `repo-cpu-tests`, `studio-backend` x2, `mlx`, `llama-cpp-smoke`) all key on `pyproject.toml` + `studio/backend/requirements/*.txt`, so every edit to those files writes a new generation in all seven at once (about 8.7 GiB), and `keep: 2` then holds the previous set beside it. ## Changes 1. **cache-janitor: keep 1 generation for `pip-v2-*`, `uv-*` and `v0-rust-*`.** All three restore with a prefix fallback (`restore-keys: <prefix>` in pip-cache-restore and uv-cache-restore; rust-cache's restore key is its key minus the lockfile hash), so a run whose exact key was pruned still starts from the newest generation and downloads only the wheels or crates that changed. The second copy only ever answered an exact revert, or a re-run on a merge ref older than the last requirements bump. `pip-v2-notebooks-colab-*` already worked this way; its special case is folded into the general rule. `fe-dist-*` has no fallback, so it still follows the `keep` input. 2. **`consolidated-zoo` restores `consolidated`'s pip entry read-only.** Both jobs run the same `core-cpu-setup` call on the same matrix (already asserted by `test_zoo_suite_parallel_isolation.py`), so the separate ~1.4 GiB entry held the same wheels twice. The shared-name rule exists because two writers of one key race and the loser's extras are never saved; with the save step removed there is one writer and nothing to lose. 3. **Tests.** `test_pip_cache_naming.py` gains a `READ_ONLY_BORROWERS` allowlist and a test that the borrower restores only its owner's name, never saves, keys on the same files, and that the owner has exactly one saving job. `test_cache_budget_discipline.py` accepts restore-without-save for that one job only. The preamble-parity test now compares the restore step whole, name included. ## Verification - `tests/studio/test_pip_cache_naming.py`, `test_cache_budget_discipline.py`, `test_zoo_suite_parallel_isolation.py`, `test_uv_cache_discipline.py`: 279 passed. - Mutations: putting back `name: consolidated-zoo` fails 2 tests; putting back the zoo save step fails 2 tests. - Janitor replay: the prune script from base and head was run against the live cache inventory (127 entries) with a stubbed `gh`. - Base: would prune 3 entries / 1.1 GiB, leaving 49.1 GiB. - Head: would prune 18 entries / 16.0 GiB, leaving 34.2 GiB. - Every deletion is an older generation of a pip-v2 / uv / rust family (plus the fe-dist entry base also prunes). No model cache or newest generation is touched, and no pip-v2 / uv / rust family keeps more than one entry per (ref, version). ## Trade-off A run that would have hit the older generation exactly now takes the prefix fallback and downloads the delta. That is a few wheels on a requirements bump, against a cache that otherwise sits over its ceiling and evicts live entries. The old `consolidated-zoo` entries are left for the janitor and GitHub's 7-day idle timer.	2026-10-19 06:42:56 -07:00
Tom Mathews	2b20770c91	feat(studio): add a setting to hide the macOS menu bar icon (#11937) ... Closes #10157 ## What Adds a macOS-only **Settings → General → Startup → Show in menu bar** switch. Turning it off hides the Unsloth menu bar icon immediately and keeps it hidden on later launches; turning it on brings it back. ## Why this shape A login start runs with `--hidden`, and `.setup` then switches the app to `ActivationPolicy::Accessory` (no Dock icon). With the menu bar icon also gone, a hidden start would leave a running app with no way back to it. So a hidden start keeps its Dock icon whenever the menu bar icon is hidden: the window still stays hidden, and clicking the Dock icon opens it through the existing `RunEvent::Reopen` → `show_main_window` path. "Run Unsloth at login" is left exactly as the user set it. ## Changes **`studio/src-tauri/src/main.rs`** - New `tray-icon-visible-v1` preference beside `close-to-tray-v1`, with the same read/write helpers. A missing, unreadable or invalid file reads as *visible*: hiding is opt-in, and a corrupt file can never strand the app. - The tray is built with `TrayIconBuilder::with_id("main")`, and on macOS the stored visibility is applied right after it is built. - `get_tray_icon_visible` returns `None` off macOS, so the row only renders on macOS. `set_tray_icon_visible` applies `TrayIcon::set_visible` first, then persists; if persisting fails it restores the previous visibility and returns the error, so the switch never disagrees with what the next launch will do. - The Accessory policy is applied only when `launched_hidden` **and** the icon is visible. - Two unit tests: the default is visible and round-trips; an invalid file reads as visible. **`studio/frontend`** - `api/tray-icon.ts` invoke wrappers, and a Startup row in `general-tab.tsx` built on the existing `useDesktopBooleanSetting` hook, mirroring the close-to-tray row. - The row is indexed for Settings search only on the macOS desktop (`createSettingsSearchIndex({ …, menuBarIcon })`), with a test extended in `tests/settings-search.test.ts`. - `menuBarIcon`, `menuBarIconDescription` and `menuBarIconSaveError` added to `en.ts` and translated in all 11 other locales, since `i18n:check:strict` requires them. ## Not in this PR The issue comment also asked for a monochrome icon. That already shipped in #10880 (`macos_tray.rs` picks light or dark artwork from the menu bar's effective appearance), and #10196 is marked superseded. ## Verification (macOS 26, Apple M1 Pro) - `cargo test --no-fail-fast`: 580 passed, including the two new tests. `cargo fmt --check` / `cargo clippy` report only findings that already exist on `main` (120 fmt diffs on `main` itself); none are in the lines this PR touches. - `npm run typecheck`, `npm test` (8776 passed), `npm run i18n:check:strict`, `npm run build` all pass. `npm run lint` reports only existing findings in files this PR does not touch. - Ran the debug desktop build and read the menu bar and Dock state through System Events: - normal launch → menu bar icon shown, Dock icon shown; - `--hidden`, icon visible → menu bar icon shown, no Dock icon (unchanged behaviour); - `--hidden`, icon hidden → no menu bar icon, Dock icon kept. - Toggled the switch in the running debug build: off → the menu bar item disappears and `tray-icon-visible-v1` holds `false`; on → it returns and the file holds `true`. After switching it off, quitting and relaunching, the icon stays hidden at startup and the switch reads off. ## macOS Desktop before/after Captured on a GitHub `macos-26` runner: the Desktop app built from the merge base (`31436b1`) and from this PR (`2a07993`), both attached to the same installed Studio backend and driven through the Accessibility API. Settings > General > Startup. The new "Show in menu bar" row appears only on the PR build:  Flipping the switch on the PR build removes the Unsloth menu bar item, and flipping it back restores it:  Read from the running app at each step: the app's menu bar extras count went 1 -> 0 -> 1, the switch value 1 -> 0 -> 1, and `tray-icon-visible-v1` held `false`, then `true`. --------- Co-authored-by: Nilay Yadav <nilayyadav10@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>	2026-10-19 06:41:35 -07:00
Vineeth Sai Varikuntla	71bf7993c4	Honor layers_to_transform and finetune_last_n_layers with regex target_modules (#14021) ... `FastModel.get_peft_model` raises `ValueError: layers_to_transform cannot be used when target_modules is a str` whenever `finetune_last_n_layers` or `layers_to_transform` is set with the default targets. **Cause:** the default targets become a regex string, and PEFT refuses `layers_to_transform` with a string. The Llama path hits the same error with a regex `target_modules`. **Fix:** new `_regex_targets_as_names` turns the regex into the module names it matches in those layers (using PEFT's own layer check), called on both paths. `"all-linear"` is unchanged. **Test:** new `tests/test_peft_layers_to_transform_regex_targets.py` on CPU with a tiny Llama. On main 3 fail with that ValueError; after, 3 pass. Existing layers_pattern, str target_modules, last-n-layers and MoE target tests pass (72 total). Not run on GPU. --------- Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>	2026-10-19 06:38:21 -07:00
Daniel Han	46e4f50bea	Studio: raise the startup bundle budget for #14247 (#14258) ... ## Summary Main is over the startup bundle budget again, so `Frontend build + bundle sanity` fails on every PR based on it: ``` over the startup budget: transfer 2012.0 KB > 2011.9 KB, raw 6681.8 KB > 6680.6 KB (main at 26e246d407) ``` #14247 (Developer mode opt-in for experimental CPU video generation) added 1,777 B raw / 654 B transfer to the startup chunks. It merged without raising the limits. ## Cause `#14247` touches eager startup code. Its new strings land in `i18n/locales/en.ts` (the English catalog ships in the startup chunks), and the developer-mode flag is read from `__root.tsx`, `app-sidebar.tsx`, `config/env.ts` and `config/hardware-verdict.ts`. The settings section itself is lazy. This is small real growth with no lazy home. The other commits since the last measurement (#14204, #14239, #14249) change no frontend code. ## Change - **`check-bundle-budget.ts`:** a BUDGET comment entry for #14247, and both limits moved to the measured build plus a 160 B raw / 189 B transfer margin: - `rawBytes`: 6,840,888 -> 6,842,343. - `transferBytes`: 2,060,219 -> 2,060,510. ## Testing One machine and build, Node 24, exact byte totals from the budget script. CI has matched this machine to the byte on earlier budget changes. | tree | raw B | transfer B | |---|---|---| | `88afa84b87` (before #14247) | 6,840,406 | 2,059,667 | | main `26e246d407` | 6,842,183 | 2,060,321 | - **Main against the old limits:** `over the startup budget: transfer 2012.0 KB > 2011.9 KB, raw 6681.8 KB > 6680.6 KB`. - **With this change:** `within budget (0.2 KB transfer, 0.2 KB raw to spare)`, 96 chunks.	2026-10-19 06:30:09 -07:00
Daniel Han	26e246d407	Studio: Developer mode opt-in for experimental CPU video generation (#14247) ... Closes #9458. Video stays hidden on hosts without a supported GPU by default. This adds an owner-only Developer mode in Settings > General with one switch, "Allow video generation on CPU (experimental)". `UNSLOTH_VIDEO_ALLOW_CPU` overrides the saved setting in either direction. ## What changes - `video_capability()` admits only the plain CPU-only case (reason `no_accelerator`) when the opt-in is on, marked `video_cpu_experimental`. Detection failures, missing or broken PyTorch, Intel Macs and GPUs PyTorch cannot reach stay blocked. - The setting lives in the owner's store (`PUT /api/settings/developer-mode`, owner only, 403 for managed accounts; any account can read it). - `/api/health` carries `video_cpu_opt_in` so the sidebar keeps the Video row enabled once opted in. - Enabling opens a confirm dialog: "CPU video generation may take hours, consume large amounts of RAM, and may freeze your system. Continue at your own risk.", plus host RAM against a 32 GB minimum and a measured speed estimate. Disabling restores the GPU gate at once. - The Video page shows a persistent "CPU (experimental)" banner while it is in effect. The backend video pipelines already resolve to a float32 CPU target when no accelerator is visible (the routes on `main` already render on CPU; only the capability gate and the UI blocked it), so no inference code changes. ## Testing - Wan2.1-T2V-1.3B with no GPU visible, through `/video/load` and `/video/generate` on a real Studio server: 832x480, 5 frames, 3 steps, generate 228 s on 16 threads, peak RSS 24.5 GiB (this sets the 32 GB minimum). After the owner PUT, `/api/system` moved from `no_accelerator` to `video_supported: true, video_cpu_experimental: true`, and the load reported device `cpu`, float32. The output matches the base tree's frame for frame (mean 141.29). - Tiny Wan pipelines render on CPU for `wan2.1-t2v-1.3b`, `wan2.2-ti2v-5b` and `wan2.2-t2v-a14b`. - Backend: 34 new tests (capability matrix, every other refusal reason staying blocked with the opt-in on, env and setting precedence, owner-only PUT, health wiring); the related suites pass (145 in the capability, developer-mode and account settings files). - Frontend: 8 new tests (RAM check, confirm flow, gate wiring); `tsc` and the strict i18n parity check pass. Not verified on CPU: LTX-2 / 2.5, HunyuanVideo 1.5 and MiniMax-H3 (no tiny repos; MiniMax-H3 already runs through the CPU sd.cpp build on such hosts). A default-length clip (33 frames, 30 steps) was not timed; "hours" is extrapolated from about 27 s per denoise step at 832x480. ## Screenshots Studio started with no GPU visible (`/api/system` reports `no_accelerator` on both sides). Settings > General: AFTER adds the Developer mode section with the switch, off by default.  Turning the switch on asks first (the issue's warning, host RAM against the minimum, and the measured speed):  After confirming, `/api/system` reports `video_supported: true, video_cpu_experimental: true` without a restart, and the Video page opens with the CPU (experimental) banner. BEFORE, the same host keeps the no-accelerator block: 	2026-10-19 06:16:30 -07:00
Daniel Han	39efb789eb	FSDP2 QLoRA: give the all-gathered 4-bit weight back its Params4bit class and 3D shape (#14204) ... ## Summary 4-bit MoE models crashed under FSDP2. FSDP2 rebuilds the all-gathered weight as a plain `nn.Parameter`, and `fsdp2_qlora` only carried the bitsandbytes quant attributes over. The unsloth-zoo 4-bit MoE paths choose their kernel with `isinstance(w, Params4bit)` and read a 3D expert stack's logical shape from `_original_shape`, so under FSDP2 they treated the packed bytes as a dense weight (`RuntimeError: size mismatch` / contraction dimension errors). `fsdp2_qlora` now: - records the original parameter class and `Params4bit.module` when FSDP2 shards the weight; - gives the all-gathered parameter that class back. It is the same object and storage; only `__class__` changes, and only from exactly `nn.Parameter` to a Parameter subclass, so DTensor (HSDP / TP) parameters are left alone; - carries `_original_shape` with the other quant attributes; - marks the gathered weight `_unsloth_fsdp2_gathered`, so the zoo MoE caches re-key their NF4 pointer tables per call (the storage moves around every forward and backward). Companion zoo PR https://github.com/unslothai/unsloth-zoo/pull/1872 (stale `__dict__` swap and gpt-oss backward pointer table). Without FSDP2 nothing here runs. ## Tests `tests/test_fsdp2_qlora_moe_gloo.py` (new, 2 gloo ranks, real bitsandbytes 4-bit 3D stacks and ModuleList `Linear4bit` experts): gathered class, `_original_shape`, `module`, the marker, `torch.chunk`, bit-exact forward and input gradients, and the reshard back to DTensor. 3 failed on main, 3 passed here. The existing FSDP2 CPU suite (42 tests) passes. 2-rank NCCL, with the zoo companion: | Case | main | this PR | |---|---|---| | ModuleList 4-bit experts, grouped MoE | `RuntimeError` | fwd 0.0, dX 0.0 vs unsharded | | transformers v5 3D 4-bit experts, every MoE backend | `RuntimeError: size mismatch` | fwd 0.0, dX 0.0 | | gpt-oss 4-bit experts | grouped path never used | grouped path used, exact | End to end (tiny Qwen3-MoE, `load_in_4bit = True`, LoRA, `UNSLOTH_FSDP2=1`, 2 ranks vs 1 GPU with GA x2): | step | 1 GPU loss / grad_norm | FSDP2 loss / grad_norm | |---|---|---| | 1 | 11.930937 / 0.02644 | 11.930938 / 0.02688 | | 2 | 11.928570 / 0.01856 | 11.928556 / 0.01904 | | 3 | 11.927559 / 0.01245 | 11.927561 / 0.01258 | | 4 | 11.927320 / 0.01094 | 11.927359 / 0.01177 | The grad-norm gap is the same as 16-bit LoRA on that model under FSDP2 (about 1.5%). Main crashes on step 1. ## Not covered here - The 2-rank component check also passes on Kaggle T4 x2 in fp16 with torch 2.10 and 2.14 (fwd 0.0, dX 0.0 in every case). The end-to-end training table is torch 2.13 only. - No real Hub checkpoint; tiny models only.	2026-10-19 06:08:27 -07:00
Daniel Han	a770de942d	Studio: build the OpenAPI schema off the event loop (#14239) ... Fixes https://github.com/unslothai/unsloth/issues/9712 ## What was wrong FastAPI's built-in `/openapi.json` route is `async`, and it builds the schema inside that route, on the event loop. Studio has about 550 routes, so the first build per backend process takes seconds, and nothing else is served while it runs: `/api/liveness`, `/api/health`, chat, everything. Every session triggers it. Since #8078 the chat sidebar fetches `/openapi.json` to check whether the backend supports guarded title patches before it repairs legacy chat titles. ## How it was found The stall watchdog from #9715 is armed in `studio-mac-ui-smoke.yml` and logs `event loop answering again after Xs` whenever a no-op scheduled onto the loop takes more than 1s. I went through the server logs from the 167 runs whose artifacts are still retained (Oct 18-19): - 126 of 167 runs had at least one loop stall over 1s, 183 stalls in total, the longest 5.3s. - 182 of the 183 overlap a slow `/openapi.json`. The one left over is a single 2.6s probe during `/api/inference/validate`. - The first `/openapi.json` per backend took 2.26s median, 3.20s p90 and 5.94s max on the macOS runner (622 backends). Later calls hit FastAPI's cache: 17ms median. None of those runs had a 10s+ stall or a faulthandler dump, and the logs from the runs in the original report have expired. So I cannot prove the 10-33s windows in #9712 were this. The shape does match, though: a request accepted and then left sitting, nothing served, and recovery on its own. The same CPU-bound build on a starved 3 vCPU runner that is also running CPU-only llama-server could plausibly last that long. ## The change `FastAPI(openapi_url=None)` turns off the built-in route, and `/openapi.json` is registered on the same path. It builds the schema with `asyncio.to_thread`, under a lock so that two cold requests build it once, and later requests read FastAPI's existing cache. The `root_path` server entry is added the same way FastAPI's route adds it. `app.openapi_url` stays `/openapi.json`, so `/docs` and `/redoc` are unchanged, and `app.openapi()` still works for callers in the same process. The served schema is byte-identical. The build still holds the GIL while it runs, but the loop now gets a turn at every switch interval, so requests keep being served during the build. ## Verification Repro: an in-process ASGI client polls `/api/liveness` on the backend's own event loop while the first `/openapi.json` is in flight. | | first `/openapi.json` | worst liveness delay during it | schema | |---|---|---|---| | main | 3.44s | 3490ms (loop blocked) | 1,049,101 bytes | | this PR | 4.10s | 292ms | 1,049,101 bytes, identical | This host was heavily loaded (loadavg ~690), so read these as ratios, not absolute times. `tests/test_openapi_off_the_event_loop.py` slows the cold build by 3s so the result does not depend on machine speed: - main: 2 failed (event loop blocked 7.6s and 7.7s), 1 passed (the behaviour-parity test) - this PR: 3 passed Route, auth and docs suites (`test_middleware`, `test_desktop_auth`, `test_keyless_api_access*`, `test_lan_access_settings`, `test_quick_tunnel_streaming_routes`, `test_mcp_forward`, `test_chat_history_routes`, `test_account_installation_mutations`, `test_health_answers_within_probe_budget`, `test_stall_watchdog`) pass when run serially. Under `-n 8` on the loaded host, `test_desktop_auth::test_health_response_reports_desktop_capability_fields` fails the same way on main. End to end on macOS (`Chat UI, API, Update and GGUF inference`, staging run 39253072492, green): the first `/openapi.json` per backend still took 1.76-2.82s to build, but none of the 6 backend logs had a single `consecutive slow probes` line, so the event loop never went more than 1s without answering. On main, 126 of 167 runs had at least one.	2026-10-19 06:04:29 -07:00
Daniel Han	d122ab5d9d	pre-commit: cut the kwarg formatter's CPU 6.5x and stop the ruff fork bomb (#14174) (#14249) ... Fixes the compute half of #14174: pre-commit.ci reported that the `ruff-format-with-kwargs` hook makes this repo its most expensive user, and that it fork-bombs ruff. ## What was wrong pre-commit.ci runs every hook with `--all-files` on every PR, so the cost that matters is one whole-repo run. On main that was **458 CPU-s** for 4314 files: - `scripts/enforce_kwargs_spacing.py` re-tokenized, re-parsed and re-dumped the same text up to seven times per file (4 full `tokenize` calls, 7 `ast.parse` calls, 2 or more whole-file `ast.dump` calls, several `ast.walk` calls), and the cyclic GC swept the resulting objects for nothing. `ruff format` itself is about 9 CPU-s of the total. - The hook had no `require_serial`, so pre-commit started one partition per CPU, and each partition started its own multithreaded `ruff format`, plus two `ruff --version` probes and two Python processes. On a 192-core box that was 168 concurrent `ruff format` processes and about 32,000 threads. ## What this changes - `.pre-commit-config.yaml`: `require_serial: true` on the hook. - `scripts/run_ruff_format.py`: one `ruff format` per hook call. The Python passes either side run in at most 8 size-balanced chunks (bounded by usable CPUs, `UNSLOTH_RUFF_FORMAT_JOBS` overrides), with a barrier between stages. Every name that reaches one file stays in one chunk, in order, and chunks are killed if the wrapper is interrupted. - `scripts/enforce_kwargs_spacing.py`: each stage parses the file once. A sparse scan finds kwarg `=` operators and string literals instead of a full tokenize (f-strings still go through the interpreter's tokenizer, so the 3.11 vs 3.12+ difference is kept), only def headers and short asserts are tokenized, and edits are checked on the edited fragment instead of dumping the whole file. Anything unusual takes the previous implementation, kept verbatim as `_reference_*`: syntax errors, line separators `splitlines` and tokenize disagree on, PEP 695 headers, possible 3.14 t-strings, shared or continued assert rows, f-folds glued to a keyword or next to a `u` prefix. - Tests: `tests/test_run_ruff_format_jobs.py` (job count, chunking, interrupts, chunked vs serial on real files); `tests/test_formatter_fixed_point.py` pins its own parallel batches to one job each. ## Results `pre-commit run ruff-format-with-kwargs --all-files`, base and head alternated in the same window, median of 3: | | main | this PR | |---|---|---| | user CPU | 402.8 s | 62.7 s | | sys CPU | 55.7 s | 7.2 s | | **total CPU** | **458 s** | **70 s** (6.5x less) | | concurrent `ruff format` | 168 | 1 | | wall (192 cores, shared host) | 20 to 25 s | 13 to 31 s | | tree after the run | clean | clean | The 53k-line `studio/backend/routes/inference.py` goes from about 12 s to about 1.7 s per run. In-process over the repo the Python passes go from roughly 400 CPU-s to 46 CPU-s. ## Output is unchanged The formatter has to stay byte-identical, so the new script was compared against the previous one with a differential harness: every public pass, both stages, `process_file` on real files, and the full pre, ruff 0.6.9, post pipeline, checking output bytes, the changed flag and the exception type. - All 4314 repo files, the CPython stdlib and `transformers/src`, each also with six kinds of unformatted variants (kwarg spaces stripped, def commas toggled, asserts exploded, blank lines after imports, split strings, redundant `pass`): about 80k texts, 0 differences. - CPython 3.9, 3.10, 3.11, 3.12, 3.13 and 3.14, including hand-written 3.14 t-string cases. - About 1.6M generated programs aimed at the fast paths. Those runs and the review found four inputs where a fast path diverged: an f-fold next to a `u` prefix on another line, an f-fold glued to a keyword, and two asserts on a continued or shared row. The last two commits fix them, and each repro now matches on every version. The final script then passed the 415-file sample with variants on all six versions, the repo, and 150k fresh programs, with 0 differences. - The only known difference is on input that does not parse: on Python 3.10 a file with a NUL byte returns unchanged where the previous script raised `ValueError`. ruff then rejects that file, so the hook fails on it either way. Existing behaviour that this PR keeps on purpose, to be fixed separately: - On 3.12+ the spacing pass rewrites `f"{x=}"` to `f"{x = }"`, which changes the printed text. - `remove_redundant_passes` can turn the next string into a docstring, leave an empty suite, or cut the wrong characters after non-ASCII text on the same line. - With PEP 695 type parameters the trailing comma can land inside the bound (`def f[T: g(1)](a, b, c=0)`). ## Not covered - Windows and macOS were not run; the chunked wrapper only uses `subprocess`, `tempfile` and `os.stat`. - The CI move that keeps this hook off pre-commit.ci entirely is a separate PR.	2026-10-19 06:01:54 -07:00
Daniel Han	88afa84b87	Studio: route clicks and scrolls to the browser page on macOS 27 (#14216) ... Fixes https://github.com/unslothai/unsloth/issues/13629 ## Problem On macOS 27, web pages in the Desktop browser panel ignore clicks and scrolling. Hover still works, so a button looks like it reacts, but nothing happens. On macOS the page sits under the app's transparent webview (#13122). The app's webview returns nil from `hitTest:` over the page, so AppKit gives the click to the page below it. macOS 27 no longer asks `hitTest:` for a left click, a scroll or a cursor update. `NSWindow` now calls `-[NSView hitTestForContext:]`, a new method that WKWebView answers on its own without calling `hitTest:`. The app's webview claims every click and scroll over the page. ## Fix `browser_layer.rs` also replaces `hitTestForContext:` when the webview class has it (macOS 27 and later), with the same rule as `hitTest:`: nil over the page, unless panel UI covers that point. The context's point is in window coordinates, not the superview's. If the context ever lacks a `point`, the call goes to the original method unchanged. Before macOS 27 the method does not exist, so nothing is replaced and nothing changes. ## Testing A standalone AppKit probe copies `browser_layer.rs`: a transparent app WKWebView over a page WKWebView of the same class, with `hitTest:` replaced after both exist. It sends real HID mouse and scroll events through `CGEventPost`. GitHub `macos-26` (26.6.2) and `xcode-27` (macOS 27.0.1) runners: | macOS | code | click on the page | scroll on the page | click on app UI next to the page | |---|---|---|---|---| | 26.6.2 | before | page | page scrolls 1000 px | app | | 26.6.2 | after | page | page scrolls 1000 px | app | | 27.0.1 | before | **app** (page gets nothing) | **app** (page stays at 0) | app | | 27.0.1 | after | page | page scrolls 1000 px | app | | 27.0.1 | after, point read as superview coordinates (control) | page | page scrolls 1000 px | **lost** (neither view) | - On macOS 27 the window's left-click path is `-[NSWindow _handleLeftMouseDownEvent:]`. It calls `hitTestForContext:` with an `NSHitTestContext` for cursor updates (event type 17), left mouse downs (1) and scroll wheels (22). The context's `point` is the window point, passed unchanged down the view tree. - The last column clicks the app 20 px left of the page's edge, with the app's webview offset 37 px in the window. Read as window coordinates, the click stays with the app. Read as superview coordinates, the app gives it up, the page refuses it, and the click is lost. - `cargo check --all-targets` (debug and `--release`) of `studio/src-tauri` with this change passes on `macos-26` and `xcode-27`.	2026-10-19 05:36:17 -07:00
Daniel Han	8de2de25bb	Studio: keep code colour while scrolling long chats (#14242) ... ## Summary Follow-up to #13919. Since that merge, dragging the chat scrollbar over a long code block turns the code into plain text until the thumb is released. This goes back to the default @Sneakr proposed in #13919, so code that has been coloured stays coloured while scrolling. ## Cause #13919 shipped with the line window as the default and a guard that holds the window still during a scrollbar thumb drag. With the line window, every line of a large fence outside the window renders as plain text. While the thumb was held, the window did not follow it, so the code being read showed as raw text until release. ## Change - `SHIP_DEFAULT` goes back to `defer`: once a fence is coloured it keeps its colour while you scroll, and nothing is swapped during a drag, so the drag stays smooth. - The scrollbar guard is removed. It only applied to the line window, which stays reachable as an override (`window.__UNSLOTH_DEFER_FENCE_HIGHLIGHT__ = "window"`). - Everything else from #13919 stays: chunked fence rendering, hoisted line colour, the progressive highlight queue, Thinking code colour, and the selection and cancelled-drag fixes. ## Measurements Real Studio, production builds, a thread of ten Python fences of 280 to 3,200 lines. The middle of the 3,200-line fence was coloured first, then the scrollbar thumb was dragged back over it and held: | | main | this PR | |---|---|---| | visible lines coloured while the thumb moves | 0 of 52 | 52 of 52 | | visible lines coloured while held still | 0 of 52 | 52 of 52 | | visible lines coloured after release | 52 of 52 | 52 of 52 | Measured with this default in #13919: a thumb drag over the large fence had 0 to 350 ms of long frames at 4x CPU throttle, against 4.2 s with the line window and no guard. Trade-off: with `defer`, every line already coloured keeps its spans, so a very long chat read end to end holds more DOM, and a theme change after such a read is slower (about 2 s at 1x on the test thread) than with the line window. ## Testing - Frontend: 12,791 tests pass, typecheck, production build, bundle check. ## Before and after Same thread on main (BEFORE) and this PR (AFTER), each from its own production build, with the scrollbar thumb held still in the middle of the 3,200-line fence after the region had been coloured. 	2026-10-19 05:32:07 -07:00
Etherl	f969957da8	Studio: keep the Read aloud speech model loaded beside the chat model (#14191) ... ## Summary Fixes #9163 (its second ask, reading replies aloud automatically, shipped in #13357). Read aloud with an Unsloth Studio speech model could not run next to the chat model. On main, loading a TTS model in Audio unloaded the chat model, even with Keep multiple models loaded set to When they fit. The next chat message reloaded the chat model and unloaded the TTS model. And with both loaded, Read aloud still failed with "No speech model is loaded", because the request named no model and reached the chat model. The backend already supports both models loaded at once, so all fixes are in the frontend: - The Audio TTS load and the chat auto-load ("Loading last used model...") now send `alongside: true`. Under When they fit or Always, both models stay loaded. Off behaves as before, since the server ignores `alongside` there. - Read aloud looks up the loaded speech model in `/v1/models` and names it in `/audio/generate`. Audio page Generate does the same when its model sits beside the chat model. - The Audio page shows, generates with and ejects its own speech model, never the chat model. With only a chat model loaded, the header says "No TTS model loaded" instead of showing the chat model. Switching speech models unloads the old one first, so two TTS models don't pile up. - Every status read updates the Keep multiple models loaded mode, so a session that starts on the Audio page still knows it. Copy changes, in all 14 locales: - Settings > Voice: the engine option "Load TTS model" is now "Unsloth model" (Desktop shows these strings, so no "Unsloth Studio", per the Tauri branding contract), and the TTS model row says which model is loaded. Under Off, a new row explains that loading a speech model unloads the chat model and offers "Keep both loaded" (switches to When they fit). - Under Off, loading a TTS model in Audio says up front that it unloads the chat model, and the success toast has the same "Keep both loaded" action. - Loads that unload another model say which one. Read aloud errors point at Audio > Text to Speech (the old text said "Speak"). The Keep multiple models loaded description now mentions speech models. `generateStudioTtsAudio(text, signal)` keeps its signature, so it stays compatible with the Voice mode work in #12384. ## Testing Colab L4 (23 GB), Qwen3-1.7B UD-Q4_K_XL for chat, Kokoro-82M Q8_0 and KittenTTS for speech, headless Chrome driven by Playwright. Before the fix (main), these failed: - Off: loading Kokoro in Audio unloaded the chat model with no warning. The next chat message reloaded chat and unloaded Kokoro, then Read aloud failed with "No speech model is loaded". - When they fit: loading Kokoro still unloaded the chat model, and the chat reload still unloaded Kokoro. - With both loaded (simulated `alongside`), Read aloud returned 400 "No speech model is loaded". After the fix (main 2fee36ffa3 + this branch), these passed: - Off: the load toast warns that the chat model will be unloaded, the success toast names it, and "Keep both loaded" in Settings > Voice switches to When they fit. - When they fit, chat loaded first: loading Kokoro in Audio keeps both. The Audio header shows Kokoro, and Generate sends `model=Kokoro` and returns 200. - When they fit, Kokoro loaded first: the chat auto-load sends `alongside: true` and both stay loaded. Read aloud sends `model=Kokoro` and returns 200. - Ejecting in Audio unloads only the speech model. Switching Kokoro to KittenTTS unloads Kokoro, then loads KittenTTS beside chat. - Cold page load on `/audio` under When they fit: loading Kokoro keeps chat loaded, and Read aloud on a chat reply returns 200. Local checks: ```bash cd studio/frontend npm test # 12,677 tests: 12,676 pass, 0 fail, 1 skipped npm run typecheck npm run i18n:check:strict npx vite build npx eslint <touched files> # no new findings vs base npm run bundle:check ``` `bundle:check` is over budget on main too (2005.9 KB vs 2003.5 KB). This change adds about 0.6 KB transfer. New and updated tests: `read-aloud-beside-chat.test.ts`, `audio-page-policy.test.ts`, `read-aloud-studio-voice.test.ts`, `openai-model-catalog.test.ts`, `audio-run-api.test.ts`, `tts-custom-connections-disabled.test.ts`. Not tested: Always mode, transformers (non-audio.cpp) TTS models, saved or cloned voices, Windows, macOS, the desktop app, and audible playback (checked by HTTP 200 and the WAV payload only). ## Before and after Two isolated Studio installs (merge base `2fee36ffa` vs this branch), CPU only, Keep multiple models loaded set to When they fit, with Qwen3-0.6B Q4_K_M for chat and Kokoro-82M Q8_0 for speech loaded together. Audio page: main shows the chat model ("The loaded model is not a TTS audio model"), this branch shows Kokoro.  Read aloud on a chat reply: main sends no model and gets 400 "No model loaded.", this branch sends `model=audio-cpp/audio.cpp-gguf/Kokoro-82M-GGUF` and gets 200.  Settings > Voice under Off, after loading Kokoro: the engine option is now "Unsloth model" (shot before it lost "Studio" for the Desktop branding contract, so the image still reads "Unsloth Studio model"), the TTS model row names the loaded model, and a row offers "Keep both loaded".  --------- Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>	2026-10-19 05:23:38 -07:00
Daniel Han	3697c7d620	Qwen3 decode: size the RoPE read by the explicit position, not only the KV length (#14156) ... ## Summary #13855 made the RoPE cache lazy: it only builds the rows that are asked for. `Qwen3Attention_fast_forward_inference` sized the cache read by the KV length only. A decode step at an explicit position past the KV length therefore indexed past the built rows and hit a CUDA device-side assert. For example: a prefill without `position_ids`, then a hand-made decode at position >= 32768. Base (before #13855) returned finite logits. Normal `generate` is not affected. The model-level decode loop already computes `rotary_seq_len = max(kv_seq_len, position_ids.max() + 1)` once per step and passes it to every layer. Qwen3 attention now sizes the RoPE extension and read from it, so this adds no host sync. Without `rotary_seq_len` the previous sizing is used. ## Tests - New `tests/test_qwen3_decode_offset_rope.py` (tiny Qwen3, GPU): decode at an explicit position past the lazily built rows. It fails on main with the device-side assert and passes here. Not fixed here: Gemma2, Cohere, Granite and FalconH1 read the cache the same way. Their model-level loops do not compute `rotary_seq_len`, so a fix there needs one host sync per decode step. Those are left for a follow-up. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-10-19 05:21:28 -07:00
Lee Jackson	e468ee3ec4	Studio: choose working folders when creating projects (#13873) ... Create project currently requires a trip to Edit project to choose an existing working folder. Add the chooser to Create and share its control with Edit, including changing the folder and returning to the Studio-managed folder. Save the new project and selected folder together using the existing owner-only native grant validation. Use **Source folders** consistently for document indexing, clarify that linking does not grant editing access to the original files, and explain unavailable folder selection. Prevent conflicting Edit actions while a working-folder change is pending. Validation: - 12,401 frontend tests and 271 backend regression tests passed. - TypeScript and production build passed. - Nine browser scenarios covered selection, cancellation, expired grants, retries, reset, pending operations, account restrictions, and a narrow viewport; 11 matching before/after comparisons are in the comment below. - The backend tests include a signed single-use grant and an actual file write in the selected folder. Browser screenshots mock native IPC and HTTP; the packaged Desktop picker has not been exercised. ### Before and after Studio in a browser, merge base `cc8e98e16` vs this branch at `0c45b9c43`, with the same seeded project on both sides. **Create project:** a Working folder section (Studio-managed folder by default, with a note that choosing an existing folder needs Unsloth Desktop), and the Linked folders label and copy become Source folders.  **Edit project:** the Working folder row now shows in the browser too, and the Source folders note says linking does not grant access to edit the original files. The "Loading folders..." row on the right is the folder list still loading when the shot was taken.  --------- Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-10-19 05:17:37 -07:00
Daniel Han	5f3a0a290e	FP8 GEMV long-indexing test: seeded inputs and a last-bit tolerance (#14220) ... `test_fp8_rowwise_gemv_long_indexing.py::test_weight_past_2_31_elements` is flaky. It compares, bit for bit, the tail rows of an N = 524544 launch (BLOCK_N 8) with an N = 512 launch (BLOCK_N 1) on unseeded random inputs. The two block sizes can round the last bit of a bf16 output differently, so the result depends on the global RNG state left by earlier tests. It showed up as false regressions in per-PR parent vs merge runs. On a B200, 32 seeds of the test body in isolation mismatch on exactly the same 4 seeds (one element, max abs diff 1.5e-5) for the #14026 parent, its merge and current main, on torch 2.13 and 2.14. ## Change - The inputs come from a seeded generator. - The tail rows are compared with `assert_close` at an absolute tolerance of 1% of the largest output, instead of `torch.equal`. The point of the test is addressing past 2^31 elements. Wrong addressing reads other rows, or faults, which is far outside that tolerance. ## Validation (B200, torch 2.14) - 3 consecutive runs: pass. - Flip check, with `LONG_INDEXING = False` forced for the GEMV in a scratch copy: the test fails with `an illegal memory access was encountered`. The unmodified copy passes. Test-only change.	2026-10-19 05:04:08 -07:00
Daniel Han	0cf4929de0	RoPE: build grown caches on the GPU again, 32768 rows at a time (#14206) ... ## Summary #13976 made the RoPE cache lazy and built the first cache on CPU, even when the first forward had already grown it past 4 * 8192 rows. That happens on every model with more than 32K positions (`max_position_embeddings`), e.g. any 128K-context model. Two effects on the default path compared with the last release, which built grown caches on the GPU: - **cos / sin differ.** The CPU build rounds differently: 188 fp16 and 16 bf16 entries differ from the release, some in rows below 2048. - **The first forward is slower:** about 0.15 to 0.5 s at 128K positions and 1.2 to 1.6 s at 1M. Grown sizes are built on `inv_freq`'s device again, as the eager `extend_rope_embedding` did; the starting size is still built on CPU, as the eager `__init__` did. The build now fills the output 32768 rows at a time, so its fp32 temporaries no longer scale with the cache size. Rows are bit-identical to the one-shot build. Measured on one B200 (last release / main / this PR): | | release | main | this PR | |---|---|---|---| | cos / sin rows vs release (fp16, bf16; 32K / 128K / 1M; with and without `position_ids`) | - | differ in 128K and 1M cells | bit-identical in all 12 cells | | peak build memory at 128K | | 128 MiB | 136.5 MiB | | peak build memory at 1M | 2084 MiB | 1024 MiB | 588 MiB | | first-forward cache build at 1M | ms | 1.2 to 1.6 s | ms | The larger cache when `position_ids` are passed (generation prefill, packing) is the out-of-bounds fix from #13976 and is kept; its test still passes. The cache is still built only on the GPU that uses it. ## Tests `tests/test_rope_cache_current_device.py`: the new `test_first_grown_cache_is_computed_on_inv_freq_device` fails on main and passes here; it replaces the test that asserted the CPU build. Six chunk-equality tests are added and pass on both. 24 passed. ## Not covered here - No end-to-end long-context generate or training run; the comparison is at the cache level. - torch 2.13 only.	2026-10-19 04:56:02 -07:00
Daniel Han	355985f3e9	doctor --gpus: read NVML P2P enum constants as ints (#14164) ... ## Summary `unsloth doctor --gpus` reads `pynvml.NVML_P2P_CAPS_INDEX_READ` for the P2P read query. nvidia-ml-py defines it as `NVML_P2P_CAPS_INDEX_READ = 0,` (a stray comma makes it the 1-tuple `(0,)`). ctypes refuses the tuple, so every P2P read query failed silently, and doctor reported P2P reads as unavailable on every GPU pair. The NVML enum constants are now read through `_nvml_int`, which unwraps a 1-tuple and falls back to the default for anything that is not an int. ## Tests - New `test_nvml_topology_accepts_tuple_p2p_read_constant` in `unsloth_cli/tests/test_doctor_gpus.py` fails on main and passes here. - On a 2-GPU host here, `doctor --gpus` reported P2P reads unavailable before and available after. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-10-19 04:49:57 -07:00
Daniel Han	d7eef8e134	Optimizer state offload: track DTensor state loaded on the host when resuming (#14155) ... ## Summary With the opt-in `UNSLOTH_OFFLOAD_OPTIMIZER_STATE=1` (#13882) and sharded parameters (DTensor, FSDP2), resuming from a checkpoint loses track of the optimizer state that `load_state_dict` places on the host. The load policy records the loaded value by `id()` of the DTensor, but `offload()` looks DTensor state up by its local shard. So the next `optimizer.step()` never moves that state back and fails with a cuda:0 vs cpu device mismatch. The load policy now also records the DTensor's local shard. Plain tensors are unchanged, and with the flag off nothing changes because the module is only used when the flag is on. ## Tests (GPU) - New `test_load_state_dict_tracks_dtensor_state_on_host` (DTensor params on a 2-rank mesh with a fake process group): it runs offload, a step, a deepcopied checkpoint and a resume into a fresh optimizer. It then checks that the step after resume is bitwise equal to an optimizer that never stopped. - `tests/test_optimizer_state_offload.py`: 1 failed / 36 passed on main, 37 passed here. Not run here: a real multi-GPU FSDP2 resume through accelerate.	2026-10-19 04:49:40 -07:00
Daniel Han	2a4b4980d4	Studio: stage prebuilt installs in dirs that inherit the parent ACL on Windows (#14212) ... Fixes https://github.com/unslothai/unsloth/issues/14052 On Windows, `tempfile.mkdtemp` creates its directory with mode `0o700`. CPython turns that into a protected DACL granting only SYSTEM, Administrators and OWNER RIGHTS: `D:P(A;OICI;FA;;;SY)(A;OICI;FA;;;BA)(A;OICI;FA;;;OW)` (Modules/posixmodule.c, gh-118486). The prebuilt installers extract into a tree inside such a staging dir and then rename that tree into place, and a rename keeps the security descriptor. So the installed `~/.unsloth/audio.cpp` (and the llama.cpp, node, sd.cpp and whisper trees) carries that ACL instead of the user profile's. When the run that installed it was elevated, the owner is BUILTIN\Administrators, so OWNER RIGHTS means Administrators and the same account running unelevated has no access at all. That is #14052 and the "corrupted ACLs" in #9928: * the backend cannot see `audiocpp_server.exe`, so every audio tool says "The audio runtime is not installed"; * `unsloth studio update` fails reading `.unsloth-studio-owned` with `[WinError 5] Access denied`, which the installer reports as "the audio.cpp runtime is in use; stop Studio and retry", and setup prints "install busy; keeping existing runtime", so updating never repairs it; * `Test-Path` on the server binary fails with Access denied. Changes: 1. `prebuilt_core.make_staging_dir`: on Windows, a plain `os.mkdir` with a random suffix, so the staging dir and everything renamed out of it inherit the parent's ACL (the user's own entry under the profile). POSIX keeps `tempfile.mkdtemp`. Used by the audio.cpp, llama.cpp, node and `install_from_bundle` (sd.cpp, whisper) staging dirs and the audio.cpp model_specs dir. 2. audio.cpp installer: a target this account cannot read is refused before anything is downloaded, with the one command that clears it (Windows: `Remove-Item -Recurse -Force "<dir>"` from an elevated PowerShell; elsewhere a root `rm -rf`), instead of "in use; stop Studio and retry". Windows refuses an unelevated rename of that tree too, so nothing unelevated can clear it. `setup.ps1` shows that line instead of the generic failure. The retired-tree sweep skips a tree it cannot read instead of raising. A readable install takes exactly the old path on every OS. ## Windows evidence (gfx1151 Windows 11 runner, CPython 3.13.16) The runner installs elevated as Administrator. The same account is then simulated unelevated with an impersonation token where Administrators is deny-only and privileges are stripped, as UAC's filtered token does. Base `a33463fb1`, head `0e0546e83`. | | base | head | |---|---|---| | ACL of the installed `audio.cpp` | `SYSTEM`, `Administrators`, `OWNER RIGHTS` only, owner Administrators | inherited: `Administrators`, `SYSTEM`, `<user>` | | unelevated `os.stat` of the server | `PermissionError: [WinError 5] Access is denied` | ok | | unelevated re-run of the installer | exit 3, `Access is denied: '...\audio.cpp\.unsloth-studio-owned'); stop Studio and retry` | exit 0, `already matches` | | unelevated update over the tree the base leaves | exit 3, same "in use" | exit 1, names the elevated `Remove-Item` | ## Tests `tests/studio/install/test_install_audio_cpp_prebuilt_logic.py` gains four cases: * Windows staging uses no mkdtemp mode. * An unreadable install is refused with the repair, nothing downloaded, and the tree left in place. * The Windows repair text is the PowerShell one. * The sweep skips a retired tree it cannot read. On the merge base the unreadable case fails with the reporter's message. The audio.cpp, llama.cpp, node, prebuilt_core, whisper, setup status and audio update suites pass. Staging CI is green on ubuntu-latest, macos-15 and windows-latest.	2026-10-19 04:41:37 -07:00
Daniel Han	a9819dd6eb	Prebuilt wheels: drop -lineinfo, unreachable SASS and flash-attn PTX; add extension-free mamba-ssm (#14142) ... Shrinks the prebuilt causal-conv1d, mamba-ssm and flash-attn wheels further without changing the machine code any GPU runs. ## Why After fatbin compression the wheels are still larger than they need to be: - causal-conv1d and mamba-ssm compile with `-lineinfo`. It only feeds profilers a source line table; `cuobjdump -sass` shows identical instructions without it, and it is more than half of every cubin. - Their setup.py hardcodes SASS for sm_87 (Jetson Orin), sm_110 (Jetson Thor) and sm_121 (GB10 / DGX Spark). Those are integrated Arm GPUs, so x86 wheels can never use it. - flash-attn at our pin embeds compute_120 PTX next to the SASS. It can only JIT for GPUs newer than sm_120; upstream 2.8.1 ships SASS only. ## Changes 1. `.github/scripts/patch_cuda_targets.py`: edits each package's setup.py by asserting on exact text (like `patch_mamba_cxx20.py`): removes `-lineinfo`, keeps only the requested SASS, removes flash-attn's PTX line. Unknown or missing targets fail loudly; running it twice is a no-op. 2. Per-platform target lists in `prebuilt_wheels.py`: x86 wheels keep sm 75;80;90;100;103;120, linux_aarch64 and win_arm64 keep the full upstream list. flash-attn keeps sm 80;90;100;120. 3. A `targets-gate` step after the size gate reads the fatbin headers in every built `.so` / `.pyd` (ELF and PE, no cuobjdump needed): every requested SASS present, nothing else, no PTX. Checked against cuobjdump on the published flash-attn Linux wheel and on the Windows causal-conv1d wheel; both current wheels fail the new gate as expected. 4. Size gates lowered to the new expected sizes; ccache tag changed with the target list. ## Expected sizes (installed, extrapolated from compiled translation units) | Package | Now | x86 after | Arm after | |---|---|---|---| | flash-attn | 229 MB | ~176 MB | ~176 MB | | causal-conv1d | 117 MB | ~31 MB | ~47-53 MB | | mamba-ssm | 241 MB | ~62 MB | ~88 MB | Per translation unit (MB, nvcc 13.1, compression on): | File | Now | Trimmed | |---|---|---| | causal_conv1d bwd (95% of the package) | 108.4 | 24.8 | | selective_scan fwd bf16 | 14.78 | 3.36 | | selective_scan bwd bf16 | 25.78 | 5.55 | | flash-attn hdim128 fwd | 3.75 | 2.87 | ## Tests `tests/python/test_prebuilt_cuda_wheels_targets.py` (new) plus the existing prebuilt suites: 230 passed. The patch was also run against the real pinned setup.py files. After merge, the release needs one republish (Windows torch 2.10 to 2.12, then all platforms for 2.13 / 2.14). A machine code diff of old vs new wheels per kept arch and a T4 / L4 kernel A/B follow the build. ## Extension-free mamba-ssm wheel When a run builds mamba-ssm, a CPU-only job also builds `mamba_ssm-2.3.2.post1-py3-none-any.whl` from the same pinned commit without its CUDA extension (about 224 KB, one file for every OS / torch / Python). Mamba-2 hybrids need only its Triton ops once unsloth-zoo runs their causal conv on Triton (unslothai/unsloth-zoo#1865); Mamba-1 models keep the CUDA wheel. - `pure-gate`: exact filename, no `.so` / `.pyd` / `.dll` / `.dylib`, `Root-Is-Purelib: true`, `Tag: py3-none-any`, then `--no-deps` install and import of `mamba_ssm`, `mamba_chunk_scan_combined` and `selective_state_update` in a fresh venv; the same import on windows-latest with triton-windows. - Sign and publish wait for it like any other planned wheel; runs without mamba-ssm are unchanged. Release notes and SHA256SUMS list it. - A local build at the pin is reproducible (identical sha256 across two builds) and passes the gate. 246 prebuilt wheel tests pass. ## Measured on real builds (linux_x86_64, cp313, torch 2.13.0+cu130) Built locally from this branch exactly as the workflow does (nvcc 13.1 here; runners use 13.0): | Wheel | Download | Installed | |---|---|---| | causal-conv1d, published today | 207.1 MB | 262.8 MB | | causal-conv1d, compressed only (current republish) | 111.3 MB | ~122 MB | | causal-conv1d, this PR | 29.2 MB | 39.9 MB | | mamba-ssm, published today | 349.7 MB | 359.3 MB | | mamba-ssm, compressed only (current republish) | 233.0 MB | ~241 MB | | mamba-ssm, this PR | 53.6 MB | 63.0 MB | - size-gate and targets-gate pass on both (SASS exactly 75;80;90;100;103;120, no PTX); the targets-gate fails the published wheel as expected. - Against an untrimmed build with the same nvcc and flags, every kernel's SASS is identical on every kept arch (causal-conv1d 1242 kernels, mamba-ssm 1440, all six archs). - T4 and L4, published vs this PR, 6 interleaved repeats: conv forward / backward / decode update, selective scan and the Triton SSD control all within the repeat spread; outputs bitwise equal except selective scan dB / dC, which differ only at the run-to-run level of its atomic backward. flash-attn was not built locally (6 to 8 hours); its change is the PTX drop only, checked by the targets-gate in the workflow.	2026-10-19 04:40:02 -07:00
Daniel Han	e6c95c292b	Attention: GQA never forces SDPA math, cuDNN-over-varlen on sm100 only, native before xformers (#14158) ... Three fixes to Unsloth's attention routing when the FlashAttention wheel is not installed, from a FA2 wheel vs PyTorch native vs xformers benchmark (Qwen3-0.6B and Llama-3.1-8B 4-bit LoRA, torch 2.10 to 2.14, T4 / L4 / A100 / RTX PRO 6000 / H100, packed, padding-free and padded). ## Why - **GQA forced SDPA's math kernel.** The dense path passes `enable_gqa=True` for every GQA model, but only flash and cuDNN take it. Where flash cannot run (sm75, fp32 / DoRA, masked calls), SDPA silently fell to math. On a T4 with torch 2.10 to 2.13 that made padded training 3x slower with 4.3x the memory of xformers (10.7 vs 2.5 GB). - **cuDNN over varlen hurt H100.** `_torch_varlen_beats_segments` prefers per-segment cuDNN SDPA above a mean segment length of 1024, a threshold measured on B200. On H100 that choice is 2.2x to 2.6x slower than torch varlen at mean lengths 1024 to 2048. - **xformers was picked over faster native kernels.** Without FA2, `select_attention_backend` prefers xformers, but xformers 0.0.35 has no kernel of its own on sm80+ and torch's varlen path matched or beat it on torch 2.12+. ## Changes 1. `enable_gqa` is passed only when SDPA's chosen kernel takes it, decided by asking `torch._fused_sdp_choice` for both the GQA and the expanded-K/V call (cached per shape class, constant under `torch.compile`, so ROCm's backend set is answered by torch too). Otherwise K/V are expanded. The packed segment runs use the same rule. 2. The cuDNN-over-varlen preference applies only on sm100-family GPUs; elsewhere torch varlen wins. `UNSLOTH_TORCH_VARLEN=0` still disables varlen. 3. Without FA2, packed batches use PyTorch's varlen path over xformers when every visible CUDA GPU is sm80+, the host is not ROCm, torch has varlen (2.12+) and the packed segment path is enabled. Dense and decode calls keep xformers first (explicit block masks, grouped FP32 KV layout); sentence unpadding keeps xformers. T4, mixed hosts and torch 2.10 / 2.11 are unchanged. Nothing changes when FA2 is installed. 4. The GQA probe uses the real call's grad state and dropout, and probes the GQA and expanded calls independently. ## Tests `tests/utils/test_attention_native_routing.py` (24 new tests, 20 fail on main). `test_sdpa_packed_segments.py`: the peak memory test asserted the old math path's footprint and now asserts linear memory. Existing attention suites: 504 passed (two order-dependent `test_tiny_mistral_packed_row_matches_the_masked_path` failures also happen on main and pass in isolation). No new graph breaks under `torch.compile`; compiled output matches eager. ## Base vs head Colab, fresh venv per torch, Qwen3-0.6B LoRA, 3 interleaved repeats, tokens/s median [spread]. Identical inputs across arms; losses of steps 1 to 10 agree within 1.8e-3 relative. | Cell | Base | Head | Reference | |---|---|---|---| | T4, torch 2.12, padded, native | 477 [4], peak 10.66 GB | 1388 [6], peak 2.46 GB | xformers 1416 [4], 2.46 GB | | A100, torch 2.13, packed, xformers installed | 11793 [130] (xformers) | 12308 [72] (native) | | | A100, torch 2.13, packed, native (control) | 12428 [211] | 12393 [204] | | | H100, torch 2.14, packed, native | 18226 [5925] | 20227 [1694] | FA2 22131 [892] | | H100, torch 2.14, padding-free, native | 20243 [1662] | 20076 [1512] | FA2 22317 [802] | Not run: ROCm (the GQA decision is torch's own answer there), B200 (cuDNN rule unchanged by design), Windows, torch 2.10 / 2.11 dense GQA on sm80+.	2026-10-19 04:32:43 -07:00
Daniel Han	7729c37fd0	Studio: restore Qwen causal-conv1d on demand, narrow fatal mamba-ssm, provision SSM kernels in export (#13971) ... Follow up to #13861 and #13808. An audit of the merged on demand SSM kernel change plus a T4 benchmark found a few gaps, fixed here. ## Why - On a Kaggle T4 (sm75, torch 2.11 cu130, fp16, Qwen3.5-0.8B), Studio chat decode was 4.4% slower without causal-conv1d (19.10 vs 19.95 tok/s, 6 alternating repeats, slower in all 9 pairs). Training speed, losses and peak memory were unchanged. The zoo Triton conv covers `causal_conv1d_fn` only, so single token decode fell back to the torch update path. Training with trainable conv weights also allocates an fp32 dW scratch in the Triton backward, and packed `seq_idx` is not accepted by the zoo dispatcher under transformers 5.20. - Mamba-family models that have a working transformers torch path (mamba, mamba2, falcon_mamba, jamba, zamba, zamba2, bamba) were made fatal when mamba-ssm could not be installed. - Export, local LoRA continuation, and inference model switching did not provision or pick up kernels correctly once setup stopped installing them. - Setup on pre-Ampere GPUs (T4) downloaded flash-attn even though Unsloth only enables FA2 at sm80+. ## Changes 1. Qwen3.5 / Qwen3.6 / Qwen3-Next request causal-conv1d on demand again in training, inference and Clef (same as before #13861, minus the setup time install). The zoo Triton platform probe is removed. 2. A missing mamba-ssm is fatal only for `nemotron_h`, `falcon_h1`, `granitemoehybrid`. Other Mamba families install best effort and continue on the torch path with a status message. 3. Local LoRA folders (`adapter_config.json`, no `config.json`) resolve kernel needs from `base_model_name_or_path`; the training `adapter_base_pin` wins when set. 4. The export worker provisions SSM kernels before importing Unsloth / the export backend, with the same required vs optional split. 5. Inference re-enables the modeling source scan by reading `transformers/models/<type>/modeling_*.py` as text, without importing transformers. 6. If a kernel is newly installed after the model's `transformers.models.<type>` module was already imported, the inference worker asks for a restart and the orchestrator respawns it once. 7. Torch repair removes causal-conv1d / mamba-ssm whose compiled extension no longer imports (never installs anything new). A hung import check keeps the package. 8. Setup and the long context runtime install skip flash-attn below sm80 (and remove a stale copy at setup). No visible GPU and ROCm keep current behavior. ## Tests New and updated tests fail on main and pass here (`test_ssm_runtime.py`, `test_ssm_restart_respawn.py`, `test_export_ssm_kernels.py`, `test_clef_fast_paths.py`, `test_kernel_install.py`, `test_training_worker_flash_attn.py`, `test_ssm_kernels_install_python_stack.py`, `test_flash_attn_install_python_stack.py`, `test_windows_torch_flavor_invariant.py`). With `test_install_python_stack.py` and `test_cuda_repair.py`: 966 passed after merging current main. ## Hardening (second commit) - Stale SSM kernel removal after torch repair needs proof: torch must import first, and only an ImportError / OSError from the extension itself removes the package. Signals, hangs, launch errors and a broken torch keep it. - `unsloth install-kernels --without-gpu` (or `UNSLOTH_INSTALL_KERNELS_WITHOUT_GPU=1`) installs GPU gated kernels on a GPU-less image builder. An already working kernel counts as success before any skip, skipped kernels are listed, and the exit code is unchanged (0) unless `--strict` is passed. The CLI respects `UV_OFFLINE` / HF offline variables and never falls back from uv to pip offline. - Inference respawn refuses to spawn while the old worker survives shutdown. SSM restart and the Xet fallback each get their own one-use retry (3 spawns max). - Restart detection only counts native `modeling_*` modules and the selected checkpoint's remote code modules from its `auto_map`, matched on the exact `transformers_modules` namespace. - Export provisions SSM kernels inside the existing transformers activation window (no extra Hub reachability probe), and skips provisioning on macOS and for the MLX / CPU export backends. - `UV_OFFLINE` counts as offline for on-demand installs (no URL probes, source builds, or uv to pip fallback). - Long context flash-attn install returns early when flash_attn already imports, reuses the worker's own torch for the capability check, and never applies the sm80 floor on ROCm. ## Install robustness (platform matrix review) - A wheel is downloaded whenever its HEAD check is not a definite 404 (proxies that refuse HEAD no longer push required hybrids into a failure), then import-verified; fallback only after a real failure. - A failed install is remembered for the worker process (keyed by kernel, torch, toolchain), so preflight and load share one attempt. Optional Mamba families get mamba-ssm only as a prebuilt wheel, never a source build. ROCm source builds get an 1800 s deadline and CUDA source builds 7200 s (`UNSLOTH_SOURCE_BUILD_TIMEOUT` overrides); a timeout kills the whole build tree. - A runtime install whose extension does not import is uninstalled, so transformers never finds a broken package. - One cross-process lock per kernel and environment covers check, install, verify and cleanup (training and runtime), with a recheck after acquiring it. - Only a CUDA or ROCm torch provisions anything; CPU, XPU, MPS and unknown torch builds skip (export included). - Optional mamba-ssm follows setup's sm80 floor (sm75 with Triton 3.4+); the required hybrids keep the previous behavior. - The resolver only returns URLs for cells that are actually published (Unsloth release: torch 2.13/2.14 cp313, win_amd64 torch 2.10 to 2.12 cp311 to cp313; upstream causal-conv1d / mamba-ssm release cells), and there is no source build of the pinned 1.6.1 / 2.3.1 sources on torch 2.13+, which need C++20. - Training stays best effort for mamba-ssm, as before #13861. - Export runs the malware and remote code consent gates before any kernel provisioning. - The extension is probed again after the install lock is acquired, so a concurrent replacement is not treated as broken. - The automatic HF offline window (Hub unreachable) no longer blocks package installs; only user set offline variables do. - `unsloth install-kernels --dry-run` prints the resolved URL while offline. - A Mamba model whose code needs mamba-ssm fails with a clear error on a torch build without CUDA or ROCm. ## GPU checks (merge base vs head 4a6429a65) Kaggle T4, torch 2.11.0+cu130, Qwen3.5-0.8B fp16: | | base | head | |---|---|---| | causal-conv1d on demand | not installed | installed, `causal_conv1d_update_kernel` engaged | | decode tok/s (6 alternating repeats) | 18.70 | 19.41 (+3.8%, 6/6 pairs) | | decode peak alloc | 1.748 GB | 1.748 GB | | LoRA step time (3 repeats) | 1975.5 ms | 1970.7 ms (inside spread) | | LoRA peak alloc | 2.338 GB | 2.338 GB | | flash-attn at setup on sm75 | downloaded | skipped | AMD gfx1151 (torch 2.11.0+rocm7.13.0): Qwen3.5 LoRA losses within 1.2e-4 relative (head vs merge commit bit identical), peak 2.389 GB both, backend suites 262/262 and 275/275 passing. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-10-19 04:32:13 -07:00
Eyera	6c3a9e9b34	Studio: fix chat scrollbar freezes on long code fences (#13919) ... ## Summary Dragging the chat scrollbar thumb across a long code fence stalled Studio: the line window inside large fences moved on every frame of the drag, swapping hundreds of thousands of nodes. This PR fixes the stall and keeps the line window, which turned out to matter for every other way people read a long chat. Original change by @Sneakr (defer highlighting by default, plus chunked fence rendering, hoisted line colour and a progressive highlight queue). The scrollbar guard is the one @oobabooga proposed in review. On top of that I merged main, kept the line window as the default, kept syntax colour in Thinking, and fixed two highlight-blocking bugs found while testing. ## What changes - **Scrollbar guard.** A thumb press targets the scroller, an ancestor of the fence, so the existing drag guard never saw it. The held scrollbar is now tracked and the window stays put until release (`code-fence-defer.tsx`, `onScrollbar` in `code-fence-window.ts`). - **Default stays `window`.** `defer` is still reachable through `__UNSLOTH_DEFER_FENCE_HIGHLIGHT__` / `VITE_UNSLOTH_DEFER_FENCE_HIGHLIGHT`. See the slow read numbers below for why. - **Rendering work from this branch, kept:** 32-line memoized line chunks, plain-text chunks, line-level colour hoisting (fewer spans), skipping source-only re-renders, and the input-aware highlight queue. - **Thinking code keeps its colour** (the #13415 path), so nothing visible changes for reasoning traces. - **Fix:** any non-empty selection on the page blocked the highlight queue, so after selecting a sentence and scrolling on, newly reached fences stayed uncoloured until the selection was cleared. Only a selection that ends inside a fence holds it now. - **Fix:** a handler that cancels `dragstart` (the model picker's pinned rows, the hub carousel) left the pointer marked as held forever, which stopped all deferred highlighting until the next real drag or a window blur. - Startup bundle: +11,012 B raw / +3,984 B transfer, budget raised with the measurement. ## Measurements Real Studio (production build, own backend), a thread of ten Python fences of 280 to 3,200 lines (about 134K characters in the largest), headless Chromium driving the native scrollbar thumb, each arm a fresh browser context, arms interleaved. "Long-frame time" is the time past one frame spent in frames over 50 ms. **Thumb drag across the 3,200-line fence, 8 s** (Linux Chromium, median of 3): | | main | main + guard | this PR as first submitted (`defer`) | this PR | |---|---|---|---|---| | 1x CPU: long-frame time | 933 ms | 50 ms | 0 ms | 33 ms | | 1x CPU: frames over 50 ms | 15 | 1 | 0 | 1 | | 4x CPU: long-frame time | 4,166 ms | 300 ms | 350 ms | 33 ms | | line-window DOM swaps during the drag | 72K to 229K | 4.9K | 0.5K | 4.9K | The same drag on a Windows 11 runner in Microsoft Edge at 4x: main 3,940 ms of long frames, main + guard 67 ms, `defer` 0 ms, window + guard 267 ms. No renderer crash or browser disconnect in any arm, including 15 rapid back-and-forth drags per arm, so the STATUS_BREAKPOINT report did not reproduce on that machine. **Reading the thread slowly** (mouse wheel, 400 px every 350 ms from the bottom to the top, then idle; two runs each, both shown): | | main | `defer` default | this PR | |---|---|---|---| | DOM elements afterwards | 34K / 35K | 111K / 111K | 28K / 28K | | theme change, 1x | 0.28 / 0.40 s | 2.60 / 2.12 s | 0.32 / 0.34 s | | theme change, 4x | 1.55 / 1.40 s | 8.06 / 10.19 s | 1.48 / 1.28 s | | thumb drag after the read, 4x (long-frame time) | 3.37 / 2.60 s | 2.53 / 1.72 s | 0.95 / 0.28 s | With `defer` every line the reader passes keeps its token spans, so a slow read of this thread ends with every token mounted. The window caps that, which is why this PR keeps it as the default. Long-frame time during the read itself was too noisy on this shared host to rank the window arms: main measured 1.2 and 3.7 s at 1x (29 and 22 s at 4x), main + guard 2.4 and 0.9 s (35 and 16 s), this PR 3.4 and 0.8 s (41 and 18 s), and `defer` 4.6 and 33.4 s (26 and 43 s). **Expanded Thinking** (six turns, three 150-line fences each, all expanded, median of 3 at 1x): expand long-frame time main 200 ms / this PR 183 ms, DOM 18.3K / 14.3K, theme change 147 / 129 ms. Screenshots of the answer fences and of Thinking are pixel-identical to main in the thread area. ## Testing - Frontend: 12,762 tests pass, typecheck, production build and bundle check. New tests for the scrollbar guard, the selection scope and the cancelled `dragstart`, each failing without its fix. - Cross-platform CI on Linux, macOS (Apple Silicon) and Windows: node tests green, Studio Playwright smoke OK. - Studio UI regression suite (auth, settings, model picker, chat UI, compaction, page crawl), base vs head: no regression. One crawl page diff is an animated remote image; two checks fail identically on main. - studiobench standard tier, A/B against main with a main-vs-main null control (4 repetitions): no metric outside the null control's detection floor. This host was heavily loaded, so the floors were wide. ## Not covered - STATUS_BREAKPOINT itself was not reproduced (Edge on Windows 11 and Chromium on Linux, headless). Users who still hit it can set `__UNSLOTH_DEFER_FENCE_HIGHLIGHT__ = "defer"`. - Overlay and left-side scrollbars and document-level scrolling are not covered by the guard; Studio's thread scrolls in its own viewport with a right-hand scrollbar. - WebKit and the macOS desktop app were covered by unit tests only. ## Before and after Same seeded thread on main (BEFORE) and this PR (AFTER), each served from its own production build. The thread area is identical in both scenes apart from one antialiased pixel.  --------- Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>	2026-10-19 04:15:42 -07:00
Daniel Han	e2b42752bd	Studio: multi-image editing for Qwen-Image-Edit (#14230) ... Closes #8987. Qwen-Image-Edit 2509 / 2511 runs on `QwenImageEditPlusPipeline`, which reads several input images in one edit: each goes through the VLM as "Picture N" and is VAE-encoded beside the source. Studio's Edit tab only ever sent the source image. ## Changes - `qwen-image-edit` takes up to 3 images in total (source plus two), matching ComfyUI's `TextEncodeQwenImageEditPlus` (image1..image3). The edit workflow accepts `reference_images` after `init_image` and passes `image=[source, *extras]` in request order. The output keeps the source's size. A 4th image is refused with a clear error rather than dropped. - Only the Plus checkpoints get it: the limit follows the loaded checkpoint (`max_condition_images_variants`, scoped at load by `family_for_checkpoint` in both engines), so the original Qwen-Image-Edit, trained on the source alone, keeps the single-image edit. - FLUX.1 Kontext is pinned to 1 image (`FluxKontextPipeline` conditions on one) and Qwen-Image-Layered stays at 1, so both still refuse extra images. - Native sd.cpp engine: its Qwen-Image-Edit conditioner already reads every ref image as "Picture N", so extra images go out as ref images instead of being refused. - The activation guard counts each extra image at the pipeline's ~1 MP VAE area. - `conditioning` advertises `multi_image_edit`. The Edit tab then shows "Image 1 (source)" plus "Add image 2/3" slots, with the same empty-slot and limit checks as the Reference tab. Restoring a recipe asks for the extra images again in order. - A single-image edit takes exactly the code path it took before. ## Screenshot The Edit tab with a Qwen-Image-Edit model loaded, before and after:  ## Testing - Backend: 895 passed, 2 skipped across the diffusion, sd.cpp edit, unified edit, MCP status and FLUX.3 suites, including the new `test_diffusion_qwen_edit_multi_image.py` and the native ordering, limit and capability tests in `test_sd_cpp_edit_families.py`. `test_diffusion_routes.py`: 147 passed. - Frontend: the new `image-multi-image-edit.test.ts` (5 tests) and the existing unified edit tests pass; `npx tsc --noEmit -p .` is clean. - A tiny random-weight `QwenImageEditPlusPipeline` (diffusers' own test sizes) loaded through `DiffusionBackend.load_pipeline` and run through `generate` on a B200: | Run | Images reaching the pipeline | VLM prompt | VAE encodes | Output | |---|---|---|---|---| | 1 image | 64x48 | Picture 1 | 1 | 64x48 | | 3 images | 64x48, 32x32, 48x64 (in order) | Picture 1, 2, 3 | 3 | 64x48 | | 4 images | refused: "takes at most 3 input images in total (got 4)" | | | | Not covered here: a render with the full Qwen-Image-Edit-2511 weights, a multi-reference render through the real sd.cpp binary (only the request it builds is tested), and a measured memory weight for the extra images (counted at the default 1.0 per ~1 MP, conservative). --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-10-19 04:10:55 -07:00
oobabooga	cefb057a4e	Block compile: keep the GatedDeltaNet self-check out of a non-reentrant checkpoint (#14195) ... Addresses the GatedDeltaNet failure in #14063. ## Problem On 2026.10.6, LoRA training of Qwen3.5 (a GatedDeltaNet hybrid) with `use_gradient_checkpointing="unsloth"` under DDP or FSDP2 dies in the backward of step 1. Reproduced here on Qwen3.5-0.8B with 2 GPUs; the issue reports the same on the 27B model with 8 GPUs: ``` torch.utils.checkpoint.CheckpointError: torch.utils.checkpoint: A different number of tensors was saved during the original forward and recomputation. Number of tensors saved during forward: 55 Number of tensors saved during recomputation: 45. ``` Single-GPU runs are not affected. `UNSLOTH_BLOCK_COMPILE=0` avoids it. ## Root cause Distributed runs use torch's non-reentrant checkpoint (`use_reentrant = not is_distributed()` in `vision.py`), whose saved-tensor hooks are active for the whole layer forward. On the first GatedDeltaNet layer, the block-compile wrapper calls `_prepare_gdn`, which runs `_gdn_op._fla_ops`. Its `_self_check` does a forward and `torch.autograd.grad` against fla's own Function. Because this happens inside the checkpointed layer forward: 1. The self-check's saved tensors are packed into the layer's checkpoint frame. 2. Its `autograd.grad` unpacks them, which starts a recompute of the whole layer during forward0. 3. That nested recompute enters the block-compile wrapper before the layer is marked checked, so the trial loop catches torch's `_StopRecomputationError` and demotes the layer kind `_FULL -> _SPLIT -> _OFF`. The self-check then fails as well, so the fla opaque op is disabled. 4. The original forward finishes compiled and saves the self-check's tensors plus the compiled set (55). The real recompute in backward runs the stock forward for that layer (45), and `check_recomputed_tensors_match` raises. The fla self-check runs for every user because unsloth_zoo vendors fla. ## Fix Run the self-check in `_fla_ops` under its own `saved_tensors_hooks`, so an enclosing checkpoint never sees its tensors. The pack hook detaches, as torch's own recompute hook does, so a check that fails before `autograd.grad` does not leave a reference cycle holding its tensors. #14065 alone does not fix the reproduction. Re-raising `_StopRecomputationError` removes the demotion, but the self-check's tensors are still in the frame, so the run still fails with 55 vs 46. With this change the nested recompute never happens. ## Verification Qwen3.5-0.8B, LoRA r=16, `use_gradient_checkpointing="unsloth"`, 3 steps, 2 GPUs, torch 2.12.1+cu130: | Launch | main | This PR | | --- | --- | --- | | `torchrun`, DDP wrapper | `CheckpointError` 55 vs 45, 18 layers OFF | trains, 24 layers FULL | | `accelerate` FSDP2, SFTTrainer | `CheckpointError` 55 vs 45, 18 layers OFF | trains, 24 layers FULL | Single GPU (reentrant) is unchanged: same step-0 loss, all layers FULL. Tests: - New `test_gdn_self_check_stays_out_of_a_non_reentrant_checkpoint` runs `_fla_ops` inside `checkpoint(..., use_reentrant=False)` with a stand-in self-check that calls `autograd.grad`. It fails on main with `CheckpointError` (10 vs 5) and passes here. It runs on CPU without fla. - `tests/test_block_compile.py`: 22 passed, 1 failed. `test_qwen3_5_gdn_block_has_no_break` fails the same way on main. ## Not covered The plain-attention case in the issue (`MuseGlimmerTextDecoderLayer`, 65 vs 64) and the reported slowdown.	2026-10-19 03:39:26 -07:00
Lee Jackson	2e95c7c329	fix(studio): reject oversized Metal alongside loads before startup (#14209) ... ## Summary An alongside GGUF load on Metal could reach llama-server startup even when the model and its context exceeded the remaining unified-memory budget. Metal has no discrete GPU row, so the existing partial-placement refusal did not catch this case. Check the fitted footprint against the shared-memory budget and Metal wired headroom before spawning. Report a structured memory shortfall so the existing route can evict an idle resident and retry, or refuse the load. Preserve the existing force, manual-budget, and forced-CPU behavior. Fixes #14120. ## Validation - 418 relevant backend tests passed locally. - [Apple Silicon macOS A/B run](https://github.com/Imagineer99/unsloth/actions/runs/39199324035): the same regression tests produce 7 expected failures and 4 passing controls on base, then all 11 pass on the fix. - Real GGUF under a reduced budget: base reaches startup; fix refuses before spawning with a 178 MiB shortfall. - Small-model control offloaded 19/19 layers to Metal and generated 11 tokens. Pinned base: `2fee36ffa3` Pinned fix: `f30f104da391f58762a93014063e276466437966` The runner uses virtualized Apple Silicon with explicit paravirtual Metal opt-in. This validates admission and a working load, not the reported physical M4 Max reboot. A local Apple Silicon check from the team would be useful. Disposable CI scripts are kept on the separate staging branch. --------- Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>	2026-10-19 03:37:27 -07:00
Vineeth Sai Varikuntla	1db40e54be	Pass layers_to_transform, use_rslora and modules_to_save to LoraConfig on the SentenceTransformer fast encoder path (#14150) ... `FastSentenceTransformer.get_peft_model` on the fast encoder path (BERT/MPNet-style models) built its `LoraConfig` without `layers_to_transform`, `layers_pattern`, `use_rslora` or `modules_to_save`, so they were accepted and silently ignored. `use_rslora=True` trained with alpha/r scaling instead of alpha/sqrt(r), and `layers_to_transform` still adapted every layer. Fix: pass the four arguments through, as the other branch already does. Test: new `tests/test_st_fast_encoder_lora_config.py` on a tiny BERT, CPU. On main it fails (`use_rslora` False; 13 adapters, scaling 2.0). With the fix it passes (6 adapters on layer 1 only, scaling 4.0). The other sentence-transformer tests I ran still pass. Not run on GPU. Follow-up fix: with `modules_to_save` now honoured, a model loaded in float16 (T4, V100) kept its trainable copies in float16 while the LoRA weights were float32, so the first optimizer step raised `ValueError: Attempting to unscale FP16 gradients.` The fast encoder path now calls `_unfreeze_modules_to_save`, the same helper the FastModel path uses, which upcasts float16 copies to float32 (#1200). Added `test_float16_modules_to_save_copy_trains_in_float32`, which fails without the upcast. Verified on GPU with `sentence-transformers/all-MiniLM-L6-v2` (r=8, alpha=16, layers [4, 5], rsLoRA, modules_to_save, 20 steps): base trains LoRA on every layer plus `pooler.dense` with 337920 trainable parameters at scaling 2.0; after this PR LoRA lands on layers 4 and 5 only, scaling is 16/sqrt(8), and the trainable count matches plain PEFT (111360). On a Colab T4 in float16 the original change crashed with the error above and the fixed head trains and saves a merged model. --------- Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>	2026-10-19 03:36:14 -07:00