Stacked on #14336. Merge only after the Linux torch 2.10 to 2.12
causal-conv1d / mamba-ssm cells from #14336 are published on
`prebuilt-wheels-cu13`; before that these URLs would 404 in front of the
upstream fallback.
## Why
Linux users on torch 2.10, 2.11 or 2.12 with a CUDA 13 torch download
upstream's causal-conv1d (205 MB) and mamba-ssm (351 MB) wheels, built
for torch 2.10 with `-lineinfo` and every arch. Our builds of the same
packages for torch 2.13 / 2.14 are about 29 / 54 MB on x86_64 and 42 /
79 MB on aarch64.
## Changes
- `_UNSLOTH_PREBUILT_EXTRA_CELLS` gains linux_x86_64 and linux_aarch64
torch 2.10 / 2.11 / 2.12 on cp311 to cp313.
- `_UNSLOTH_PREBUILT_EXTRA_CELL_PREFIXES` limits those Linux cells to
causal_conv1d and mamba_ssm, so flash-attn keeps upstream 2.8.1 there
(125 MB, smaller than ours).
- Unchanged: a CUDA 12 torch, cp310 and cp314 still resolve upstream,
and torch 2.13 / 2.14 and Windows resolve as before.
## Tests
Updated the tests that pinned upstream URLs for cu13 Linux torch 2.10 to
2.12. They now assert our wheel for cu13 cp311 to cp313 and upstream for
cu12 / cp310. Added tests that flash-attn stays upstream and that every
Linux extra cell the resolver asks for is one the workflow builds.
Affected suites: 1125 passed.
## Problem
Two tests on main fail in CI for reasons in the tests, not the product.
They are red on unrelated PRs, for example #14409 and #14365.
**1. `tests/python/test_v100_fullft_precision.py` (10 tests, `Repo tests
(CPU, studio)`)**
```
tests/python/test_v100_fullft_precision.py:80: in _decide
exec(CODE, {"torch": torch, "os": os}, {"args": args, "model": model})
E NameError: name 'args' is not defined
```
The test runs the `mixed_precision` template from `unsloth/models/rl.py`
with separate globals and locals dicts. #14394 added
`_unsloth_precision_requested = tuple(getattr(args, k, False) for k in
(...))` to that template. In the generated trainer this runs inside a
function, where the generator sees `args`, so the product is fine. A
generator inside `exec` with a separate locals dict cannot see those
locals, so the test raises `NameError`.
**2.
`tests/test_legacy_block_compile_install.py::test_packed_calls_keep_the_unsloth_layer_forward`
(`Repo tests (CPU, rest)`)**
```
tests/test_legacy_block_compile_install.py:105: in test_packed_calls_keep_the_unsloth_layer_forward
E assert []
```
The test (from #14398) expects an unpacked call to go through the
compiled block. `_block_compile._wrap` keeps every call on the stock
forward when `UNSLOTH_COMPILE_DISABLE` is `1`, `UNSLOTH_BLOCK_COMPILE`
is `0`, or `UNSLOTH_FORCE_FLOAT32` is `1`. Eight test modules run
`os.environ.setdefault("UNSLOTH_COMPILE_DISABLE", "1")` at import time,
so the result depends on which modules a worker collected. The test
passes alone and fails with any of those variables set.
## Fix
Test only:
- `test_v100_fullft_precision.py`: `exec` the template in one namespace,
as in the trainer's function scope. Results are still read from `args`
and the environment.
- `test_legacy_block_compile_install.py`: the packing test clears the
three variables with `monkeypatch.delenv` before wrapping.
## Verification
- `test_v100_fullft_precision.py`: on main 10 failed (the `NameError`
above); with this change 10 passed.
- `test_legacy_block_compile_install.py` on main:
- with any one of `UNSLOTH_COMPILE_DISABLE=1`, `UNSLOTH_BLOCK_COMPILE=0`
or `UNSLOTH_FORCE_FLOAT32=1` set: 1 failed (the packing test), 5 passed;
- with this change: 6 passed in each of those three cases and with none
set.
Addresses the Windows update hang reported on Reddit ([Reddit r/unsloth:
"Update install
failing"](https://www.reddit.com/r/unsloth/comments/1x7n5jq/update_install_failing/)):
it halves the prebuilt-wheel download and shows that the install is
still running. A slow connection can still take a while.
## Problem
Since #13262, Studio setup on Windows installs a 427 MB prebuilt
flash-attn wheel at step `6/17 flash-attn`. Users reported that the step
hangs, and one killed it after 30 minutes. Reporters blamed the torchao
fallback warning printed just before the step, but that warning is
unrelated: it prints on every Windows CUDA install.
Our reproduction completed, but slowly and silently. To install a wheel
from a URL, uv normally fetches only the wheel's metadata, using HTTP
range requests. For GitHub release assets those range requests fail, so
uv downloads the whole wheel just to read its metadata, discards it, and
downloads it again to install (related: astral-sh/uv#18998).
`install_wheel` captures uv's output, so the progress bar shows nothing
for the whole 2 x 427 MB. At 256 KiB/s on `windows-latest`, step 6/17
took about 60 minutes
([job](https://github.com/oobabooga/unsloth/actions/runs/38959270916/job/117169940368)).
## Fix
- `install_wheel` (`studio/backend/utils/wheel_utils.py`) supplies the
wheel's metadata to uv up front, so uv skips the metadata fetch and
downloads the wheel once. It writes the package name and version, taken
from the wheel filename, into a `uv.toml` (`[[dependency-metadata]]`) in
a temp dir, and points uv at it with `--project`. uv still installs from
the URL into its own cache, so a later reinstall reuses the cached copy.
The user's and system uv configs still apply, and relative paths such as
`UV_CACHE_DIR` still resolve against the working directory.
- `_ensure_flash_attn` (`studio/install_python_stack.py`) shows that the
install is still running. On a terminal the progress bar shows the
elapsed time; when output is captured, a `still installing flash-attn (N
min)` note is printed each minute.
Every caller of `install_wheel` gets the single download: setup,
`unsloth install-kernels`, the on-demand causal-conv1d / mamba-ssm
installs, and the training worker.
## What the user sees
In a terminal, the progress line now counts up in place, once a second,
until the install finishes:
```
deps [=======-------------] 6/17 flash-attn 0:03
```
When output is captured, from the Windows A/B below at 1 MiB/s. The `[
Ns]` prefix is the CI wrapper's elapsed-time stamp.
Before, nothing for 21 minutes:
```
[ 18s] deps [======--------------] 6/18 dependency overrides
[ 22s] torch 2.13.0+cu130 detected -- installing torchao==0.18.0 from
[ 22s] https://download.pytorch.org/whl/cu130
[ 22s] https://download.pytorch.org/whl/cu130 did not serve torchao==0.18.0 -- retrying
[ 22s] from the default index; its kernels may be skipped
[ 1,289s] deps [=======-------------] 7/18 flash-attn
[ 1,289s] deps [========------------] 8/18 Unsloth Studio deps
```
After:
```
[ 16s] deps [======--------------] 6/18 dependency overrides
[ 18s] torch 2.13.0+cu130 detected -- installing torchao==0.18.0 from
[ 18s] https://download.pytorch.org/whl/cu130
[ 18s] https://download.pytorch.org/whl/cu130 did not serve torchao==0.18.0 -- retrying
[ 18s] from the default index; its kernels may be skipped
[ 81s] deps [=======-------------] 7/18 flash-attn
[ 81s] still installing flash-attn (1 min)
[ 141s] still installing flash-attn (2 min)
[ 201s] still installing flash-attn (3 min)
[...]
[ 621s] still installing flash-attn (10 min)
[ 644s] deps [========------------] 8/18 Unsloth Studio deps
```
(The update has 18 steps instead of 17 because it adds a base-packages
step. Captured output is block-buffered, so the `7/18` line appears with
the first note.)
## Behavior boundaries
- uv runs as before when a project config would apply from the working
directory (`uv.toml`, or `pyproject.toml` with `[tool.uv`), or when
`UV_PROJECT` selects a project, because pointing `--project` elsewhere
would make uv ignore that config. It also runs as before when
`UV_CONFIG_FILE` or `UV_NO_CONFIG` is set, or when the URL has no
parseable wheel filename.
- The pip fallback is unchanged; pip already downloads once.
## Verification
Windows A/B on `windows-latest`: arm A
([run](https://github.com/oobabooga/unsloth/actions/runs/38972789221))
and two copies of arm B
([run](https://github.com/oobabooga/unsloth/actions/runs/38980373037)).
Each installs with the public one-liner, then runs `unsloth studio
update` twice, with GitHub downloads throttled to 1 MiB/s. Arm B applies
this diff first.
- Cold update: flash-attn and the Studio uv cache are deleted first, so
the update starts like a fresh install.
- Warm update: flash-attn is deleted again but the cache is kept, to
check that a reinstall still reuses uv's cached copy.
| | Cold update | Warm update |
|---|---|---|
| A, before
([job](https://github.com/oobabooga/unsloth/actions/runs/38972789221/job/117214948654))
| 856,182,074 B (2 downloads), 1,314 s, silent | 13,105 B, 44 s |
| B1, this PR
([job](https://github.com/oobabooga/unsloth/actions/runs/38980373037/job/117239567781))
| 428,140,611 B (1 download), 698 s, a note each minute | 6,938 B, 30 s
|
| B2, this PR
([job](https://github.com/oobabooga/unsloth/actions/runs/38980373037/job/117239567498))
| 428,192,144 B (1 download), 667 s, a note each minute | 6,938 B, 40 s
|
In arm B a single `uv --project <temp dir> pip install` process held the
wheel URL. One earlier arm B run of the same code
([job](https://github.com/oobabooga/unsloth/actions/runs/38977254602/job/117229573031))
moved 357 MB and then 428 MB: the download was cut off partway and uv
retried it from the start. It was not the metadata pass, because the
wheel's `METADATA` sits in its last 20 KB.
flash-attn imports afterwards in both arms.
Tests:
- New `studio/backend/tests/test_wheel_download.py` covers when the
metadata file is written and how uv and pip are invoked. It also runs
real uv against a local server that, like GitHub, does not answer range
requests: 2 downloads before, 1 after, none on reinstall, with a
relative `UV_CACHE_DIR` that must stay in the working directory. Another
checks that a project chosen with `UV_PROJECT` still applies its config.
Both pass with uv 0.11.16 and 0.13.1.
- `tests/python/test_install_python_stack.py` covers the progress output
on a terminal and when captured.
- The installer and `wheel_utils` suites pass, and `ruff check` and
`scripts/run_ruff_format.py` are clean.
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>
With only `UNSLOTH_HOME` set, the installers put Studio's venv in the
default directory, but the CLI looks in `<UNSLOTH_HOME>/studio`. Linux
and macOS finish installing, then fail to launch with `Unsloth Studio
not set up`. Windows fails during setup because it cannot find the venv.
Make both installers and setup scripts use `<UNSLOTH_HOME>/studio`.
`UNSLOTH_STUDIO_HOME` and `STUDIO_HOME` still take precedence. On Linux
and macOS, the installer writes `<UNSLOTH_HOME>/llama.cpp` to the
existing `studio/share/studio.conf`, which the launcher reads on
startup.
Windows setup reuses Python from that venv before looking for system
Python. Direct `install.sh --tauri` runs reject custom Studio paths
because the desktop cannot find them. The CLI keeps its fallback for
older installations.
Validation:
- Live A/B on Linux, macOS and Windows: reproduced the failures above;
with this change, fresh installs launch and generate text from a GGUF.
Windows logs:
[before](https://github.com/oobabooga/unsloth/actions/runs/39305114211),
[after](https://github.com/oobabooga/unsloth/actions/runs/39306042012).
- Verified launch from a new shell and after `unsloth studio update`.
Uninstall with `UNSLOTH_STUDIO_HOME` set removes the installed runtimes
and preserves unrelated files.
- 51 focused tests passed, including 22 regression cases that fail
before the fix. Native macOS and [Windows PowerShell
5.1](https://github.com/oobabooga/unsloth/actions/runs/39313291674)
tests also passed.
Live installs used `--local --no-torch`. Training was not tested.
---------
Co-authored-by: Lyxot <longyixing331@gmail.com>
Linux users on torch 2.10, 2.11 or 2.12 still get upstream's
causal-conv1d and mamba-ssm wheels: 205 MB and 351 MB downloads
(torch2.10 builds, reused on 2.11 and 2.12), with `-lineinfo` and every
arch. This lets the prebuilt wheel workflow build our trimmed versions
for those cells, on both linux_x86_64 and linux_aarch64.
## Changes
- `linux_x86_64` and `linux_aarch64` gain `extra_torches` 2.10.0 /
2.11.0 / 2.12.0, restricted to causal-conv1d and mamba-ssm by a new
`extra_torch_packages` key. flash-attn is not built there: upstream's
2.8.1 for torch 2.10 is already smaller (125 MB, no PTX) than ours.
- `_platform_packages(platform, torch)` is the one place both
`build_matrix` and the `wheel-name` CLI read, so the CLI now also
refuses a package a platform does not build (it used to print a name for
flash-attn on win_arm64).
- Release notes describe packages with different coverage separately, so
Linux does not read as having flash-attn for torch 2.10 to 2.12.
- A default dispatch builds exactly what it did before (12 cells); the
older torches stay opt-in.
## Rollout
The Studio resolver does not ask for these cells yet
(`_UNSLOTH_PREBUILT_EXTRA_CELLS` is unchanged, and the existing test
keeps it a subset of what the workflow builds). After this merges:
dispatch `platforms=linux_x86_64,linux_aarch64`,
`packages=causal-conv1d,mamba-ssm`,
`torch_versions=2.10.0,2.11.0,2.12.0`, `python_versions=3.11,3.12,3.13`
(36 cells), then a Studio change adds the Linux cells to the resolver
for those two packages only.
Expected sizes from the torch 2.13 / 2.14 Linux builds just published
from the same recipe: causal-conv1d about 29 MB (x86_64) / 42 MB
(aarch64) download, mamba-ssm about 54 / 79 MB.
## Tests
`tests/python/test_prebuilt_cuda_wheels*.py`: 256 passed. The updated
matrix and CLI tests fail on main (Linux refused every torch below
2.13); a new test covers the release notes split.
## What changed
Unsloth docs links now go to `unsloth.ai/docs` instead of
`docs.unsloth.ai`, which only 301-redirects there.
- **Studio browser new tab page:** the "Unsloth Docs" suggestion opens
`https://unsloth.ai/docs` in both the native and the proxied view.
Suggestions used to be grouped by host, and the docs and the Unsloth
home page now share `unsloth.ai`, so the docs tile would have been
dropped. Suggestions are now grouped per site, with `unsloth.ai/docs` as
its own entry. If you hid the old `docs.unsloth.ai` tile, it stays
hidden.
- **Installer messages** (`install.sh`,
`studio/install_python_stack.py`): the AMD ROCm hints now link
`https://unsloth.ai/docs/get-started/install/amd`. That is where the old
`install-and-update/amd` path ends up after the redirects. The short
`docs.unsloth.ai/amd` link redirected to `unsloth.ai/docs/amd`, which
returns 404, so it now points at the AMD guide too.
- **Docker Hub READMEs and CONTRIBUTING.md:** the docs and Docker guide
links point at `unsloth.ai/docs`.
Every new URL returns 200.
Left as is: test fixtures that use `docs.unsloth.ai` as a sample host
for address and link parsing, and a markdown security comment that uses
it as an example string. The No GPU message in
`unsloth/tokenizer_utils.py` also still uses the old link. It belongs to
the core library, so it should be changed separately.
## Verification
- `npm run typecheck`, `npm run build`, full `npm test` (12895 pass).
The new `tests/browser-docs-link.test.ts` covers the suggestion URL and
the per-site keys.
- `pytest tests/python/test_install_python_stack.py
tests/studio/install/test_rocm_support.py` (988 passed), `bash -n
install.sh`.
Not exercised here: the Desktop native browser view, and an AMD install
that prints these messages.
## Summary
Follow-up to the #13880 review.
1. **Process-global QLoRA flag.** `prepare_fsdp2_qlora` set
`unsloth.kernels.utils._FSDP2_QLORA_PREPARED = True`, and the LoRA
getters and `fast_lora_forward` read it before falling back to the
`Linear4bit`'s own `quant_state`. Dynamo guards on a global it reads.
Once one FSDP2 QLoRA model had been prepared, every compiled dense LoRA
forward in the process compiled a second graph. The fallback is now
gated on the weight itself: only an FSDP2 all-gathered 4-bit weight
loses `quant_state`, and its packed bytes are `(N, 1)`. The flag and its
setter are removed.
2. **Block compile and non-reentrant checkpointing.** A non-reentrant
checkpoint ends a recompute by raising `_StopRecomputationError` through
the forward. The block compile wrapper treated that as a compile failure
and put the stock forward back in every block. It now re-raises it.
Without FSDP2 the getters take the same path as before: `getattr(W,
"quant_state")` first, and the shape check only runs when that is
`None`.
## Tests
`tests/python/test_get_lora_parameters_fsdp2_quant_state.py`:
| Case | main | this PR |
|---|---|---|
| a compiled dense LoRA forward, before and after an FSDP2 QLoRA model
is prepared | 2 graphs | 1 graph |
| the getters read no process-global flag | fails | passes |
With the existing fp8 getter tests, `tests/test_block_compile.py`,
`tests/test_fsdp2_qlora.py` and
`tests/test_fsdp2_qlora_prepare_order.py`: 59 passed.
Under a 2-rank launch with Unsloth's block compile, Llama, Qwen3,
Qwen3-MoE, Gemma3, Gemma4 and Qwen2.5-VL show 0 graph breaks and nothing
compiling from step 2 on, LoRA and QLoRA.
## Not covered here
- torch 2.13 only so far; torch 2.10 and 2.14 runs on T4 x2 are in
progress.
- The GatedDeltaNet self-check under a distributed checkpoint is fixed
separately in #14195.
Shrinks the prebuilt causal-conv1d, mamba-ssm and flash-attn wheels
further without changing the machine code any GPU runs.
## Why
After fatbin compression the wheels are still larger than they need to
be:
- causal-conv1d and mamba-ssm compile with `-lineinfo`. It only feeds
profilers a source line table; `cuobjdump -sass` shows identical
instructions without it, and it is more than half of every cubin.
- Their setup.py hardcodes SASS for sm_87 (Jetson Orin), sm_110 (Jetson
Thor) and sm_121 (GB10 / DGX Spark). Those are integrated Arm GPUs, so
x86 wheels can never use it.
- flash-attn at our pin embeds compute_120 PTX next to the SASS. It can
only JIT for GPUs newer than sm_120; upstream 2.8.1 ships SASS only.
## Changes
1. `.github/scripts/patch_cuda_targets.py`: edits each package's
setup.py by asserting on exact text (like `patch_mamba_cxx20.py`):
removes `-lineinfo`, keeps only the requested SASS, removes flash-attn's
PTX line. Unknown or missing targets fail loudly; running it twice is a
no-op.
2. Per-platform target lists in `prebuilt_wheels.py`: x86 wheels keep sm
75;80;90;100;103;120, linux_aarch64 and win_arm64 keep the full upstream
list. flash-attn keeps sm 80;90;100;120.
3. A `targets-gate` step after the size gate reads the fatbin headers in
every built `.so` / `.pyd` (ELF and PE, no cuobjdump needed): every
requested SASS present, nothing else, no PTX. Checked against cuobjdump
on the published flash-attn Linux wheel and on the Windows causal-conv1d
wheel; both current wheels fail the new gate as expected.
4. Size gates lowered to the new expected sizes; ccache tag changed with
the target list.
## Expected sizes (installed, extrapolated from compiled translation
units)
| Package | Now | x86 after | Arm after |
|---|---|---|---|
| flash-attn | 229 MB | ~176 MB | ~176 MB |
| causal-conv1d | 117 MB | ~31 MB | ~47-53 MB |
| mamba-ssm | 241 MB | ~62 MB | ~88 MB |
Per translation unit (MB, nvcc 13.1, compression on):
| File | Now | Trimmed |
|---|---|---|
| causal_conv1d bwd (95% of the package) | 108.4 | 24.8 |
| selective_scan fwd bf16 | 14.78 | 3.36 |
| selective_scan bwd bf16 | 25.78 | 5.55 |
| flash-attn hdim128 fwd | 3.75 | 2.87 |
## Tests
`tests/python/test_prebuilt_cuda_wheels_targets.py` (new) plus the
existing prebuilt suites: 230 passed. The patch was also run against the
real pinned setup.py files.
After merge, the release needs one republish (Windows torch 2.10 to
2.12, then all platforms for 2.13 / 2.14). A machine code diff of old vs
new wheels per kept arch and a T4 / L4 kernel A/B follow the build.
## Extension-free mamba-ssm wheel
When a run builds mamba-ssm, a CPU-only job also builds
`mamba_ssm-2.3.2.post1-py3-none-any.whl` from the same pinned commit
without its CUDA extension (about 224 KB, one file for every OS / torch
/ Python). Mamba-2 hybrids need only its Triton ops once unsloth-zoo
runs their causal conv on Triton (unslothai/unsloth-zoo#1865); Mamba-1
models keep the CUDA wheel.
- `pure-gate`: exact filename, no `.so` / `.pyd` / `.dll` / `.dylib`,
`Root-Is-Purelib: true`, `Tag: py3-none-any`, then `--no-deps` install
and import of `mamba_ssm`, `mamba_chunk_scan_combined` and
`selective_state_update` in a fresh venv; the same import on
windows-latest with triton-windows.
- Sign and publish wait for it like any other planned wheel; runs
without mamba-ssm are unchanged. Release notes and SHA256SUMS list it.
- A local build at the pin is reproducible (identical sha256 across two
builds) and passes the gate. 246 prebuilt wheel tests pass.
## Measured on real builds (linux_x86_64, cp313, torch 2.13.0+cu130)
Built locally from this branch exactly as the workflow does (nvcc 13.1
here; runners use 13.0):
| Wheel | Download | Installed |
|---|---|---|
| causal-conv1d, published today | 207.1 MB | 262.8 MB |
| causal-conv1d, compressed only (current republish) | 111.3 MB | ~122
MB |
| causal-conv1d, this PR | 29.2 MB | 39.9 MB |
| mamba-ssm, published today | 349.7 MB | 359.3 MB |
| mamba-ssm, compressed only (current republish) | 233.0 MB | ~241 MB |
| mamba-ssm, this PR | 53.6 MB | 63.0 MB |
- size-gate and targets-gate pass on both (SASS exactly
75;80;90;100;103;120, no PTX); the targets-gate fails the published
wheel as expected.
- Against an untrimmed build with the same nvcc and flags, every
kernel's SASS is identical on every kept arch (causal-conv1d 1242
kernels, mamba-ssm 1440, all six archs).
- T4 and L4, published vs this PR, 6 interleaved repeats: conv forward /
backward / decode update, selective scan and the Triton SSD control all
within the repeat spread; outputs bitwise equal except selective scan dB
/ dC, which differ only at the run-to-run level of its atomic backward.
flash-attn was not built locally (6 to 8 hours); its change is the PTX
drop only, checked by the targets-gate in the workflow.
Follow up to #13861 and #13808. An audit of the merged on demand SSM
kernel change plus a T4 benchmark found a few gaps, fixed here.
## Why
- On a Kaggle T4 (sm75, torch 2.11 cu130, fp16, Qwen3.5-0.8B), Studio
chat decode was 4.4% slower without causal-conv1d (19.10 vs 19.95 tok/s,
6 alternating repeats, slower in all 9 pairs). Training speed, losses
and peak memory were unchanged. The zoo Triton conv covers
`causal_conv1d_fn` only, so single token decode fell back to the torch
update path. Training with trainable conv weights also allocates an fp32
dW scratch in the Triton backward, and packed `seq_idx` is not accepted
by the zoo dispatcher under transformers 5.20.
- Mamba-family models that have a working transformers torch path
(mamba, mamba2, falcon_mamba, jamba, zamba, zamba2, bamba) were made
fatal when mamba-ssm could not be installed.
- Export, local LoRA continuation, and inference model switching did not
provision or pick up kernels correctly once setup stopped installing
them.
- Setup on pre-Ampere GPUs (T4) downloaded flash-attn even though
Unsloth only enables FA2 at sm80+.
## Changes
1. Qwen3.5 / Qwen3.6 / Qwen3-Next request causal-conv1d on demand again
in training, inference and Clef (same as before #13861, minus the setup
time install). The zoo Triton platform probe is removed.
2. A missing mamba-ssm is fatal only for `nemotron_h`, `falcon_h1`,
`granitemoehybrid`. Other Mamba families install best effort and
continue on the torch path with a status message.
3. Local LoRA folders (`adapter_config.json`, no `config.json`) resolve
kernel needs from `base_model_name_or_path`; the training
`adapter_base_pin` wins when set.
4. The export worker provisions SSM kernels before importing Unsloth /
the export backend, with the same required vs optional split.
5. Inference re-enables the modeling source scan by reading
`transformers/models/<type>/modeling_*.py` as text, without importing
transformers.
6. If a kernel is newly installed after the model's
`transformers.models.<type>` module was already imported, the inference
worker asks for a restart and the orchestrator respawns it once.
7. Torch repair removes causal-conv1d / mamba-ssm whose compiled
extension no longer imports (never installs anything new). A hung import
check keeps the package.
8. Setup and the long context runtime install skip flash-attn below sm80
(and remove a stale copy at setup). No visible GPU and ROCm keep current
behavior.
## Tests
New and updated tests fail on main and pass here (`test_ssm_runtime.py`,
`test_ssm_restart_respawn.py`, `test_export_ssm_kernels.py`,
`test_clef_fast_paths.py`, `test_kernel_install.py`,
`test_training_worker_flash_attn.py`,
`test_ssm_kernels_install_python_stack.py`,
`test_flash_attn_install_python_stack.py`,
`test_windows_torch_flavor_invariant.py`). With
`test_install_python_stack.py` and `test_cuda_repair.py`: 966 passed
after merging current main.
## Hardening (second commit)
- Stale SSM kernel removal after torch repair needs proof: torch must
import first, and only an ImportError / OSError from the extension
itself removes the package. Signals, hangs, launch errors and a broken
torch keep it.
- `unsloth install-kernels --without-gpu` (or
`UNSLOTH_INSTALL_KERNELS_WITHOUT_GPU=1`) installs GPU gated kernels on a
GPU-less image builder. An already working kernel counts as success
before any skip, skipped kernels are listed, and the exit code is
unchanged (0) unless `--strict` is passed. The CLI respects `UV_OFFLINE`
/ HF offline variables and never falls back from uv to pip offline.
- Inference respawn refuses to spawn while the old worker survives
shutdown. SSM restart and the Xet fallback each get their own one-use
retry (3 spawns max).
- Restart detection only counts native `modeling_*` modules and the
selected checkpoint's remote code modules from its `auto_map`, matched
on the exact `transformers_modules` namespace.
- Export provisions SSM kernels inside the existing transformers
activation window (no extra Hub reachability probe), and skips
provisioning on macOS and for the MLX / CPU export backends.
- `UV_OFFLINE` counts as offline for on-demand installs (no URL probes,
source builds, or uv to pip fallback).
- Long context flash-attn install returns early when flash_attn already
imports, reuses the worker's own torch for the capability check, and
never applies the sm80 floor on ROCm.
## Install robustness (platform matrix review)
- A wheel is downloaded whenever its HEAD check is not a definite 404
(proxies that refuse HEAD no longer push required hybrids into a
failure), then import-verified; fallback only after a real failure.
- A failed install is remembered for the worker process (keyed by
kernel, torch, toolchain), so preflight and load share one attempt.
Optional Mamba families get mamba-ssm only as a prebuilt wheel, never a
source build. ROCm source builds get an 1800 s deadline and CUDA source
builds 7200 s (`UNSLOTH_SOURCE_BUILD_TIMEOUT` overrides); a timeout
kills the whole build tree.
- A runtime install whose extension does not import is uninstalled, so
transformers never finds a broken package.
- One cross-process lock per kernel and environment covers check,
install, verify and cleanup (training and runtime), with a recheck after
acquiring it.
- Only a CUDA or ROCm torch provisions anything; CPU, XPU, MPS and
unknown torch builds skip (export included).
- Optional mamba-ssm follows setup's sm80 floor (sm75 with Triton 3.4+);
the required hybrids keep the previous behavior.
- The resolver only returns URLs for cells that are actually published
(Unsloth release: torch 2.13/2.14 cp313, win_amd64 torch 2.10 to 2.12
cp311 to cp313; upstream causal-conv1d / mamba-ssm release cells), and
there is no source build of the pinned 1.6.1 / 2.3.1 sources on torch
2.13+, which need C++20.
- Training stays best effort for mamba-ssm, as before #13861.
- Export runs the malware and remote code consent gates before any
kernel provisioning.
- The extension is probed again after the install lock is acquired, so a
concurrent replacement is not treated as broken.
- The automatic HF offline window (Hub unreachable) no longer blocks
package installs; only user set offline variables do.
- `unsloth install-kernels --dry-run` prints the resolved URL while
offline.
- A Mamba model whose code needs mamba-ssm fails with a clear error on a
torch build without CUDA or ROCm.
## GPU checks (merge base vs head 4a6429a65)
Kaggle T4, torch 2.11.0+cu130, Qwen3.5-0.8B fp16:
| | base | head |
|---|---|---|
| causal-conv1d on demand | not installed | installed,
`causal_conv1d_update_kernel` engaged |
| decode tok/s (6 alternating repeats) | 18.70 | 19.41 (+3.8%, 6/6
pairs) |
| decode peak alloc | 1.748 GB | 1.748 GB |
| LoRA step time (3 repeats) | 1975.5 ms | 1970.7 ms (inside spread) |
| LoRA peak alloc | 2.338 GB | 2.338 GB |
| flash-attn at setup on sm75 | downloaded | skipped |
AMD gfx1151 (torch 2.11.0+rocm7.13.0): Qwen3.5 LoRA losses within 1.2e-4
relative (head vs merge commit bit identical), peak 2.389 GB both,
backend suites 262/262 and 275/275 passing.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Fixes https://github.com/unslothai/unsloth/issues/14031
## Problem
Every release raises the installer floors to itself
(`unsloth>=2026.10.6`, `unsloth-zoo>=2026.10.6`), and Desktop asks for
its own shipped backend (`UNSLOTH_DESKTOP_BACKEND_VERSION`). A package
index that lags PyPI (a delayed corporate feed, a mirror, an upload
cutoff) never has that release yet, so with releases every two or three
days and a 7 day feed delay the install is unsatisfiable on every run:
```
× No solution found when resolving dependencies:
╰─▶ Because only unsloth<=2026.10.3 is available and you require
unsloth>=2026.10.6, we can conclude that your requirements are
unsatisfiable.
```
Desktop then cannot launch either: even when the backend is installed by
hand, preflight sees it below the shipped version
(`desktop_backend_version_outdated`) and starts a repair, which hits the
same wall.
## Change
- `install.sh` / `install.ps1`: one dependency-free `uv pip compile`
(about 0.05 s warm) checks the floors against whatever index
configuration uv reads (env, `uv.toml`, cutoffs). On PyPI it passes and
nothing else changes. When the index cannot satisfy them, the installer
takes that index's newest `unsloth-zoo`, then the newest `unsloth` not
above it, pins that pair (`==`) and prints a warning naming both
versions. The probes force `--resolution highest` and write to their own
scratch file, so neither `UV_RESOLUTION=lowest` nor a configured
`output-file` changes the result or touches user files.
- `studio/install_python_stack.py`: the same fallback on the `studio
update` path Desktop runs, for Desktop's backend floor.
- When Desktop asked for the backend, the installer writes
`unsloth_feed_ceiling.json` (`{"requested": ..., "available": ...}`)
next to `pyvenv.cfg`, and removes it on every run where it is not
needed.
- Desktop preflight: an `outdated` backend counts as ready only when
that marker names this build's expected version and the installed
backend is at or above the marker's version. A newer Desktop, or any
other stale reason, is graded as before.
- The zoo floor literal moves into one variable per installer;
`tests/test_installer_zoo_floor_parity.py` reads it from there.
## Tests
- `tests/sh/test_relax_unsloth_floors_to_index.sh`,
`tests/studio/test_relax_unsloth_floors_to_index.ps1` and
`TestDesktopBackendVersionConstraint` (uv stubbed): floors kept when the
index has the release or is unreachable, lowered and pinned when it lags
(env index, uv.toml index, cutoff), zoo lagging unsloth, hashed resolver
output, the marker written only for Desktop and cleared once the feed
catches up, the probe scratch file removed.
- Rust: `a_feed_ceiling_marker_covers_only_the_release_it_names`; the
full `cargo test` suite passes.
- Base vs head installer on a local mirror that serves only unsloth /
unsloth-zoo 2026.10.3 ahead of PyPI, `--no-torch`, Desktop floor set: on
ubuntu-latest, windows-latest and macos-15, main fails with the error
above, this branch installs 2026.10.3, warns, writes the marker and the
backend reports itself ready.
- Same comparison with the full CUDA install on a Colab A100 and T4:
main fails, this branch installs 2026.10.3 with a working CUDA torch; a
plain PyPI install on this branch still gets 2026.10.6 and writes no
marker.
In the Studio Docker images, a skill made from the Skills dialog or by
the model's create_skill tool was saved under /root/.agents/skills,
which is inside the container and not on the unsloth-studio volume.
After `docker rm` and a new `docker run` with the same volume, the
documented way to move to a newer image, every created skill was gone
with no warning, while accounts, chats and project folders came back.
The Studio images now save the owner's skills in
/opt/unsloth-studio/skills on the volume, next to project folders and
the same layout other accounts already use for their own skills. They
are listed, edited and deleted there, and the dialog shows that path.
Installs outside Docker keep using ~/.agents/skills, and
~/.claude/skills is still read as before. The Docker Hub pages now list
skills among the data the volume keeps.
Skills made by an image from before this change stay in that old
container's /root/.agents/skills, so copy them into
/opt/unsloth-studio/skills before removing it.
Follow-up to #13265, #13395 and #13701. Each one was rechecked on other
hosts and environments, and these four gaps showed up. Each one
reproduces on `main` and is fixed here. No product code changes.
## Studio `--help` test (#13265)
`COLUMNS=200` is not enough. Typer passes an inherited `TERMINAL_WIDTH`
to Rich as an explicit width. `GITHUB_ACTIONS`, `FORCE_COLOR` or
`PY_COLORS` force terminal mode, whose colour codes split every flag
(`--model` no longer appears as a substring). On staging, the test fails
on `ubuntu-latest` and `macos-15` for exactly this reason. The test now
also sets `TERMINAL_WIDTH=200` and `_TYPER_FORCE_DISABLE_TERMINAL=1`
(both present from Typer 0.19, the floor), and it decodes the child's
output as UTF-8 so a CJK Windows code page cannot break the decode.
| Environment | `main` | Here |
| --- | --- | --- |
| none | pass | pass |
| `TERMINAL_WIDTH=80` | fail | pass |
| `GITHUB_ACTIONS=true` | fail | pass |
| `FORCE_COLOR=1` | fail | pass |
| `PY_COLORS=1` | fail | pass |
| `GITHUB_ACTIONS=true TERM=dumb` | fail | pass |
## Media scan tests (#13395)
`test_media_model_index_accounts.py` and `test_openai_catalog.py`
cleared `HF_HOME` and redirected `XDG_CACHE_HOME`. The scan also reads:
- `hf_default_cache_dir()`, which is `Path.home() / ".cache" /
"huggingface" / "hub"` and ignores `XDG_CACHE_HOME`;
- the LM Studio, Hermes, Ollama and oMLX model folders, including the
system-wide Ollama paths;
- `./models` under the working directory.
So the most common developer setup, an image model in the default HF
cache, still added rows. Both fixtures now point those helpers at the
test's temp dir and run from it. Patching the helpers rather than `HOME`
means the fix works the same on Windows (`USERPROFILE`), macOS and
Linux.
Result with a fake image model planted in each location (33 tests in
each case):
| Case | `main` | Here |
| --- | --- | --- |
| clean | 33 passed | 33 passed |
| model in `~/.cache/huggingface/hub`, `~/.lmstudio/models`,
`OLLAMA_MODELS` | 4 failed | 33 passed |
| pytest run from a folder with `models/<x>/model_index.json` | 4 failed
| 33 passed |
## Installer differential comparer (#13265)
The rule that drops a repeated 100% bar (a winget redraw) compared lines
after sizes were normalised. That made two genuinely different events
collapse into one bar and compare equal:
- a second transfer with no header line, where a partial frame sits
between the two full bars;
- two finished downloads of different sizes.
A repeat is now dropped only when no partial frame was seen since the
kept bar and the raw numbers match. Two new tests cover these cases;
they fail on `main` and pass here. The existing redraw test still passes
(94 passed).
## Node harness resolver (#13701)
#13701 lets a `const` bound to an arrow or function expression be lifted
even when some of its references cannot be. Two shapes could now
load-crash a harness:
- an eagerly evaluated const that calls such a helper on load (`const
TABLE = { v: f() }`, directly or through another helper);
- an immediately invoked `const v = function () { ... }();`, which runs
on load.
The first is now dropped like a const that reads the missing reference
itself. A const that only names the helper (`[g]`) is kept. The second
stays eager.
No current frontend source has either shape: a sweep of every top-level
declaration in `studio/frontend/src` found no harness that loads only on
one side. All 278 tests that use the resolver pass, and the two new
tests fail with the old resolver.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Stacked on #13865 (FSDP2 core); review that one first.
QLoRA under FSDP2 currently leaves every 4-bit weight replicated on each
GPU: bitsandbytes stores the packed bytes as uint8, which accelerate
excludes from sharding. This change shards them.
- When FSDP2 is requested, Unsloth quantizes with
`bnb_4bit_quant_storage` set to the compute dtype (float16 on T4,
bfloat16 elsewhere). Pre-quantized uint8 checkpoints are reinterpreted
in place as the same dtype (a view of the same bytes, no copy, same
QuantState).
- FSDP2 rebuilds the gathered weight as a plain Parameter and drops
`quant_state`. FSDPParam is wrapped so the sharded and the gathered
weight keep it, and the fused LoRA kernels fall back to the Linear4bit's
own QuantState if it is still missing.
- Mixed precision never casts packed 4-bit bytes.
- An all-gather holding packed weights always copies bytes. When all
inputs of a unit share one float dtype, FSDP2 copies them with
`torch._foreach_copy_`, which on CUDA rewrites float16 and bfloat16 NaN
bit patterns to the canonical NaN. About 1.7% of packed float16 words
were changed this way, which corrupted every 4-bit weight once LoRA was
gathered in the storage dtype.
- The FSDPParam and collective hooks are method wrappers, checked
against torch 2.6 to 2.14. If an anchor is missing, nothing is patched
and the weights stay uint8 (replicated, as before). The same happens
with accelerate older than 1.13, whose FSDP2 wrap upcasts the whole
model.
Results on 2 GPUs with a 2-rank FSDP2 run against the 1-GPU run at the
same per-forward batch, fused LoRA kernels on:
- tinyllama-bnb-4bit (2 x B200): the 4-bit weights per GPU go from 484
MB to 242 MB, and peak training memory from 1.56 GB (replicated) to 1.25
GB.
- Through the Trainer, the loss stays within 7e-4 of the 1-GPU run over
6 steps and the LoRA update cosine is 0.9999.
- The manual fully_shard run matches a 2-rank replicated run bit for
bit.
- On Kaggle T4x2 (fp16, torch 2.14), the Trainer FSDP2 QLoRA run stays
within 5e-4 of the 1-GPU loss with LoRA update cosine 0.99997, and the
packed weights per GPU are halved.
- On Kaggle T4x2 (fp16, torch 2.14) with tinyllama-bnb-4bit through the
Trainer: loss within 7.1e-4 of the 1-GPU run, LoRA update cosine
0.99989, 4-bit weights per GPU 484 MB to 242 MB and peak training memory
1.33 GB to 1.16 GB per GPU. On torch 2.9.1 a 100M model through the
Trainer stays within 1.4e-4 (cosine 1.0000).
Tests: `tests/test_fsdp2_qlora.py` (CPU, gloo, real bitsandbytes layers)
and `tests/python/test_get_lora_parameters_fsdp2_quant_state.py`.
The all-gather corruption also affects accelerate's documented bf16
FSDP2 QLoRA recipe (all-bf16 units), so it may be worth reporting
upstream separately.
Not covered: torch 2.6 to 2.8 keep uint8 storage (those versions require
one dtype per unit including frozen weights), so 4-bit weights stay
replicated there, as today; torch 2.10 to 2.12 were checked against
source only; accelerate older than 1.13 and transformers 4.57 are unit
tested only; gradient checkpointing under FSDP2 QLoRA, vision models and
8-bit were not run.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
## Problem
On a host with NVIDIA GPUs of different generations, several
process-wide decisions were made from device 0 only. Older cards later
in the list were ignored:
- **bfloat16:** `SUPPORTS_BFLOAT16` and `torch.cuda.is_bf16_supported()`
read `get_device_capability()` for the current device (`_gpu_init.py`,
`models/_utils.py`). With an RTX 3090 at index 0 and a T4 or V100 at
index 1, a model split with `device_map="auto"` gets bfloat16 on the old
card.
- **torch.compile kill switches:** the pre-Volta switch
(`device_type.py`) and the gfx906 switch read device 0 only. An sm_61
card at index 1 keeps compile on.
- **GRPO compile options:** the persistent TMA matmul and the cutlass
TMA options were enabled when device 0 is sm90+, even if another visible
card is not.
- **Capability cache:** `torch.cuda.get_device_capability` was replaced
process-wide by `functools.cache(...)`. A call with no arguments is
cached once, so after `torch.cuda.set_device(1)` every later no-argument
call, in any library, still returned device 0's answer.
The HIP branch already checks every visible GPU for bfloat16 and the
gfx10 workarounds (#7922). This brings CUDA in line.
## Fix
- **New `cuda_visible_capabilities()` in `device_type.py`:** returns the
capability of every visible CUDA device. It is guarded per device like
`hip_visible_archs()`: an unreadable card is skipped, and only a failed
device count returns `[]`.
- **New `cuda_gate_capabilities()`:** what the process-wide gates judge:
every card that could be read. That always includes the current device
when it is readable, so a gate is never less strict than the rule before
this PR, and a known old card beside an unreadable one still counts.
- **bfloat16:** enabled only when every visible card is sm80+. On a
mixed host one line says which GPUs turn it off and how to keep
bfloat16: hide them with `CUDA_VISIBLE_DEVICES`.
- **Compile kill switches:** the pre-Volta switch uses the lowest
visible major, and any visible gfx906 disables compile. The gfx906
switch now also sets the live Dynamo config, like the pre-Volta one, so
regions compiled while unsloth_zoo imported are turned off too.
- **GRPO TMA options:** require every visible card to be sm90+.
- **Capability cache:** keyed per resolved device index. No argument
means the current device. An int, `"cuda:N"`, `"cuda"` or a
`torch.device` resolve through `torch.device`, which Dynamo traces, so
`get_device_capability(Q.device)` inside a `fullgraph=True` region still
compiles. Non-CUDA or unparsable devices are passed through uncached.
- **Flash Attention 2 probe:** still uses the current device. Mixed
placements keep the existing per-placement handling from #13698.
Homogeneous hosts get exactly the same flags and messages, because
"every visible card" and "device 0" agree there.
`CUDA_VISIBLE_DEVICES=""` and `UNSLOTH_ALLOW_CPU=1` still take the
existing no-device branches.
## Tests
- **New `tests/test_mixed_nvidia_capabilities.py`:**
- spoofed capabilities for sm80x2, sm86+sm75 in both orders, sm90+sm61,
and sm61 alone;
- both bfloat16 chains, the notice and the pre-Volta switch;
- gfx906 in either position;
- `set_device(1)` after a no-argument call;
- device spellings, an unreadable card (the gates still judge every
readable card), and the GRPO options.
- **Base vs head:** 10 cases fail on main and pass here. The rest pass
on both sides, since device 0 is already the weakest card in those
shapes.
- **Related suites** (driverless import, compile options, flex
attention, xformers and fp8 gates, pre-Volta fast inference, ROCm bf16):
unchanged on both sides.
- **On a B200**, `test_capability_cache_traces_in_fullgraph` compiles
the wrapper with `fullgraph=True` for no argument, `0`, `"cuda:0"`,
`torch.device("cuda")` and `x.device`: 5 passed. The previous version of
the wrapper failed 3 of them.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The win_arm64 legs of `prebuilt-cuda-wheels.yml` stopped at "Install the
CUDA 13.4 toolkit (Windows ARM64)": the pinned sha256 of NVIDIA's
`redistrib_13.4.1.json` no longer matches (`manifest sha256 e2467758...,
pinned 29117893...`, run 38756105636).
NVIDIA re-published the manifest on 2026-10-15 (Last-Modified 16:34
GMT). Diffing it against the copy pinned on 2026-10-11, the only changes
are five windows-arm64 archives with the same versions and new hashes,
each about 10 KB smaller:
| Archive | Old sha256 | New sha256 |
|---|---|---|
| libcufft 12.4.0.34 | f5f9d4cb... | fdc54525... |
| libcurand 10.4.4.49 | 74430277... | 252c1fbb... |
| libcusolver 12.3.2.15 | 84e43a78... | 5e2fc326... |
| libcusparse 12.8.6.49 | 5bd281e5... | 91d0d125... |
| libnpp 13.2.0.35 (not installed by the workflow) | 95e1c497... |
49836103... |
Every other entry, including nvcc, libnvvm, cudart and cccl, is
unchanged. This updates the pin in both jobs and its test.
`prebuilt_wheels_cuda_redist.py` with the new pin installs `cuda_nvtx`
and the re-published `libcurand` with their per-archive hashes verified.
Studio on Windows x64 with torch 2.12 now installs flash-attn,
causal-conv1d and mamba-ssm from our prebuilt wheels, as it does for
2.10 and 2.11.
Follows #13807 (the workflow change that builds them). Merge after the
torch 2.12 `win_amd64` wheels are published on `prebuilt-wheels-cu13`;
before that, these URLs would 404.
## What changes
`studio/backend/utils/wheel_utils.py`: `("win_amd64", "2.12")` joins the
extra published cells, for cp311, cp312 and cp313, e.g.
`flash_attn-2.8.4+cu13torch2.12cxx11abiTRUE-cp312-cp312-win_amd64.whl`.
Unchanged: Linux 2.12 keeps reusing upstream's torch 2.10 wheels;
Windows 2.10 / 2.11 / 2.13 / 2.14 as before; Windows 2.9, CUDA 12
builds, free-threaded builds, other interpreters and `win_arm64` resolve
to nothing.
## Tests
The cases that used Windows 2.12 as "no wheel" move to 2.9; Windows 2.12
resolution for each interpreter and kernel, plus `kernel_install`; the
workflow/resolver agreement check covers all 27 extra cells. The
resolver, kernel_install and prebuilt wheel test files pass (388).
Fixes https://github.com/unslothai/unsloth/issues/5405
GGUF chat in the AMD Studio image (`unsloth/unsloth-rocm`, the `studio`
tag and `-studio` tags) ran on the CPU. Training, the notebooks and
PyTorch inference already used the GPU, but `Dockerfile.studio-rocm`
installed the CPU llama.cpp bundle, because a ROCm bundle is built per
gfx family and would pin the image to one card.
This installs the Vulkan llama.cpp bundle instead, run by Mesa's RADV
driver. Vulkan is not tied to one architecture, the `/dev/dri` node it
opens is one ROCm already needs, and it is the backend Studio picks by
itself on the Strix APUs outside Docker.
- `libvulkan1` + `mesa-vulkan-drivers` (RADV). No `VK_DRIVER_FILES` pin:
ggml already drops lavapipe as a CPU device, and an image-wide pin would
override a user's own `VK_ICD_FILENAMES`.
- `UNSLOTH_LLAMA_CPP_BACKEND=vulkan` for `install.sh` in place of `cpu`,
plus a build-time assert that the installed marker says `vulkan`.
- Docker Hub page text updated, including that Vulkan does not read
`HIP_VISIBLE_DEVICES` (on a multi-GPU host, pick the chat GPU in Studio
or pass only that card's render node). The CUDA images, the ROCm
training image (`latest` built from `Dockerfile.rocm`) and the WSL2 DXG
path are unchanged.
## Tested on a Strix Halo (Radeon 8060S, gfx1151), Linux, Docker
The head image was built on the runner from this branch's `docker/`, on
the same published base digest (`sha-133a807`) and the same Studio
commit as the published `sha-133a807-studio`, which is the base arm.
Only `Dockerfile.studio-rocm` differs. Each arm is a fresh container
with the docs' flags (`--device /dev/kfd --device /dev/dri` plus numeric
group ids), Studio started by the image's own launcher,
`unsloth/Qwen3-4B-Instruct-2507-GGUF` Q4_K_M loaded through
`/api/inference/load`, then three 256-token chats through
`/v1/chat/completions`, in the order base, head, head, base.
| | base (published) | head (this PR) |
|---|---|---|
| llama.cpp bundle | `linux-x64-cpu`, `Available devices: (none)` |
`linux-x64-vulkan`, `Vulkan0: Radeon 8060S Graphics (RADV GFX1151)` |
| llama-server placement | `-ngl -1 --fit off` | `-ngl -1 --fit off
--device Vulkan0` |
| model weights | `CPU model buffer size = 692.63 MiB`, `CPU_REPACK ...
= 1683.28 MiB` | `Vulkan0 model buffer size = 2375.91 MiB` |
| GPU memory in use (GTT) | 43 MiB | 3342 MiB |
| chat tok/s, run 1 (3 prompts) | 38.3 / 38.8 / 38.8 | 69.5 / 69.0 /
73.3 |
| chat tok/s, run 2 (3 prompts) | 38.6 / 39.0 / 39.0 | 71.6 / 70.8 /
72.6 |
tok/s is wall time per request including prefill, with Studio's default
settings. The answers start with the same text on both sides.
Build: `llama.cpp b11541-mix-2500203
app-b11541-mix-2500203-linux-x64-vulkan.tar.gz` from the new assert, and
`Vulkan selected for GGUF inference; the PyTorch training backend is
unchanged` from `install.sh`. The image grows by about 245 MB
uncompressed.
A second build at this head (no ICD pin) on the same runner covered the
fallback and the upgrade path, same model and prompts:
| | result |
|---|---|
| head with RADV disabled (`VK_LOADER_DRIVERS_DISABLE=*radeon*`,
lavapipe still installed: how a CDNA host looks to llama.cpp) |
`Available devices: (none)` (ggml drops the CPU rasterizer), weights in
`CPU` / `CPU_REPACK` buffers, chat works at 38.6 / 39.0 / 39.1 tok/s, no
error |
| Studio volume created by the published image, then the head image
started on it | the home relinks to the new bundle: `--device Vulkan0`,
`Vulkan0 model buffer size = 2375.91 MiB`, 70.0 / 70.7 / 72.8 tok/s |
| head, plain | `using device Vulkan0 (Radeon 8060S Graphics (RADV
GFX1151))`, 70.5 / 70.6 / 71.6 tok/s |
Tests: `tests/python/test_docker_rocm.py` gains two checks on the
Dockerfile (Vulkan backend plus the marker assert, the RADV packages);
both fail on `main`. Every test file that reads the changed files passes
(123 tests).
Not covered on real hardware: discrete RDNA cards, CDNA (RADV does not
drive it, so llama.cpp sees no Vulkan device and stays on the CPU, the
case simulated above), and the WSL2 DXG path, which has no `/dev/dri`.
The Studio regression switchboard (auth, settings, page crawl) at this
head: NO_REGRESSION, 36 same, 3 DOM-only with 0 changed pixels (no
Studio source differs).
Studio setup stops downloading causal-conv1d and mamba-ssm. They are
installed when a model that needs them is first trained or loaded, and
Qwen3.5 / Qwen3.6 / Qwen3-Next no longer request causal-conv1d at all.
## Why
#13396 added both kernels to every NVIDIA Linux / Windows setup next to
flash-attn: about 555 MB of downloads (~620 MB installed) for every
user, while only a few model families use them. Studio already installs
them on demand when such a model is trained (`core/training/worker.py`)
or loaded for chat (`core/inference/worker.py` through
`utils/ssm_runtime.ensure_ssm_runtime`), so the setup step only
duplicated that.
Qwen3.5 / 3.6 / Qwen3-Next do not need causal-conv1d with Unsloth:
unsloth-zoo runs their GatedDeltaNet with its vendored FLA kernels and
its own Triton causal conv
(`unsloth_zoo/temporary_patches/gdn_causal_conv1d.py`, model types
`qwen3_5`, `qwen3_5_moe`, `qwen3_next`). Studio still downloaded the
CUDA package for them because their transformers modeling files mention
`causal_conv1d`.
## What changes
- `studio/install_python_stack.py`: the setup-time SSM install is gone
from all three places (initial setup, Linux torch repair, Windows torch
flavor repair), with its progress step. flash-attn stays where it was.
`unsloth install-kernels` still installs every kernel, for offline
images.
- `studio/backend/utils/ssm_runtime.py`: `ZOO_GDN_MODEL_TYPES` (equal to
zoo's list; a test reads zoo's source to keep them equal). Names and
configs of those models, including `_text` / `_vision` sub-configs,
resolve to no causal-conv1d request, checked before the modeling-source
scan. A config that mixes in another model type that needs it still asks
for it. Zoo's own kill switch `UNSLOTH_DISABLE_TRITON_CAUSAL_CONV1D=1`
brings the old request back.
- `core/training/worker.py` and `core/systemone/clef_runtime.py` use the
same predicate, so the eager install, the `is_causal_conv1d_available`
hook and Clef (a Qwen3.5 backbone) all agree.
- Only where zoo's Triton conv can run: a CUDA torch (not ROCm, not
CPU), Triton importable, not macOS. This is read from torch's
distribution version (`+rocm` / `+cpu` / `+cu`), so no load imports
torch to decide. On ROCm, macOS or Windows without triton-windows, Qwen
keeps getting causal-conv1d as before.
- With setup no longer covering for it, the on-demand install now
decides from the model's config, not only its name:
`resolved_model_is_ssm` maps `nemotron_h`, `falcon_h1`,
`granitemoehybrid`, `bamba`, `jamba`, `zamba`, `zamba2`, `mamba`,
`mamba2`, `falcon_mamba` (and their `_text` sub-configs) to mamba-ssm +
causal-conv1d, for training and for the inference load of the resolved
base. Previously `NVIDIA-Nemotron-Nano-9B-v2`,
`granite-4.0-tiny-preview`, renamed local checkpoints, Jamba, Zamba and
Bamba never got mamba-ssm on demand.
- After an install inside an inference worker that already imported
transformers, the cached `is_causal_conv1d_available` /
`is_mamba_ssm_available` / `is_mamba_2_ssm_available` and cached
hub-kernel misses are cleared, so switching from Qwen3.5 to Nemotron-H
in one session gets the fast path.
- The mamba-ssm error raised when it cannot be installed (e.g. offline)
says to run `unsloth install-kernels` once while online.
- `install_kernel(install_without_gpu=)` is gone; setup was its only
caller.
- Clef still applies the FLA TileLang guard when it skips the install.
Unchanged: LFM2, Mamba / Jamba / Zamba / Bamba still get causal-conv1d
on demand; Nemotron-H, Falcon-H1 and Granite-4.0-H still get
causal-conv1d then mamba-ssm on demand; nothing already installed is
removed, and a usable causal_conv1d still wins inside zoo.
## Tests
- `tests/python/test_ssm_kernels_install_python_stack.py`: setup has no
SSM step; flash-attn is still installed at all three sites; the CLI
kernels are still there.
- `test_windows_torch_flavor_invariant.py`: the torch progress totals
drop by one on Windows and Linux.
- `studio/backend/tests/test_ssm_runtime.py`: Qwen3.5 / 3.6 / Next names
and configs install nothing, the kill switch restores the request, a
mixed config still asks, the Mamba2 hybrids still install both kernels
in order, and the model-type list equals zoo's.
- 21 of the new cases fail against main, and the follow-up cases
(config-based SSM detection, platform check, cache refresh, offline
message) fail against this PR's first commit; all pass here. The touched
test files and the setup / resolver suites pass (test_ssm_runtime 84,
test_training_worker_flash_attn 67, test_install_python_stack 295,
test_cuda_repair 284, test_windows_torch_flavor_invariant 54, among
others).
## Accuracy and speed without causal-conv1d (Qwen3.5)
Tiny Qwen3.5 / Qwen3-Next through Unsloth, bf16: zoo's Triton conv is
bit-identical to torch's reference conv (logits, loss, LoRA grads, 5
optimizer steps); the CUDA package differs from both at bf16 rounding
(5.9e-3 logits, grads cos 0.99993). Prefill plus cached token-by-token
decode matches a full forward equally with and without the package
(4.0e-3 vs 3.9e-3). The profiler shows which kernels ran in every arm.
Qwen3.5-0.8B bf16 on a Colab L4, torch 2.14.1+cu130, 3 alternating
repeats per arm:
| | Without causal-conv1d (zoo Triton) | With causal-conv1d (CUDA) |
|---|---|---|
| LoRA training, batch 4 x 1024, 30 timed steps | 911.5 ms/step (spread
11.2) | 917.5 ms/step (spread 15.9) |
| Greedy decode, 256 new tokens, batch 1 | 19.3 tok/s (spread 0.6) |
19.0 tok/s (spread 0.2) |
Both differences are inside the run-to-run spread. Without the package,
single-token decode uses torch's `conv_depthwise2d`; with it,
`causal_conv1d_update`.
A tiny NemotronH also runs identically with a pure-Python mamba-ssm (the
Mamba2 path never calls `selective_scan_cuda`), and a merged 16-bit
export of a NemotronH LoRA works with neither kernel installed.
## Not covered here
- `test_decision_native.py` / `test_decision_serving.py` were not
collected locally (missing `fastmcp`).
Our prebuilt flash-attn wheels are 431 MB to download and 1.49 GB
installed; upstream's 2.8.1 wheel for the same GPU targets is 125 MB /
141 MB. causal-conv1d (263 MB installed) and mamba-ssm (359 MB) have the
same problem. This compresses the device code in all three.
## Why they are so large
- `flash_attn_2_cuda.so` in our torch 2.13 cp313 Linux wheel is 1,488
MB, of which the `.nv_fatbin` section is 1,466 MB. Upstream 2.8.1's is
139 MB with a 121 MB fatbin.
- In default mode flash-attn's cubins went into the fatbin uncompressed.
Upstream 2.8.1 passed `-Xfatbin -compress-all -compress-mode=size`; the
commit we build from (`edb5c76`, the one with the c++20 switch torch
2.13 needs) no longer does.
- causal-conv1d's and mamba-ssm's large cubins are already
zstd-compressed in default mode, so they gain less: repacking the
released cubins in size mode takes `causal_conv1d_cuda` from 248 to 103
MB and `selective_scan_cuda` to about two thirds.
- The zip only gets 3.4x out of our wheel (1,488 -> 431 MB), while
upstream's barely shrinks (139 -> 125 MB): already-compressed data.
One translation unit each, rebuilt locally with nvcc 13.1 and the same
gencode lists:
| TU | No flags | `-Xfatbin -compress-all --compress-mode=size` |
|---|---|---|
| flash-attn `flash_fwd_hdim128_bf16_sm80` (sm_80/90/100/120 +
compute_120 PTX) | 18.80 MB | 3.55 MB |
`balance` / default mode with `-compress-all` gave 4.84 MB and `speed`
7.63 MB for the flash-attn TU. Every variant loaded on a B200 with all
kernels forced to load, and the compressed module did not load slower.
## What changes
- `.github/scripts/prebuilt_wheels.py`: `NVCC_APPEND_FLAGS = "-Xfatbin
-compress-all --compress-mode=size"` and `CCACHE_TAG`, carried into
every matrix cell; a per-package `max_extension_mb` (flash-attn 450,
causal-conv1d 160, mamba-ssm 300) and a `size-gate` subcommand that
reads the extension sizes from the wheel's zip directory.
- `.github/workflows/prebuilt-cuda-wheels.yml`: the warm and build
compile steps append the flags to `NVCC_APPEND_FLAGS` (keeping the
`_USE_MATH_DEFINES` Windows already puts there; nvcc reads the variable
itself, so no setup.py is patched), and the build step fails if they are
missing. ccache does not hash that variable and the Linux runners'
ccache 4.5 has no namespaces, so the tag is part of the actions/cache
key and no cache from before the flags is restored. A "Check the
extension size" step runs before the smoke test, so an uncompressed
wheel is never signed or published.
- The compute_120 PTX the pinned setup.py adds for forward compatibility
is kept.
`--compress-mode=size` needs a CUDA 12.4+ driver to load; every cu13
wheel already needs a CUDA 13 driver.
## Expected sizes
Estimates, not yet measured on full wheels: flash-attn ~285 MB installed
(from 1,488, extrapolated from one TU), causal-conv1d ~105 MB (from 263)
and mamba-ssm ~230 MB (from 358), both from repacking the released
cubins. The republish runs will give the real numbers.
## Tests
`TestFatbinCompression` in `tests/python/test_prebuilt_cuda_wheels.py`:
every cell on all four platforms carries the flags and none go through
`build_env`, both compile steps append them, the cache keys carry the
tag, the size gate sits before the smoke test, and it passes, fails and
rejects a wheel with no extension. The tests also cover every job's
cache steps and that the gate step really runs the gate. The three
prebuilt wheel test files pass (203).
* Studio Desktop: choose where Unsloth installs on Windows
* Desktop install location: commit under the install lock, keep first-install logs out of the chosen folder, and let the uninstaller find it
* Desktop install location: safe write probe, keep a quarantined install locked, canonical sensitive root, private ACL on a new chosen root
* Desktop install location: load the location row on use, keeping it out of the startup bundle
* Desktop install location: trim comments
* Desktop install location: refuse a derived root that already holds someone else's files
* Desktop install location: support reports also read the default folder's logs while a chosen folder is in use
* Desktop install location: redact the chosen folder in support reports even before it exists
* Desktop install location: re-check the chosen folder when the install starts
* Desktop install location: trim a comment
* Run CI on the converged head
* Desktop install location: import the location row statically and raise the startup budget by its growth
[skip ci]
* Run CI on the converged head
Add 2.12.0 to win_amd64's extra_torches (Python 3.11-3.13, flash-attn, causal-conv1d,
mamba-ssm). Linux reuses upstream's torch2.10 wheels on 2.12, which have no Windows
build. torch 2.12.0 pins triton 3.7.0, so the smoke test uses triton-windows 3.7.0.post26.
A default dispatch still builds only 2.13.0 and 2.14.0.
* Build Windows x86_64 prebuilt CUDA wheels for torch 2.10
Add 2.10.0 to win_amd64's extra_torches (Python 3.11-3.13, flash-attn, causal-conv1d,
mamba-ssm) with triton-windows 3.6.0.post26 for the smoke test, since torch 2.10 pins
triton 3.6.0 like 2.11. Upstream's torch2.10 wheels have no Windows build. A default
dispatch still builds only 2.13.0 and 2.14.0.
* Resolve Windows x86_64 torch 2.10 kernels to our prebuilt wheels
Existing Windows installs on torch 2.10 (cu130) got no flash-attn, causal-conv1d or
mamba-ssm wheel, since upstream's torch2.10 wheels are Linux only. Point win_amd64
torch 2.10 on Python 3.11-3.13 at the prebuilt-wheels-cu13 release, like 2.11.
* fix: align Python installer warnings with amber palette
* Test installer warnings use the shell installer warning color
---------
Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>
* Keep the Docker notebooks and the models they save after docker rm
* Give run.sh --rocm its own workspace volume
* Mount the workspace volume in the core Jupyter example
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Run the ROCm smoke test without the workspace volume so a rebuilt image tests its own copy
* Mount the workspace volume in the CPU-only and ROCm quick starts
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>
Add 2.10.0 to win_amd64's extra_torches (Python 3.11-3.13, flash-attn, causal-conv1d,
mamba-ssm) with triton-windows 3.6.0.post26 for the smoke test, since torch 2.10 pins
triton 3.6.0 like 2.11. Upstream's torch2.10 wheels have no Windows build. A default
dispatch still builds only 2.13.0 and 2.14.0.
* Keep the pad embedding row when the tokenizer emits the pad token
* Share one pad-row zeroing helper and never zero an EOS row
* Tighten pad-row comments
* Prebuilt CUDA wheels: torch 2.11 for win_amd64
Linux torch 2.11 reuses upstream's torch2.10 wheels, which have no Windows build, so a
Windows install that keeps torch 2.11+cu130 gets no flash-attn, causal-conv1d or mamba-ssm.
2.11.0 is accepted from a dispatch and built only on win_amd64 (triton-windows 3.6.0.post26
for the mamba smoke test); a default dispatch is unchanged. Release notes list torch minors
with different interpreter sets separately.
* Studio: resolve Windows torch 2.11 kernels to our prebuilt wheels
win_amd64 torch 2.11 + cu13 on cp311, cp312 or cp313 now resolves flash-attn, causal-conv1d
and mamba-ssm to prebuilt-wheels-cu13, named exactly as the workflow names them. Linux 2.11
keeps upstream's torch2.10 reuse; Windows 2.12, other interpreters and win_arm64 2.11 still
resolve to nothing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks"
This reverts commit b353b15b710d5ce291055b4d8699c31a631f03c9.
* Tests: read the built torches through _platform_torches
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: train, export and chat with text-diffusion LMs
* Studio: resolve a loaded diffusion model's profile from its config
* Studio: end a cancelled text-diffusion chat without an error
* Studio: refuse merged export for every text-diffusion model and catch a diffusion model the config probe missed
* Studio: load remote-code text-diffusion checkpoints with the transformers 5.5 sidecar
* Studio: log when chat loads a text-diffusion model
* Studio: classify exports from the loaded model, neutralize chat markup for diffusion, keep max_length on unpatched TRL
* Studio: offer the latest-transformers install for DiffusionGemma when the 5.5 sidecar does not ship it
* Studio: format datasets in one process when a remote-code tokenizer cannot reach the worker processes
* Studio: tokenize in one process when a remote-code tokenizer cannot be pickled for dataset workers
* Studio export: skip the diffusion refusal lookup when no diffusion model is loaded
* Trim redundant comments in text-diffusion Studio changes
* Studio diffusion: route LLaDAMoEModel to the 5.5 tier, keep the CPT embedding learning rate, refuse MLX training of text-diffusion models
* Studio diffusion: resolve CPT embedding names to the model's own modules, refuse MLX chat for text-diffusion models, turn layer offloading off before the VRAM budget
* Studio diffusion: import the cancellation error from diffusion_samplers, refuse an autoregressive load in diffusion chat, resolve Hub LoRA bases before the MLX refusal, keep the denied-credential sentinel in detection
* Studio diffusion: clear stale generation stats in diffusion chat, probe export limits with the caller's credential
* Skip the torch-backed text-diffusion Studio tests where torch is not installed
* Count the text-diffusion loader in the observer test, skip torch-backed diffusion Studio tests without unsloth_zoo
* Studio diffusion: prompt for latest transformers before the 5.5 sidecar exists, classify a Hub LoRA by its base before the vision loader
* Studio diffusion: lower the default LoRA learning rate to 5e-5 for Dream and SDAR
Their output-head rows for <|im_end|>, <tool_call>, <|fim_pad|> and the other special tokens are near-duplicates,
so at 2e-4 a LoRA run swaps them: replies end in <tool_call> or come back empty. Clean at 5e-5. Only the
untouched Studio default is lowered; any other value is kept.
* Tighten Studio diffusion comments
The 2025 Unsloth Mistral Small 3.2 chat template only emitted [AVAILABLE_TOOLS]
when the conversation ended on a user or tool turn, so every fine-tuning row
ending in an assistant reply silently lost the tool list. Rewrite that template
at load time to place the tool list before the last user message, matching
mistral_common in both serving and finetuning mode. Serving prompts are
unchanged; other templates are left untouched.
Fixes https://github.com/unslothai/unsloth/issues/3096
* Prebuilt CUDA wheels: torch 2.11 for win_amd64
Linux torch 2.11 reuses upstream's torch2.10 wheels, which have no Windows build, so a
Windows install that keeps torch 2.11+cu130 gets no flash-attn, causal-conv1d or mamba-ssm.
2.11.0 is accepted from a dispatch and built only on win_amd64 (triton-windows 3.6.0.post26
for the mamba smoke test); a default dispatch is unchanged. Release notes list torch minors
with different interpreter sets separately.
* Prebuilt wheels: win_amd64 inherits TORCH_VERSIONS, wheel-name refuses unbuilt cells
- win_amd64 adds 2.11.0 on top of TORCH_VERSIONS instead of a copy of it, so a
new global torch is not silently skipped on Windows.
- wheel-name refuses a (platform, torch) cell build_matrix refuses, e.g. 2.11.0
on the linux_x86_64 default.
- The dispatch form says 2.11.0 builds on win_amd64 only.
* Make UNSLOTH_STUDIO_KEEP_LEGACY=0 work through docker/run.sh
* Trim comments in PR scope
---------
Co-authored-by: mahiatlinux <mahiatlinux@users.noreply.github.com>
* Keep uv, its cache and the uv-managed Python inside a custom UNSLOTH_STUDIO_HOME
* Prefer the home's uv, keep its Python store at runtime and on update, and only isolate a usable cache
* Give the home's uv its own directory, keep it first on PATH, pass the Python store to engine installs
* Tighten comments
* Clean-machine Mach-O scan: require uv to be classified wherever it lives
* Read the private uv directory safely when Install-UvFromRelease runs under strict mode
* Opt in at runtime only where the custom-home installer created <home>/python and <home>/uv
* Claim the custom root before creating python/ or uv/ in it; replace a stale private uv
* Update: let setup version-check and repair the home's uv in place instead of putting it first on PATH
* install.ps1: let the stale flag reach Install-UvFromRelease past the inner version guard
* Leave main's files as main has them
* Windows installer: always build Studio on a uv-managed Python, never one already installed
A fresh install on a machine with Python 3.12 took that interpreter, so the
cu130 torch 2.13 route (cp313 only) closed and the box got torch 2.11 with
no prebuilt flash-attn / causal-conv1d / mamba-ssm. uv only provided Python
when nothing 3.11-3.13 was installed, and Windows on ARM never used it.
The environment is now always built on a uv-managed CPython, requested by
minor and architecture (cpython-3.13-windows-x86_64-none, or aarch64 for the
native Windows on ARM CUDA stack and UNSLOTH_ALLOW_ARM64_PYTHON), resolved
after uv is installed and before the existing environment is touched. A
reinstall keeps the environment's minor so its cached wheels are reused;
UNSLOTH_TORCH_UPGRADE=1 moves it to 3.13. When uv cannot provide one, an
earlier uv-installed runtime is used, else the installer stops with the
mirror / pre-install fix; there is no system fallback. --python X.Y[.Z] now
works as the header always said, and an explicit request is exact. Legacy
layouts on a non-uv base are rebuilt instead of moved in.
* Tests: drop the system-Python bootstrap cases; winget now only installs uv
* Legacy environments are reused only on the Python this install resolved
A legacy .venv / ~/unsloth_studio on a uv-managed base was moved in and kept
even when --python, UNSLOTH_PYTHON or UNSLOTH_TORCH_UPGRADE asked for another
version. It is now reused only when its pyvenv.cfg version matches the
resolved interpreter's minor (its exact patch when one was requested).
* Trim comments
* Windows installer: legacy layouts keep their minor and arch, UTF-8 pyvenv.cfg, older cached patch past a skip
- A legacy .venv / ~/unsloth_studio reinstall keeps its Python minor, as the
current layout does.
- A legacy environment is reused only on a uv build of the selected
architecture.
- pyvenv.cfg is read as UTF-8 (uv writes it without a BOM; Windows
PowerShell 5.1 read it as ANSI and rejected a non-ASCII store path).
- Past a skipped patch the resolver tries the next one, then an older one,
so an offline machine with 3.13.7 cached still installs.
* Trim comments
* Studio setup: install causal-conv1d and mamba-ssm prebuilt wheels next to flash-attn
Setup already installed flash-attn from prebuilt-wheels-cu13 but left
causal-conv1d and mamba-ssm to the first Mamba / hybrid model load. Install
both at setup through the same wheel-only install_kernel the CLI and SSM
runtime use: never a source build, never a torch change, skipped where no
wheel is published, GPU-gated for mamba-ssm, and resynced wherever
flash-attn is after a torch repair. UNSLOTH_STUDIO_SKIP_SSM_KERNELS_INSTALL=1
opts out.
* SSM kernel setup step: local check first, bounded probes, fresh runner, no-GPU install, stale removal
Skip the wheel lookup when the resident kernel already loads, so an offline
re-run does not wait on two HEAD requests. Each call gets its own runner,
which bounds the import probes and resets install_kernel's per-runner GPU
and Triton cache after a torch repair. Setup installs mamba-ssm where no GPU
is visible yet (an image build); a visible pre-Ampere GPU still skips it. A
resident build that no longer loads and has nothing replacing it is removed,
as flash-attn's resync does.
* SSM kernel setup step: honour UV_OFFLINE
No wheel lookup and no install when uv is told to stay offline: the pip
fallback in install_wheel does not read UV_OFFLINE. A kernel that already
loads is still reported.
* SSM kernel setup step: offline still removes a stale build; a probe that cannot launch is inconclusive
* SSM kernel setup step: bound the torch probe on every platform and skip on a launch failure
* SSM kernel setup step: an uninstall that cannot launch warns instead of aborting
* Trim comments
* SSM kernel setup step: redact captured install output before printing it
install_kernel only replaces the wheel URL; uv / pip failure text or an
exception can name another index carrying credentials.
* Tests: pin the no-GPU mamba-ssm case to Linux so Windows' Triton gate does not answer first
* Studio: document the intentional torch and torchcodec floor gap
* Test both shell floors and patch-version constraints
* Tighten torch 2.4 torchcodec notes and check every install.sh floor
---------
Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>
* Studio: install prebuilt flash-attn, causal-conv1d and mamba-ssm wheels on Windows
Windows x64 resolves to our win_amd64 builds on prebuilt-wheels-cu13 (upstream
naming, cxx11abiFALSE, cp313, torch 2.13 / 2.14 cu13). Windows on ARM resolves
flash-attn to NVIDIA's Flash-Attention-for-WoA wheel for torch 2.14, pinned by a
#sha256= fragment that uv and pip both verify.
Windows is wheel-only everywhere: no upstream name is ever built (no upstream
release has a Windows asset) and a missing or rejected wheel never falls back to a
PyPI source build. Setup, long-context training, the causal-conv1d fast path and
its lazy hook, SSM inference and install-kernels all take the Windows path;
mamba-ssm on Windows first requires an importable triton. Linux URLs are unchanged.
* Windows wheels are cxx11abiTRUE: the Windows cu130 torch reports the flag as True
* Bound the Windows flash-attn setup probe like its import probe
* install-kernels: Windows mamba_ssm waits for Triton; rejected-wheel log says what Windows does
* Trim comments in the Windows kernel wheel install paths
* Re-check flash-attn when the Windows torch flavor invariant replaces torch
* Resolve the linux_aarch64 and win_arm64 prebuilt kernel wheels
* Build the prebuilt CUDA 13 wheels for Windows (win_amd64)
Dao-AILab and state-spaces have never published a Windows wheel of
flash-attn, causal-conv1d or mamba-ssm, so prebuilt-cuda-wheels.yml now
builds win_amd64 cells next to the Linux ones, for the same torch 2.13 /
2.14 cu130 and cp313 set.
- prebuilt_wheels.py: PLATFORMS table and an allowlisted `platforms`
dispatch input. Windows names follow each upstream setup.py
(`+cu13torch2.13cxx11abiFALSE-cp313-cp313-win_amd64.whl`). Linux cells
keep every existing value and only gain keys. gpu_matrix keeps the
self-hosted GPU job on Linux cells. Release notes describe each
platform; a new `sums` subcommand writes SHA256SUMS.
- prebuilt_wheels_shard.py: understand Windows `.obj` targets with drive
letters, fail on an empty object inventory, and never run a
targetless ninja for an empty slice.
- prebuilt_wheels_timeout.py: deadline wrapper that ends the whole
process tree (taskkill /T on Windows), used by the Windows legs.
- Workflow: warm and build run on the cell's runner. Windows setup uses
NVIDIA's CUDA 13.0.2 network installer (Authenticode checked, explicit
subpackages, C:\cuda junction), vcvars64 via vswhere, a sha256-pinned
ccache, and a memory/disk report. Artifact names and cache keys carry
the platform. The Windows smoke test hides the toolkit and installs
triton-windows pinned to the torch minor for mamba-ssm.
- Publish: SHA256SUMS is rebuilt over every wheel on the release, so a
one-platform publish no longer drops the other platform's entries.
* Windows wheels: cxx11abiTRUE as torch reports, and a 16-32 GB page file on D:
* Windows wheels: define M_LOG2E for MSVC, patch mamba's in-macro USE_ROCM block, verbose ccache stats
* Windows wheels: put MSVC's link.exe ahead of Git Bash's coreutils link in the build step
* Windows wheels: WHEEL_PLATFORM, since vcvars' Platform=x64 shadows PLATFORM case-insensitively
* Trim comments in the Windows prebuilt wheel pipeline
* List release packages per platform in the prebuilt wheel notes
* Build the prebuilt kernel wheels for linux_aarch64 and win_arm64
* Use MSVC's conforming preprocessor for the win_arm64 builds (CUDA 13.4 CCCL requires it)
* Link win_arm64 extensions against CUDA's lib\arm64 through the lib\x64 path torch uses
* Test the win_arm64 lib\x64 junction
* fix(gkd): pass GradScaler to the chunked JSD loss
Co-authored-by: Codex <noreply@openai.com>
* Test that GKD passes an enabled GradScaler to the chunked JSD
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>
* Studio installer: drop a flash-attn built for a torch that is gone
The Linux torch repair (step 13) runs after the flash-attn step. When it moves
torch, e.g. from 2.12+cu130 to 2.9+cu126 on a CUDA-12 driver, the cu13torch2.10
wheel the earlier step installed can no longer import. The repair re-selected
torchao and evicted a mismatched xFormers, but never looked at flash-attn.
Later passes did not heal it either: _ensure_flash_attn only removed a copy it
had just installed itself. With no wheel for the new torch (flash-attn v2.8.3
ships no cu12/torch2.9/cp313 asset) it warned and kept the broken one. xformers
imports flash_attn whenever find_spec sees it, so every diffusers pipeline then
failed with "libcudart.so.13: cannot open shared object file".
- Re-run _ensure_flash_attn when the step 13 repair moves torch.
- A resident flash-attn that does not import, under a torch that does, is
removed when no working wheel replaces it, and force-replaced when one does
(a same-version build for another CUDA otherwise counts as installed).
Fixes#13244 (flash-attn half).
* Studio installer: pair torch 2.9 with torchao 0.15.0
diffusers 0.41.0 imports torchao.quantization.FqnToConfig whenever torchao is
installed, and FqnToConfig first shipped in torchao 0.15.0. torch 2.9 hosts got
0.14.0, so once a broken flash-attn was out of the way, every diffusers pipeline
still failed to import with "cannot import name 'FqnToConfig'". transformers 5's
TorchAoConfig has the same 0.15 floor.
0.15.0 is also the build for this torch: its C++ extensions target torch 2.9.1,
while 0.14.0 targets 2.8.0 and skipped them on every 2.9 release. It is
published on every download.pytorch.org leaf that serves torch 2.9 (cu126,
cu128, cu129, cu130, rocm6.3, rocm6.4, xpu, cpu), which also closes the cu129
gap, where only 0.14.1 existed. torch <= 2.8 keeps 0.14.0.
Fixes#13244 (torchao half).
* Trim comments in PR scope
---------
Co-authored-by: mahiatlinux <mahiatlinux@users.noreply.github.com>
- test_varlen_int32_overflow_guard: since #12740 SDPA attends packed rows one
segment length at a time, so the guarded fallback is two SDPA calls (the
one-token documents and the padded tail). Assert that only SDPA ran.
- compare_installer_evidence: drop a full winget download bar that repeats
the line right before it. A staging run of the same uv download left it
once on one side and twice on the other.
- test_studio_api test_help_output: render help at 200 columns. At Rich's
80-column default the option cell is cut to --max-seq-lengt...
1ca60dc80e rewrote the README Docker section and dropped the sentence naming unsloth/unsloth:core, so tests/python/test_docker_hub_readme.py::test_the_repo_readme_run_command_matches_the_image fails on main and on every PR's Repo tests (CPU, studio). Keep the new layout and add one sentence pointing notebook-only users at the core tag and Docker Hub. The test's assertion now says why it pins the tag.
* Studio: treat a Smart App Control block of unsloth.exe like the AppLocker one
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten Application Control classifier comment
* Shorten Application Control winerror comment
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>
A main-vs-main run of the Windows installer differential (head 6c723f7477)
failed on progress frames alone: one side logged an extra partly filled
winget bar ('█████▒▒▒▒▒ <percent>' and '<size> / <size>'). How many partial
frames reach the log depends on the network, the same class of noise as the
spinner frames #12965 dropped.
Drop a line only when it is a partial frame (a bar with a '▒' left followed
by a normalised percent or size). The full bar a finished download ends on
still compares, so a download that stops finishing is still reported, and
the real differences in that run (diffusers main now skipped, audio.cpp now
installing) still fail the comparison.
* Add loraplus_lr_ratio to UnslothTrainingArguments (LoRA+)
* Keep a custom optimizer_cls_and_kwargs on the LoRA+ path
* Trim a LoRA+ comment
* Skip the CUDA LoRA+ arguments test where MLX supplies UnslothTrainingArguments
* Trim LoRA+ test comments
* Online DPO left-pad patch: accept TRL 0.18-0.19's logits slice
TRL 0.18.2 and 0.19.x slice the completion logits with
output.logits[:, prompt_ids.size(1) - 1 : -1] instead of naming start_idx, so the
anchor missed and every import unsloth on those versions (inside the supported
trl>=0.18.2 range) printed 'Online DPO's _forward changed upstream ... Please file
a bug report', even in notebooks that never use Online DPO. Match both spellings
and gather from whichever start the source uses.
* Sentence transformers: give Dense heads their own weight dtype
The float32 pooled vector from #12865 reaches a Dense head whose weights stay
float16 on T4 (EmbeddingGemma's 2_Dense / 3_Dense), and with autocast off F.linear
raises 'expected mat1 and mat2 to have the same dtype, but got: float != c10::Half'.
That broke the EmbeddingGemma notebook's first evaluation on Kaggle T4. Cast the
input to the head's dtype, which is what it received before pooling moved to
float32.
* Keep a model that fits one GPU on one GPU under the default device map
The default device map became the head-aware planner, which splits every model
across all visible cards. On Kaggle T4x2 that split models a single T4 trains
fine, and the split broke them: Gemma 3n audio, DeepSeek-OCR and a Qwen3-0.6B
distillation notebook all died with 'Expected all tensors to be on the same
device, but found at least two devices, cuda:0 and cuda:1'. When the planned
weights, head headroom and load transient fit on the first card with a fifth of
it left, load the way a single GPU would. device_map = 'unsloth_balanced' still
splits, and models too large for one card are planned as before.
* One torchcodec warning per broken codec
A torchcodec built for another torch (Colab's 0.11+cu130 under vLLM's torch
2.9.1+cu128, Kaggle's 0.5 under torch 2.11) printed two UserWarnings at import:
the version remedy, then 'torchcodec is installed but cannot load its native
libraries'. Put the remedy, or the accelerator-provenance hint, into the one
fallback report. A codec that loads still gets the version warning alone.
* Keep Unsloth LoRA models off nn.DataParallel under any Trainer
The TRL trainers set args._n_gpu = 1 for an Unsloth model; a plain Trainer only
dropped DataParallel inside _wrap_model, after __init__ had already sized the
batch with n_gpu, and only when the marker sat on the top-level model. On Kaggle
T4x2 Whisper's Seq2SeqTrainer then failed with "'int' object has no attribute
'mean'" and EmbeddingGemma's SentenceTransformerTrainer failed inside
nn.DataParallel. Settle n_gpu before Trainer.__init__ when any submodule carries
the marker.
* Mark sentence-transformer encoder LoRA models for one-GPU training
The encoder path builds its PEFT model directly, so it never carried the
DataParallel marker FastModel.get_peft_model sets. Also pin the Dense test's
activation to identity so its reference matches.
* Single-GPU rule: load onto the card the fit check measured
Return {"": first} instead of "sequential": the measured card is the first one
the caller let the planner use, so a planner-only max_memory that withholds cuda:0
is honoured, and accelerate cannot spill a model that fits onto a second card.
* Trim comments on the notebook sweep fixes
* Guard Trainer init from the marker itself; keep a passed activation reserve on the single card
Fast encoders mark their PeftModel but never call patch_gradient_accumulation_fix, so
the pre-init n_gpu guard never wrapped Trainer.__init__ for them. Install it with the
_wrap_model patch every marker call already makes.
The single-card collapse now also keeps an activation_reserve_bytes the caller passed,
which the planner treats as a hard constraint.
* Tighten the single-card check's docstring
* Keep the model a training script saves when it runs in Docker
* Keep unsloth-run local notebooks relative to /workspace
* Only switch to /workspace/host when the first path is a host file
* Ignore a host option value when a later path names the input
* Keep a host script after a boolean flag as the workdir trigger
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restrict mounted workdir detection to script runners
* Trim comments in PR scope
* Handle launcher aliases and option forms
* Skip launcher option path values
* Recognize mounted launcher input files
* Respect launcher and interpreter option values
* Stop workdir scanning at interpreter scripts
* Support the legacy PyTorch launcher
* Stop launcher parsing at relative scripts
* Stop launcher scans at positional targets
* Handle launcher and shell option aliases
* Parse clustered Python interface options
* Handle the legacy launcher environment flag
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: mahiatlinux <mahiatlinux@users.noreply.github.com>
