mirror of
https://github.com/unslothai/unsloth.git
synced 2026-10-22 14:31:25 +00:00
15 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Daniel Han
|
16e32f3820
|
Studio: Linux torch 2.10 to 2.12 take our causal-conv1d and mamba-ssm wheels (#14338)
Stacked on #14336. Merge only after the Linux torch 2.10 to 2.12 causal-conv1d / mamba-ssm cells from #14336 are published on `prebuilt-wheels-cu13`; before that these URLs would 404 in front of the upstream fallback. ## Why Linux users on torch 2.10, 2.11 or 2.12 with a CUDA 13 torch download upstream's causal-conv1d (205 MB) and mamba-ssm (351 MB) wheels, built for torch 2.10 with `-lineinfo` and every arch. Our builds of the same packages for torch 2.13 / 2.14 are about 29 / 54 MB on x86_64 and 42 / 79 MB on aarch64. ## Changes - `_UNSLOTH_PREBUILT_EXTRA_CELLS` gains linux_x86_64 and linux_aarch64 torch 2.10 / 2.11 / 2.12 on cp311 to cp313. - `_UNSLOTH_PREBUILT_EXTRA_CELL_PREFIXES` limits those Linux cells to causal_conv1d and mamba_ssm, so flash-attn keeps upstream 2.8.1 there (125 MB, smaller than ours). - Unchanged: a CUDA 12 torch, cp310 and cp314 still resolve upstream, and torch 2.13 / 2.14 and Windows resolve as before. ## Tests Updated the tests that pinned upstream URLs for cu13 Linux torch 2.10 to 2.12. They now assert our wheel for cu13 cp311 to cp313 and upstream for cu12 / cp310. Added tests that flash-attn stays upstream and that every Linux extra cell the resolver asks for is one the workflow builds. Affected suites: 1125 passed. |
||
|
Daniel Han
|
7729c37fd0
|
Studio: restore Qwen causal-conv1d on demand, narrow fatal mamba-ssm, provision SSM kernels in export (#13971)
Follow up to #13861 and #13808. An audit of the merged on demand SSM
kernel change plus a T4 benchmark found a few gaps, fixed here.
## Why
- On a Kaggle T4 (sm75, torch 2.11 cu130, fp16, Qwen3.5-0.8B), Studio
chat decode was 4.4% slower without causal-conv1d (19.10 vs 19.95 tok/s,
6 alternating repeats, slower in all 9 pairs). Training speed, losses
and peak memory were unchanged. The zoo Triton conv covers
`causal_conv1d_fn` only, so single token decode fell back to the torch
update path. Training with trainable conv weights also allocates an fp32
dW scratch in the Triton backward, and packed `seq_idx` is not accepted
by the zoo dispatcher under transformers 5.20.
- Mamba-family models that have a working transformers torch path
(mamba, mamba2, falcon_mamba, jamba, zamba, zamba2, bamba) were made
fatal when mamba-ssm could not be installed.
- Export, local LoRA continuation, and inference model switching did not
provision or pick up kernels correctly once setup stopped installing
them.
- Setup on pre-Ampere GPUs (T4) downloaded flash-attn even though
Unsloth only enables FA2 at sm80+.
## Changes
1. Qwen3.5 / Qwen3.6 / Qwen3-Next request causal-conv1d on demand again
in training, inference and Clef (same as before #13861, minus the setup
time install). The zoo Triton platform probe is removed.
2. A missing mamba-ssm is fatal only for `nemotron_h`, `falcon_h1`,
`granitemoehybrid`. Other Mamba families install best effort and
continue on the torch path with a status message.
3. Local LoRA folders (`adapter_config.json`, no `config.json`) resolve
kernel needs from `base_model_name_or_path`; the training
`adapter_base_pin` wins when set.
4. The export worker provisions SSM kernels before importing Unsloth /
the export backend, with the same required vs optional split.
5. Inference re-enables the modeling source scan by reading
`transformers/models/<type>/modeling_*.py` as text, without importing
transformers.
6. If a kernel is newly installed after the model's
`transformers.models.<type>` module was already imported, the inference
worker asks for a restart and the orchestrator respawns it once.
7. Torch repair removes causal-conv1d / mamba-ssm whose compiled
extension no longer imports (never installs anything new). A hung import
check keeps the package.
8. Setup and the long context runtime install skip flash-attn below sm80
(and remove a stale copy at setup). No visible GPU and ROCm keep current
behavior.
## Tests
New and updated tests fail on main and pass here (`test_ssm_runtime.py`,
`test_ssm_restart_respawn.py`, `test_export_ssm_kernels.py`,
`test_clef_fast_paths.py`, `test_kernel_install.py`,
`test_training_worker_flash_attn.py`,
`test_ssm_kernels_install_python_stack.py`,
`test_flash_attn_install_python_stack.py`,
`test_windows_torch_flavor_invariant.py`). With
`test_install_python_stack.py` and `test_cuda_repair.py`: 966 passed
after merging current main.
## Hardening (second commit)
- Stale SSM kernel removal after torch repair needs proof: torch must
import first, and only an ImportError / OSError from the extension
itself removes the package. Signals, hangs, launch errors and a broken
torch keep it.
- `unsloth install-kernels --without-gpu` (or
`UNSLOTH_INSTALL_KERNELS_WITHOUT_GPU=1`) installs GPU gated kernels on a
GPU-less image builder. An already working kernel counts as success
before any skip, skipped kernels are listed, and the exit code is
unchanged (0) unless `--strict` is passed. The CLI respects `UV_OFFLINE`
/ HF offline variables and never falls back from uv to pip offline.
- Inference respawn refuses to spawn while the old worker survives
shutdown. SSM restart and the Xet fallback each get their own one-use
retry (3 spawns max).
- Restart detection only counts native `modeling_*` modules and the
selected checkpoint's remote code modules from its `auto_map`, matched
on the exact `transformers_modules` namespace.
- Export provisions SSM kernels inside the existing transformers
activation window (no extra Hub reachability probe), and skips
provisioning on macOS and for the MLX / CPU export backends.
- `UV_OFFLINE` counts as offline for on-demand installs (no URL probes,
source builds, or uv to pip fallback).
- Long context flash-attn install returns early when flash_attn already
imports, reuses the worker's own torch for the capability check, and
never applies the sm80 floor on ROCm.
## Install robustness (platform matrix review)
- A wheel is downloaded whenever its HEAD check is not a definite 404
(proxies that refuse HEAD no longer push required hybrids into a
failure), then import-verified; fallback only after a real failure.
- A failed install is remembered for the worker process (keyed by
kernel, torch, toolchain), so preflight and load share one attempt.
Optional Mamba families get mamba-ssm only as a prebuilt wheel, never a
source build. ROCm source builds get an 1800 s deadline and CUDA source
builds 7200 s (`UNSLOTH_SOURCE_BUILD_TIMEOUT` overrides); a timeout
kills the whole build tree.
- A runtime install whose extension does not import is uninstalled, so
transformers never finds a broken package.
- One cross-process lock per kernel and environment covers check,
install, verify and cleanup (training and runtime), with a recheck after
acquiring it.
- Only a CUDA or ROCm torch provisions anything; CPU, XPU, MPS and
unknown torch builds skip (export included).
- Optional mamba-ssm follows setup's sm80 floor (sm75 with Triton 3.4+);
the required hybrids keep the previous behavior.
- The resolver only returns URLs for cells that are actually published
(Unsloth release: torch 2.13/2.14 cp313, win_amd64 torch 2.10 to 2.12
cp311 to cp313; upstream causal-conv1d / mamba-ssm release cells), and
there is no source build of the pinned 1.6.1 / 2.3.1 sources on torch
2.13+, which need C++20.
- Training stays best effort for mamba-ssm, as before #13861.
- Export runs the malware and remote code consent gates before any
kernel provisioning.
- The extension is probed again after the install lock is acquired, so a
concurrent replacement is not treated as broken.
- The automatic HF offline window (Hub unreachable) no longer blocks
package installs; only user set offline variables do.
- `unsloth install-kernels --dry-run` prints the resolved URL while
offline.
- A Mamba model whose code needs mamba-ssm fails with a clear error on a
torch build without CUDA or ROCm.
## GPU checks (merge base vs head
|
||
|
Daniel Han
|
50b9deacb0
|
Trim comments in tests (#13844)
Comment and docstring changes only, in test files, split out of #12968 so the test-only part can merge first. |
||
Lee Jackson
|
5c17163f41
|
Fix installer warning colors on light backgrounds (#13572)
* fix: align Python installer warnings with amber palette * Test installer warnings use the shell installer warning color --------- Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com> |
||
|
Daniel Han
|
14ad75e857
|
Studio: install prebuilt flash-attn, causal-conv1d and mamba-ssm wheels on Windows and ARM (#13262)
* Studio: install prebuilt flash-attn, causal-conv1d and mamba-ssm wheels on Windows Windows x64 resolves to our win_amd64 builds on prebuilt-wheels-cu13 (upstream naming, cxx11abiFALSE, cp313, torch 2.13 / 2.14 cu13). Windows on ARM resolves flash-attn to NVIDIA's Flash-Attention-for-WoA wheel for torch 2.14, pinned by a #sha256= fragment that uv and pip both verify. Windows is wheel-only everywhere: no upstream name is ever built (no upstream release has a Windows asset) and a missing or rejected wheel never falls back to a PyPI source build. Setup, long-context training, the causal-conv1d fast path and its lazy hook, SSM inference and install-kernels all take the Windows path; mamba-ssm on Windows first requires an importable triton. Linux URLs are unchanged. * Windows wheels are cxx11abiTRUE: the Windows cu130 torch reports the flag as True * Bound the Windows flash-attn setup probe like its import probe * install-kernels: Windows mamba_ssm waits for Triton; rejected-wheel log says what Windows does * Trim comments in the Windows kernel wheel install paths * Re-check flash-attn when the Windows torch flavor invariant replaces torch * Resolve the linux_aarch64 and win_arm64 prebuilt kernel wheels |
||
alkinun
|
6d7c6e962d
|
Studio installer: drop a flash-attn built for a torch that is gone, and pair torch 2.9 with torchao 0.15.0 (#13247)
* Studio installer: drop a flash-attn built for a torch that is gone The Linux torch repair (step 13) runs after the flash-attn step. When it moves torch, e.g. from 2.12+cu130 to 2.9+cu126 on a CUDA-12 driver, the cu13torch2.10 wheel the earlier step installed can no longer import. The repair re-selected torchao and evicted a mismatched xFormers, but never looked at flash-attn. Later passes did not heal it either: _ensure_flash_attn only removed a copy it had just installed itself. With no wheel for the new torch (flash-attn v2.8.3 ships no cu12/torch2.9/cp313 asset) it warned and kept the broken one. xformers imports flash_attn whenever find_spec sees it, so every diffusers pipeline then failed with "libcudart.so.13: cannot open shared object file". - Re-run _ensure_flash_attn when the step 13 repair moves torch. - A resident flash-attn that does not import, under a torch that does, is removed when no working wheel replaces it, and force-replaced when one does (a same-version build for another CUDA otherwise counts as installed). Fixes #13244 (flash-attn half). * Studio installer: pair torch 2.9 with torchao 0.15.0 diffusers 0.41.0 imports torchao.quantization.FqnToConfig whenever torchao is installed, and FqnToConfig first shipped in torchao 0.15.0. torch 2.9 hosts got 0.14.0, so once a broken flash-attn was out of the way, every diffusers pipeline still failed to import with "cannot import name 'FqnToConfig'". transformers 5's TorchAoConfig has the same 0.15 floor. 0.15.0 is also the build for this torch: its C++ extensions target torch 2.9.1, while 0.14.0 targets 2.8.0 and skipped them on every 2.9 release. It is published on every download.pytorch.org leaf that serves torch 2.9 (cu126, cu128, cu129, cu130, rocm6.3, rocm6.4, xpu, cpu), which also closes the cu129 gap, where only 0.14.1 existed. torch <= 2.8 keeps 0.14.0. Fixes #13244 (torchao half). * Trim comments in PR scope --------- Co-authored-by: mahiatlinux <mahiatlinux@users.noreply.github.com> |
||
Nilay
|
b84349a1f8
|
Studio: keep working when GitHub rate limits or is down (#10670)
Anonymous GitHub API calls are limited to 60 an hour per IP. When that limit hit, the update banner, the llama.cpp changelog, release notes, the sd.cpp installer and the prebuilt wheel probes failed or misreported. - A 403/429 that GitHub explains as throttling (429, Retry-After, X-RateLimit-Remaining: 0, or a rate-limit body) records one process-wide lockout until the reset, capped at an hour; a plain permission 403 records nothing. The freshness check, changelog and release notes honour it. - Release notes send GITHUB_TOKEN / GH_TOKEN to https://api.github.com only, as an unredirected header. - url_exists returns None when availability cannot be checked, so a refusal is no longer reported as "no published wheel"; callers still take their fallback path. - The sd.cpp installer stops at the first rate-limited release lookup and says how to lift the limit, instead of trying every fallback repo. - The llama.cpp installer resolves the diffusion visual server from the approved checksum manifest instead of listing release assets over the API. |
||
|
Daniel Han
|
38f78af7c3
|
Fold duplicated backend, hub and package tests into parametrized runs (#10499)
* Collapse duplicated tool call parser tests into parametrized cases * Parametrize transformers tier detection tests and share a config writer * Parametrize sandbox tool policy allow and block cases * Fold duplicated test cases into parametrized runs and shared setup helpers * Hoist verbatim repeated test setup runs into shared helpers * Drop the scratch area file list from the branch * Name the generated parametrize arguments after what each value is * Share the hardware verdict guard and fold the emission form loop tests * Share the Responses stream drivers across the adapter tests * Share the page fetch stub across the content type tests * Restore the fuzz input labels and the finding tag lost when folding parametrized cases * Name the folded parametrized tests and their arguments for what they check * Restore the repo quote and keyword spacing style on the rewritten test lines * Drop the dead return and its assignment from the tool output stream helper * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-record the patch targets the parametrized folds collapsed * Preserve the full former test name in every folded parameter ID The parametrize folds shortened or reworded some case IDs, so the pre-fold test name could no longer be recovered from the node ID by grep. Restore it: every folded case now carries the original test name minus its test_ prefix, and originals that were themselves parametrized carry that name plus their original case label. IDs only. No test input, expected value or assertion is changed. Collected node id count is unchanged at 57741. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Return only the routes module from the shared KB setup helper _shared_setup_4 built its knowledge base inside `with _connection() as conn` and then returned that connection. _connection is a `closing(...)`, so the handle is already closed by the time the caller sees it. All four call sites happen to shadow it with their own `with _connection() as conn` before any use, which is why nothing fails today, but the helper is new here and handing a closed connection to whoever writes the next test against it is a trap. Return the routes module alone and unpack a single value at the call sites. * Return only the paths from the shared archive and record setup helpers _shared_setup_7 built its zip inside `with zipfile.ZipFile(archive, "w") as zf` and returned that handle; _shared_setup_10 did the same with the file object from `with open(...) as f`. Both are closed by the time the caller sees them. Nothing fails today because the four zip call sites immediately shadow `zf` with their own `with zipfile.ZipFile(archive) as zf`, and the three record call sites never touch `f` at all. Both helpers are new here, so this is a trap we would be creating rather than inheriting. Return the archive path and the root directory alone, and unpack a single value at the seven call sites. * Trim comments in the folded test files Drop the generated per-helper setup narration and collapse multi-line case notes to one line where a single line carries the same fact. Comments only; the parsed AST of every touched file is unchanged. --------- Co-authored-by: danielhanchen <elliegouldingstuff@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
971eb36f48
|
Trim comments in tests, scripts, CI helpers and unsloth_cli (#10113)
* Trim comments in tests, scripts and CI helpers
Comment-only pass over 453 changed files under tests/ (excluding the
studiobench harness), scripts/ and .github/scripts/.
17,749 -> 10,635 comment lines (-40.1%)
1,183,079 -> 757,190 comment characters (-36.0%)
Distinctive-fact retention against the base commit: 93.2%. A sample of 100
restoration and deletion decisions was judged by hand: 98 correct.
No comment line exceeds 120 physical characters. Nothing outside comments
changes: no code, docstrings, string literals, licence headers or pragmas.
Verified with an AST code-only signature of every file against the base.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Put the moved comments back on the code they describe
The signal-handler rationale in launch.py had been reattached to the
handler registration loop, which has no try/finally, and its except-branch
note had been split into a fragment starting mid-sentence with the noqa
marker duplicated into the prose. The PORT_FILE launcher comments had been
reattached to the _studio_root_id caching test, which does not touch
PORT_FILE.
* Put the section banners back, and use plain dashes in comments
The trim removed the rule-line comments that index these files. Tests locate a
region by finding its banner and end it on the next one, so a missing banner is
a broken anchor rather than lost decoration. Each is restored above the code
line it preceded at
|
||
|
Daniel Han
|
947b4bbd7c
|
Studio: verify the flash-attn import after installing it (#8465)
* Studio: verify the flash-attn import after installing it A prebuilt wheel can install with exit code 0 and then fail to load, so the exit code on its own is not proof the install is usable. Both Blackwell incidents were that shape: in #5420 the older-arch wheels installed and raised on import, and the arch gate added to work around it became the bug in #6961 once Dao-AILab started shipping sm_100 wheels. Verify the import after a zero exit code in the two paths that did not, and treat a failure as not installed. ssm_runtime._install_kernel already did exactly this, for the same reason, so this is that check applied to the setup installer and the long-context training worker. worker._is_importable() catches any exception rather than only ImportError: an arch/ABI mismatch surfaces as OSError or RuntimeError ("undefined symbol"), which the old pre-check would have let escape mid-training. Also drop has_blackwell_gpu() and its two call sites. It has returned False unconditionally since |
||
|
Daniel Han
|
5e6b4aa1ee
|
Reuse the torch2.10 prebuilt accelerator wheels on torch 2.12 (#7495) | ||
Thomas Eric 🇧🇷
|
03cbe211a3
|
Studio: fix flash-attn and torchao install on Blackwell (sm_100+) GPUs (Closes #6961) (#6970)
* fix: Remove moot has_blackwell_gpu() function Fixes unslothai/unsloth#6961. This function skipped flash-attn on Blackwell GPUs because no prebuilt wheel existed; Dao-AILab now ships one and url_exists() already gates resolution. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: use torchao 0.17.0 for Blackwell Fixes #6961. Torchao 0.16.0's cpp extensions are built against CUDA 12, so on a CUDA-13 torch (cu130 / Blackwell) they fail to load with "libcudart.so.12: cannot open shared object file". Select 0.17.0 there instead: its cpp targets torch 2.11, so it is skipped cleanly rather than crashing. CUDA-12 / ROCm / CPU torch 2.10 keeps 0.16.0 and its working kernels. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Condense torchao version-selection comments (no behavior change) * Support torch 2.11 in the Studio installer via the torch2.10 prebuilt wheels Map torch 2.11 to the torch2.10 prebuilt wheels for flash-attn, causal-conv1d, and mamba through wheel_utils.prebuilt_wheel_torch_mm, applied in direct_wheel_url (filename) and flash_attn_wheel_url (version). Those torch2.10 CUDA wheels load and pass each project's own test suite on torch 2.11 (verified on B200), so a torch 2.11 environment gets the prebuilt accelerators instead of skipping or building from source. Raise _CUDA_TORCH_PKG_SPEC to <2.12.0 (torchvision <0.27.0, torchaudio <2.12.0) so the CUDA torch repair path can install torch 2.11, where torchao 0.17's cpp kernels load cleanly. Add tests for the mapping. * Keep has_blackwell_gpu as a False stub for future arch gating * Restore has_blackwell_gpu as a return-False probe kept for future arch gating Keep the nvidia-smi compute_cap detection and its two call sites, but short-circuit with return False at the top so flash-attn is no longer skipped on Blackwell (sm_100+ now has prebuilt wheels and url_exists gates resolution). Drop the early return to re-enable arch-based detection later. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> |
||
|
Daniel Han
|
3ce187da02
|
Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079)
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent. |
||
Roland Tannous
|
79adfd9c71
|
studio: skip flash-attn install on Blackwell GPUs (sm_100+) (#5420)
* studio: skip flash-attn install on Blackwell GPUs (sm_100+) Dao-AILab does not publish prebuilt flash-attn wheels for sm_100, sm_120, or sm_121, and the older-arch wheels fail to load on Blackwell. Add a shared has_blackwell_gpu() helper and gate both the install-time (install_python_stack._ensure_flash_attn) and runtime (worker._ensure_flash_attn_for_long_context) paths on it. Detection uses nvidia-smi --query-gpu=compute_cap, which works on Linux and Windows. * test: stub has_blackwell_gpu in pre-existing runtime flash-attn tests prefers_prebuilt_wheel and falls_back_to_pypi exercise the install paths that the Blackwell guard now short-circuits. Make them explicit about non-Blackwell so they pass on real Blackwell hosts. * studio: cache has_blackwell_gpu, skip Blackwell warning under NO_TORCH - Wrap has_blackwell_gpu in functools.lru_cache so repeated calls in a single process avoid redundant nvidia-smi spawns. Tests clear the cache via setup_method/teardown_method. - In _ensure_flash_attn, run the NO_TORCH short-circuit before the Blackwell check so GGUF-only users (who never install torch anyway) do not see a Blackwell warning. Blackwell check still runs above the IS_WINDOWS / IS_MACOS gates so Blackwell-on-Windows users still see the explicit reason rather than a silent OS skip. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test: add has_blackwell_gpu to mlx worker test wheel_utils stub test_mlx_training_worker_config loads worker.py against a hand-rolled utils.wheel_utils stub. Adding has_blackwell_gpu to the stub symbol list so worker's import line resolves. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
Datta Nimmaturi
|
da78c6be71
|
[Studio] Install flash attn at setup time for linux (#4979)
* [Studio] Install flash attn at setup time for linux * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * cleanup changes Signed-off-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Test cases * wheel_utils: narrow url_exists exceptions and log at debug level --------- Signed-off-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com> Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai> |