mirror of
https://github.com/unslothai/unsloth.git
synced 2026-10-19 08:54:21 +00:00
Hybrid packing + block compile: fix the guard crash, the skipped varlen, and the cross-document leak (#13956)
Some checks are pending
CodeQL / Languages to analyse (push) Waiting to run
CodeQL / Analyze () (push) Blocked by required conditions
CodeQL / Prune superseded CodeQL overlay bases (push) Blocked by required conditions
Core / Resolve the unsloth_zoo revision (push) Waiting to run
Core / Core (HF=default + TRL=default) (push) Blocked by required conditions
Core / Core (HF=4.57.6 + TRL<1) (push) Blocked by required conditions
Core / Core (HF=latest + TRL=latest) (push) Blocked by required conditions
Core / Core zoo (HF=default + TRL=default) (push) Blocked by required conditions
Core / Core zoo (HF=4.57.6 + TRL<1) (push) Blocked by required conditions
Core / Core zoo (HF=latest + TRL=latest) (push) Blocked by required conditions
Core / llama.cpp build + smoke (push) Waiting to run
Cross-platform parity / parity (ubuntu-latest) (push) Waiting to run
Cross-platform parity / parity (windows-latest) (push) Waiting to run
Interrupted install recovery / select legs (push) Waiting to run
Interrupted install recovery / kill@ (push) Blocked by required conditions
Interrupted install recovery / windows kill@ (push) Blocked by required conditions
Kaggle T4 Notebook CI / gate (push) Waiting to run
Kaggle T4 Notebook CI / T4 smoke (push) Blocked by required conditions
Kaggle T4 Unsloth GPU CI / gate (push) Waiting to run
Kaggle T4 Unsloth GPU CI / Unsloth GPU smoke (push) Blocked by required conditions
Lint CI / Source lint (Python + shell + YAML + JSON + safety nets) (push) Waiting to run
Local Agent Guides CI / connection (vibe, stable) (push) Waiting to run
Local Agent Guides CI / file-edit (claude) (push) Waiting to run
Local Agent Guides CI / file-edit (codex) (push) Waiting to run
Local Agent Guides CI / file-edit (dsh) (push) Waiting to run
Local Agent Guides CI / file-edit (hermes) (push) Waiting to run
Local Agent Guides CI / file-edit (openclaw) (push) Waiting to run
Local Agent Guides CI / file-edit (opencode) (push) Waiting to run
Local Agent Guides CI / file-edit (pi) (push) Waiting to run
Local Agent Guides CI / file-edit (vibe) (push) Waiting to run
Local Agent Guides CI / resume (claude) (push) Waiting to run
Local Agent Guides CI / resume (codex) (push) Waiting to run
Local Agent Guides CI / resume (opencode) (push) Waiting to run
Local Agent Guides CI / resume (pi) (push) Waiting to run
Local Agent Guides CI / prompt-cache (gemma-3-270m) (push) Waiting to run
Local Agent Guides CI / connection (claude, stable) (push) Waiting to run
Local Agent Guides CI / connection (codex, stable) (push) Waiting to run
Local Agent Guides CI / connection (dsh, stable) (push) Waiting to run
Local Agent Guides CI / connection (hermes, stable) (push) Waiting to run
Local Agent Guides CI / connection (openclaw, stable) (push) Waiting to run
Local Agent Guides CI / connection (opencode, stable) (push) Waiting to run
Local Agent Guides CI / connection (opencode, v2) (push) Waiting to run
Local Agent Guides CI / connection (pi, stable) (push) Waiting to run
MLX CI on Mac M1 / dispatch (push) Waiting to run
Security audit / advisory audit (pip + npm + cargo) (push) Waiting to run
Security audit / pip scan-packages :: audio-torch280 (push) Waiting to run
Security audit / pip scan-packages :: audio-torch290 (push) Waiting to run
Security audit / pip scan-packages :: extras (push) Waiting to run
Security audit / pip scan-packages :: studio (push) Waiting to run
Security audit / pip scan-packages :: hf-stack (push) Waiting to run
Security audit / npm scan-packages (Unsloth frontend tarballs) (push) Waiting to run
Security audit / npm provenance + new install-script diff (push) Waiting to run
Unsloth API CI / Unsloth API & Auth Tests (push) Waiting to run
Backend CI / (Python 3.11, floor) (push) Waiting to run
Backend CI / (Python 3.13, a-k) (push) Waiting to run
Backend CI / (Python 3.13, l-r) (push) Waiting to run
Backend CI / (Python 3.13, rest) (push) Waiting to run
Backend CI / Repo tests (CPU, rest) (push) Waiting to run
Backend CI / Repo tests (CPU, studio) (push) Waiting to run
Backend CI / Repo tests (CPU, studio-install) (push) Waiting to run
Backend CI / Shell installer tests (push) Waiting to run
Unsloth export capability / capability (windows-latest) (push) Waiting to run
Frontend CI / Frontend build + bundle sanity (push) Waiting to run
Frontend CI / Frontend unit tests (Windows) (push) Waiting to run
Unsloth GGUF CI / GGUF inference smoke (API, tools, vision) (push) Waiting to run
Mac Unsloth UI + API + Update + Inference CI / Chat UI, API, Update and GGUF inference (push) Waiting to run
Unsloth Tauri CI / Tauri Linux debug build (no codesign) (push) Waiting to run
Unsloth Tauri CI / Rust unit tests (windows) (push) Waiting to run
Unsloth Tauri CI / Rust unit tests (macos) (push) Waiting to run
Unsloth UI CI / Chat UI Tests (extra) (push) Waiting to run
Unsloth UI CI / Chat UI Tests (chat) (push) Waiting to run
Unsloth Update CI / Unsloth Updating Tests (macOS) (push) Waiting to run
Studio VAE seam guard / seam-guard (push) Waiting to run
Unsloth UI CI / Loaded-models indicator (cross-browser) (push) Waiting to run
Unsloth Update CI / Unsloth Updating Tests (push) Waiting to run
Unsloth Update CI / Unsloth Updating Tests (Windows) (push) Waiting to run
Windows Unsloth API CI / Unsloth API & Auth Tests (push) Waiting to run
Windows Unsloth GGUF CI / GGUF inference smoke (API, tools, vision) (push) Waiting to run
Windows Unsloth GGUF CI / Unsloth install + inference without Visual Studio (push) Waiting to run
Windows Unsloth GGUF CI / real-VS detection (VS 2022) (push) Waiting to run
Windows Unsloth GGUF CI / real-VS detection + VC++ round-trip (VS 2026) (push) Waiting to run
Windows Unsloth GGUF CI / setup.ps1 units, no-VS resolve, VC++ round-trip (windows-latest) (push) Waiting to run
Windows Unsloth UI CI / Chat UI Tests (push) Waiting to run
Windows Unsloth Update CI / Unsloth Updating Tests (push) Waiting to run
studiobench CI / Selftests, lint and the single-file build (push) Waiting to run
studiobench CI / Real-path session, every action ran (push) Waiting to run
Wheel CI / Wheel build + content sanity + import smoke (push) Waiting to run
Windows Application Control CI / installer survives a denied unsloth.exe (push) Waiting to run
Windows ARM64 CI / woa (pwsh-7) (push) Waiting to run
Windows ARM64 CI / woa (windows-powershell-5.1) (push) Waiting to run
Windows No Compiler CI / the installer spawns no C# compiler (push) Waiting to run
Workflow trigger lint / workflow-trigger lint (pull_request_target / cache-poisoning) (push) Waiting to run
Workflow trigger lint / security suites (torch-dependent) (push) Waiting to run
Some checks are pending
CodeQL / Languages to analyse (push) Waiting to run
CodeQL / Analyze () (push) Blocked by required conditions
CodeQL / Prune superseded CodeQL overlay bases (push) Blocked by required conditions
Core / Resolve the unsloth_zoo revision (push) Waiting to run
Core / Core (HF=default + TRL=default) (push) Blocked by required conditions
Core / Core (HF=4.57.6 + TRL<1) (push) Blocked by required conditions
Core / Core (HF=latest + TRL=latest) (push) Blocked by required conditions
Core / Core zoo (HF=default + TRL=default) (push) Blocked by required conditions
Core / Core zoo (HF=4.57.6 + TRL<1) (push) Blocked by required conditions
Core / Core zoo (HF=latest + TRL=latest) (push) Blocked by required conditions
Core / llama.cpp build + smoke (push) Waiting to run
Cross-platform parity / parity (ubuntu-latest) (push) Waiting to run
Cross-platform parity / parity (windows-latest) (push) Waiting to run
Interrupted install recovery / select legs (push) Waiting to run
Interrupted install recovery / kill@ (push) Blocked by required conditions
Interrupted install recovery / windows kill@ (push) Blocked by required conditions
Kaggle T4 Notebook CI / gate (push) Waiting to run
Kaggle T4 Notebook CI / T4 smoke (push) Blocked by required conditions
Kaggle T4 Unsloth GPU CI / gate (push) Waiting to run
Kaggle T4 Unsloth GPU CI / Unsloth GPU smoke (push) Blocked by required conditions
Lint CI / Source lint (Python + shell + YAML + JSON + safety nets) (push) Waiting to run
Local Agent Guides CI / connection (vibe, stable) (push) Waiting to run
Local Agent Guides CI / file-edit (claude) (push) Waiting to run
Local Agent Guides CI / file-edit (codex) (push) Waiting to run
Local Agent Guides CI / file-edit (dsh) (push) Waiting to run
Local Agent Guides CI / file-edit (hermes) (push) Waiting to run
Local Agent Guides CI / file-edit (openclaw) (push) Waiting to run
Local Agent Guides CI / file-edit (opencode) (push) Waiting to run
Local Agent Guides CI / file-edit (pi) (push) Waiting to run
Local Agent Guides CI / file-edit (vibe) (push) Waiting to run
Local Agent Guides CI / resume (claude) (push) Waiting to run
Local Agent Guides CI / resume (codex) (push) Waiting to run
Local Agent Guides CI / resume (opencode) (push) Waiting to run
Local Agent Guides CI / resume (pi) (push) Waiting to run
Local Agent Guides CI / prompt-cache (gemma-3-270m) (push) Waiting to run
Local Agent Guides CI / connection (claude, stable) (push) Waiting to run
Local Agent Guides CI / connection (codex, stable) (push) Waiting to run
Local Agent Guides CI / connection (dsh, stable) (push) Waiting to run
Local Agent Guides CI / connection (hermes, stable) (push) Waiting to run
Local Agent Guides CI / connection (openclaw, stable) (push) Waiting to run
Local Agent Guides CI / connection (opencode, stable) (push) Waiting to run
Local Agent Guides CI / connection (opencode, v2) (push) Waiting to run
Local Agent Guides CI / connection (pi, stable) (push) Waiting to run
MLX CI on Mac M1 / dispatch (push) Waiting to run
Security audit / advisory audit (pip + npm + cargo) (push) Waiting to run
Security audit / pip scan-packages :: audio-torch280 (push) Waiting to run
Security audit / pip scan-packages :: audio-torch290 (push) Waiting to run
Security audit / pip scan-packages :: extras (push) Waiting to run
Security audit / pip scan-packages :: studio (push) Waiting to run
Security audit / pip scan-packages :: hf-stack (push) Waiting to run
Security audit / npm scan-packages (Unsloth frontend tarballs) (push) Waiting to run
Security audit / npm provenance + new install-script diff (push) Waiting to run
Unsloth API CI / Unsloth API & Auth Tests (push) Waiting to run
Backend CI / (Python 3.11, floor) (push) Waiting to run
Backend CI / (Python 3.13, a-k) (push) Waiting to run
Backend CI / (Python 3.13, l-r) (push) Waiting to run
Backend CI / (Python 3.13, rest) (push) Waiting to run
Backend CI / Repo tests (CPU, rest) (push) Waiting to run
Backend CI / Repo tests (CPU, studio) (push) Waiting to run
Backend CI / Repo tests (CPU, studio-install) (push) Waiting to run
Backend CI / Shell installer tests (push) Waiting to run
Unsloth export capability / capability (windows-latest) (push) Waiting to run
Frontend CI / Frontend build + bundle sanity (push) Waiting to run
Frontend CI / Frontend unit tests (Windows) (push) Waiting to run
Unsloth GGUF CI / GGUF inference smoke (API, tools, vision) (push) Waiting to run
Mac Unsloth UI + API + Update + Inference CI / Chat UI, API, Update and GGUF inference (push) Waiting to run
Unsloth Tauri CI / Tauri Linux debug build (no codesign) (push) Waiting to run
Unsloth Tauri CI / Rust unit tests (windows) (push) Waiting to run
Unsloth Tauri CI / Rust unit tests (macos) (push) Waiting to run
Unsloth UI CI / Chat UI Tests (extra) (push) Waiting to run
Unsloth UI CI / Chat UI Tests (chat) (push) Waiting to run
Unsloth Update CI / Unsloth Updating Tests (macOS) (push) Waiting to run
Studio VAE seam guard / seam-guard (push) Waiting to run
Unsloth UI CI / Loaded-models indicator (cross-browser) (push) Waiting to run
Unsloth Update CI / Unsloth Updating Tests (push) Waiting to run
Unsloth Update CI / Unsloth Updating Tests (Windows) (push) Waiting to run
Windows Unsloth API CI / Unsloth API & Auth Tests (push) Waiting to run
Windows Unsloth GGUF CI / GGUF inference smoke (API, tools, vision) (push) Waiting to run
Windows Unsloth GGUF CI / Unsloth install + inference without Visual Studio (push) Waiting to run
Windows Unsloth GGUF CI / real-VS detection (VS 2022) (push) Waiting to run
Windows Unsloth GGUF CI / real-VS detection + VC++ round-trip (VS 2026) (push) Waiting to run
Windows Unsloth GGUF CI / setup.ps1 units, no-VS resolve, VC++ round-trip (windows-latest) (push) Waiting to run
Windows Unsloth UI CI / Chat UI Tests (push) Waiting to run
Windows Unsloth Update CI / Unsloth Updating Tests (push) Waiting to run
studiobench CI / Selftests, lint and the single-file build (push) Waiting to run
studiobench CI / Real-path session, every action ran (push) Waiting to run
Wheel CI / Wheel build + content sanity + import smoke (push) Waiting to run
Windows Application Control CI / installer survives a denied unsloth.exe (push) Waiting to run
Windows ARM64 CI / woa (pwsh-7) (push) Waiting to run
Windows ARM64 CI / woa (windows-powershell-5.1) (push) Waiting to run
Windows No Compiler CI / the installer spawns no C# compiler (push) Waiting to run
Workflow trigger lint / workflow-trigger lint (pull_request_target / cache-poisoning) (push) Waiting to run
Workflow trigger lint / security suites (torch-dependent) (push) Waiting to run
## What breaks
`SFTTrainer` turns on the hybrid varlen shim
(`patch_hybrid_linear_attention_varlen`) for any GatedDeltaNet model,
and padding-free is on by default. On transformers 5.16 and later,
Qwen3.5 mixers run in kwargs mode, so a normal Qwen3.5 fine-tune runs
with both the shim and the per-layer block compile active. Three bugs
sit on that path. Each one is hidden by the one before it, so the fixes
have to land together.
1. **Guard crash, then a fallback that takes minutes.** Unsloth's
compiled modules run `from <transformers modeling> import __name__`, and
Dynamo resolves an inlined function's globals through
`sys.modules[__name__]`. The varlen probes and the block's causal-conv
adapter were only bound in the compiled module's namespace. The block
guarded the transformers binding and traced a different object, which
raised `TypeError: 'NoneType' object is not subscriptable` in a
`CLOSURE_MATCH` guard. It then retried with flex outside, then kept the
stock forward. On Qwen3.5-0.8B the first step took 522 s.
2. **Varlen skipped under compile.** `_varlen_seq_idx_applies` used
`total in (a, b)`. After a dynamic-shape recompile, Dynamo evaluates
that as False for a row that matches, so the mixer skips varlen and the
handshake stops training with `hybrid packing cannot continue because
the varlen boundary kernels were not invoked`.
3. **Packed documents leak into each other.** On torch 2.10 and 2.11,
Dynamo constant-folds `torch.compiler.is_exporting()` to `True`
(`torch/_dynamo/variables/torch.py`); 2.12 and later read the real flag.
transformers' `use_kernel_func_from_hub_with_fallback` reads that as
`torch.export` and calls its pure-PyTorch reference kernel, which
ignores `seq_idx` / `cu_seqlens`. So inside a compiled block, the packed
GDN scan (and the conv) runs across document boundaries. Nothing raises,
and the hidden states are just wrong.
On current main, bug 1 sends every block back to eager, which hides bug
3. Fixing bug 1 alone would make packed compiled training silently
wrong.
## Fix
- `_block_compile.py`: `_route_causal_conv` keeps the packing probe
outermost, puts the adapter under the probe's `_fn` default, and mirrors
the binding into the alias namespace (`_alias_namespace`).
`_route_fla_kernel` also routes the GDN chunk kernel held as a probe's
`_fn`, so the unpacked compiled path still reaches the opaque fla op
instead of the torch reference.
- `packing.py`: `_install_kwargs_probes` mirrors each probe into the
alias namespace. `_varlen_seq_idx_applies` now compares with `==`. In
`_wrap_kwargs_mixer_forward`, the packed branch runs under
`torch.compiler.disable` where Dynamo folds `is_exporting()`
(`_dynamo_folds_is_exporting`, a one-time eager probe, not a version
string; an unknown answer keeps it eager), so the real varlen kernels
run. On torch 2.12+ the packed mixer stays in the graph: the
unconditional disable cost 6-20% per step there (table below). Unpacked
calls are still traced.
## Results on real checkpoints (one B200, bf16, LoRA r=16, Unsloth
gradient checkpointing)
Training mode with block compile. One packed row of 3 documents
(311/330/398 tokens) is compared with each document run alone. The
number is the cosine of the final hidden states per document; 1.0 means
no leak.
**Qwen/Qwen3.5-0.8B** (dense, 18 GDN + 6 attention layers)
| build | first step | GDN layers | per-document cosine |
|---|---|---|---|
| main | 522 s, guard `TypeError`, stock forward | eager | 1.0, 1.0, 1.0
(eager only) |
| this PR | 30 s | compiled | 0.99996, 0.99995, 0.99995 |
| this PR without fix 2 | `hybrid packing cannot continue ...` | | |
| this PR without fix 3 | 96 s | compiled | 0.99995, **0.777, 0.711** |
**ibm-granite/granite-4.0-h-350m** (Mamba2 hybrid, kwargs-mode
`mamba2_split_conv1d_scan_combined` / `mamba2_chunk_scan`)
Mamba2 packing is opt-in (`UNSLOTH_EXPERIMENTAL_HYBRID_PACKING=1`); by
default the shim stays off for it.
| build (opt-in) | result | per-document cosine |
|---|---|---|
| eager (`UNSLOTH_BLOCK_COMPILE=0`) | reference | 0.99991, 0.99373,
0.99186 |
| main | training stops: `varlen boundary kernels were not invoked` | |
| this PR | 0.22 s/step, all 32 blocks compiled | 0.9999, 0.994, 0.992
(matches eager) |
| this PR without fix 2 | same error as main | |
| this PR without fix 3 | | 0.9999, **0.969, 0.974** |
Bug 1 doesn't occur on Granite, since there is no GDN conv adapter
closure, but fixes 2 and 3 are needed there too. They live in the shared
kwargs-mixer wrapper, so they already cover the Mamba2 kernels.
On the Drama 3 backbone (Qwen3.5-MoE 35B), packed + compiled is 0.91
s/step against 1.23 to 1.36 s eager. The first compiled step is 45 s,
down from 157 to 242 s. Step-0 losses match eager (4.724 / 5.135 vs
4.715 / 5.133).
## Packed mixer gate per torch version (Colab L4, bf16,
unsloth/Qwen3.5-0.8B LoRA SFT, 20 steps, padding-free on by default)
Steady step ms; base = main at the merge base, "always eager" = this PR
before the gate, "gated" = this head. Two runs where shown.
| torch | load | base | always eager | gated |
|---|---|---|---|---|
| 2.14 | 16-bit | 222.2 | 234.6 / 243.6 | 224.6 / 226.3 |
| 2.14 | 4-bit | 251.9 | 319.7 / 314.3 | 256.9 / 260.6 |
| 2.12 | 16-bit | crashes (varlen kernels not invoked) | 265.5 | 262.3 /
263.7 |
| 2.12 | 4-bit | crashes | 347.7 | 298.9 / 295.0 |
| 2.10 | 16-bit | | 254.56 | 254.57 (same eager path) |
Leak probe (per-document cosine, 1.0 = no leak): gated 0.99994-0.99995
on 2.10, 2.12 (16-bit and 4-bit) and 2.14, the same as base on 2.14 and
the always-eager path. Graph breaks on 2.14 (all "Skip calling
`torch.compiler.disable()`d function", the hub kernels' own disables):
base 30, always eager 6, gated 38. The gated first packed step compiles
the mixer too (2.12: 113 s vs 74 s, once). Kaggle T4 (fp16) torch
2.10-2.14, this PR before the gate: Qwen3.5 16-bit / 4-bit / packing and
inference NO_REGRESSION on every version.
## Tests
Three CPU tests in `tests/utils/test_hybrid_packing_default.py`. Each
fails on main and passes here:
- `test_probes_and_block_adapters_match_the_namespace_dynamo_reads`
builds a fake compiled module that imports the modeling `__name__`. It
checks that both namespaces hold the same probe, and that the conv and
chunk adapters sit under the probe's `_fn`.
- `test_boundary_check_holds_after_a_dynamic_recompile` compiles
`_varlen_seq_idx_applies` with `backend="eager"` across a layout change.
With `in`, the second call returns False. This one also fails with only
fix 2 reverted.
- `test_a_packed_mixer_runs_outside_a_compiled_block` checks that a
packed call to a wrapped mixer runs outside Dynamo where
`is_exporting()` folds and stays traced where it does not, and that an
unpacked call is always traced. This one also fails with only fix 3
reverted. `test_is_exporting_fold_probe_matches_this_torch` checks the
probe against the torch version.
`tests/utils/test_hybrid_packing_default.py`,
`tests/utils/test_packing.py` and `tests/test_block_compile.py`: 166
passed, 1 failed. The failure,
`test_string_model_config_detects_stateful_mixers[mamba2-True-extra1]`,
fails the same way on main. It's an environment issue: with `fla`
installed, `AutoConfig.for_model("mamba2")` returns fla's
`Mamba2Config`, and shrinking `hidden_size` to 64 breaks fla's
`num_heads * head_dim == expand * hidden_size` check.
## Not in this PR
- With packing on, Qwen3.5's full-attention layers still fall back to
the stock forward. Inductor fails to compile transformers' flash-varlen
prep (`cu_seq_lens_q.diff().max()`), with `Loop-carried variable ...
int32 ... int64`. This also happens on main, and GDN layers are not
affected.
- Granite's eager packed run gives cosine about 0.993 on documents 2 and
3, against 0.9999 on the first. That is the same with and without
compile, so it's independent of this PR. It needs a separate look.
---------
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
This commit is contained in:
parent
0174c538fb
commit
a33463fb16
3 changed files with 236 additions and 24 deletions
|
|
@ -377,7 +377,27 @@ def _zoo_style_dispatch(torch_function, name_seq_idx = True):
|
|||
return causal_conv1d_fn
|
||||
|
||||
|
||||
def scan_fast(
|
||||
x,
|
||||
cu_seqlens = None,
|
||||
**kwargs,
|
||||
):
|
||||
return scan_exact(x, cu_seqlens)
|
||||
|
||||
|
||||
def _kwargs_model(name, conv_impl):
|
||||
modeling = types.ModuleType(name)
|
||||
_install_kwargs_mixer(modeling.__dict__, conv_impl)
|
||||
modeling.FakeKwargsGatedDeltaNet.__module__ = name
|
||||
sys.modules[name] = modeling
|
||||
return FakeHybrid(modeling.FakeKwargsGatedDeltaNet())
|
||||
|
||||
|
||||
def _install_kwargs_mixer(
|
||||
namespace,
|
||||
conv_impl,
|
||||
scan_impl = scan_fast,
|
||||
):
|
||||
def conv_ref(
|
||||
hidden_states,
|
||||
weight,
|
||||
|
|
@ -390,17 +410,9 @@ def _kwargs_model(name, conv_impl):
|
|||
def scan_ref(x, **kwargs):
|
||||
return scan_exact(x)
|
||||
|
||||
def scan_fast(
|
||||
x,
|
||||
cu_seqlens = None,
|
||||
**kwargs,
|
||||
):
|
||||
return scan_exact(x, cu_seqlens)
|
||||
|
||||
modeling = types.ModuleType(name)
|
||||
impl = conv_impl(conv_ref)
|
||||
modeling.causal_conv1d_fn = _hub_wrapped(impl, impl)
|
||||
modeling.torch_chunk_gated_delta_rule = _hub_wrapped(scan_ref, scan_fast)
|
||||
namespace["causal_conv1d_fn"] = _hub_wrapped(impl, impl)
|
||||
namespace["torch_chunk_gated_delta_rule"] = _hub_wrapped(scan_ref, scan_impl)
|
||||
exec(
|
||||
"import torch\n"
|
||||
"class FakeKwargsGatedDeltaNet(torch.nn.Module):\n"
|
||||
|
|
@ -412,11 +424,8 @@ def _kwargs_model(name, conv_impl):
|
|||
" x = causal_conv1d_fn(hidden_states.transpose(1, 2), self.conv1d.weight.squeeze(1), None,\n"
|
||||
" activation = 'silu', **kwargs).transpose(1, 2)\n"
|
||||
" return torch_chunk_gated_delta_rule(x, cu_seqlens = kwargs.pop('cu_seq_lens_q', None), **kwargs)\n",
|
||||
modeling.__dict__,
|
||||
namespace,
|
||||
)
|
||||
modeling.FakeKwargsGatedDeltaNet.__module__ = name
|
||||
sys.modules[name] = modeling
|
||||
return FakeHybrid(modeling.FakeKwargsGatedDeltaNet())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mode", [None, "1"])
|
||||
|
|
@ -596,3 +605,129 @@ def test_a_stale_stash_never_reaches_a_call_of_another_length(monkeypatch):
|
|||
mixer._unsloth_varlen_conv_hit = mixer._unsloth_varlen_scan_hit = False
|
||||
torch.testing.assert_close(mixer(hidden), plain)
|
||||
assert not mixer._unsloth_varlen_conv_hit and not mixer._unsloth_varlen_scan_hit
|
||||
|
||||
|
||||
def test_boundary_check_holds_after_a_dynamic_recompile():
|
||||
# Traced, `total in (dim, dim)` came out False once the conv layout recompiled with dynamic shapes.
|
||||
seen = []
|
||||
|
||||
def check(seq_idx, x):
|
||||
seen.append(packing_module._varlen_seq_idx_applies(seq_idx, (x,), {}))
|
||||
return x + 1
|
||||
|
||||
seq_idx = torch.zeros(1, 300, dtype = torch.int32)
|
||||
torch._dynamo.reset()
|
||||
try:
|
||||
compiled = torch.compile(check, backend = "eager")
|
||||
compiled(seq_idx, torch.zeros(1, 300, 8)) # (batch, seqlen, ...)
|
||||
compiled(seq_idx, torch.zeros(1, 8, 300)) # causal_conv1d_fn: (batch, dim, seqlen)
|
||||
finally:
|
||||
torch._dynamo.reset()
|
||||
assert seen == [True, True]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("folds", [True, False])
|
||||
def test_a_packed_mixer_runs_outside_a_compiled_block(monkeypatch, folds):
|
||||
# Traced, transformers' hub-kernel wrappers run their torch reference where Dynamo folds
|
||||
# torch.compiler.is_exporting() to True (torch 2.10 / 2.11), and it ignores seq_idx / cu_seqlens.
|
||||
# Where it does not, the packed mixer stays in the graph.
|
||||
monkeypatch.setattr(packing_module, "_FOLDS_IS_EXPORTING", [folds])
|
||||
traced = []
|
||||
|
||||
class Mixer(torch.nn.Module):
|
||||
def forward(self, hidden_states, **kwargs):
|
||||
traced.append(torch.compiler.is_compiling())
|
||||
return hidden_states * 2
|
||||
|
||||
mixer = Mixer()
|
||||
packing_module._wrap_kwargs_mixer_forward(mixer)
|
||||
mixer._unsloth_varlen = (
|
||||
torch.tensor([0, 4, 9], dtype = torch.int32),
|
||||
torch.tensor([[0] * 4 + [1] * 5], dtype = torch.int32),
|
||||
)
|
||||
hidden = torch.randn(1, 9, 8)
|
||||
torch._dynamo.reset()
|
||||
try:
|
||||
block = torch.compile(lambda h: mixer(h), backend = "eager")
|
||||
block(hidden)
|
||||
mixer._unsloth_varlen = None
|
||||
block(hidden)
|
||||
finally:
|
||||
torch._dynamo.reset()
|
||||
assert traced == ([False, True] if folds else [True, True]) # only the packed call leaves it
|
||||
|
||||
|
||||
def test_is_exporting_fold_probe_matches_this_torch():
|
||||
from packaging.version import Version
|
||||
folds = packing_module._dynamo_folds_is_exporting()
|
||||
assert folds == (Version(torch.__version__.split("+")[0]).release[:2] in ((2, 10), (2, 11)))
|
||||
|
||||
|
||||
def test_probes_and_block_adapters_match_the_namespace_dynamo_reads(monkeypatch):
|
||||
# Unsloth's compiled modules run `from <modeling> import __name__`, so Dynamo resolves their globals
|
||||
# through the modeling module; a kernel bound differently there failed the block's closure guard.
|
||||
from unsloth.models import _block_compile as block
|
||||
|
||||
monkeypatch.delenv(ENV, raising = False)
|
||||
names = (
|
||||
"fake_alias.fla.ops.gated_delta_rule.chunk",
|
||||
"fake_modeling_kwargs_alias_gdn",
|
||||
"fake_unsloth_compiled_module_alias_gdn",
|
||||
)
|
||||
fla, modeling, compiled = (types.ModuleType(n) for n in names)
|
||||
|
||||
def chunk_gated_delta_rule(
|
||||
x,
|
||||
cu_seqlens = None,
|
||||
**kwargs,
|
||||
):
|
||||
return scan_exact(x, cu_seqlens)
|
||||
|
||||
fla.chunk_gated_delta_rule = chunk_gated_delta_rule
|
||||
compiled.__name__ = modeling.__name__
|
||||
for module in (modeling, compiled):
|
||||
_install_kwargs_mixer(vars(module), _zoo_style_dispatch, scan_impl = chunk_gated_delta_rule)
|
||||
for n, module in zip(names, (fla, modeling, compiled)):
|
||||
monkeypatch.setitem(sys.modules, n, module)
|
||||
model = FakeHybrid(compiled.FakeKwargsGatedDeltaNet())
|
||||
assert patch_hybrid_linear_attention_varlen(model) is True
|
||||
# Route only these modules: routing a real fla import would outlive this test.
|
||||
monkeypatch.setattr(block, "sys", SimpleNamespace(modules = {n: sys.modules[n] for n in names}))
|
||||
block._route_fla_kernel()
|
||||
block._route_causal_conv(model.linear_attn)
|
||||
for kernel in ("causal_conv1d_fn", "torch_chunk_gated_delta_rule"):
|
||||
probe = vars(compiled)[kernel]
|
||||
assert vars(modeling)[kernel] is probe, kernel
|
||||
assert probe._unsloth_varlen_probe_of is not None, kernel
|
||||
assert getattr(probe.__kwdefaults__["_fn"], "_unsloth_block_adapter", False), kernel
|
||||
|
||||
|
||||
def test_a_failed_probe_restores_the_mirrored_alias_namespace(monkeypatch):
|
||||
# A stale compiled module of another model family is walked too (its name says unsloth_compiled), and its
|
||||
# probes are mirrored into the modeling module it imports. A failed exactness probe rolls back both, or
|
||||
# the compiled global and sys.modules[__name__] disagree and the block's closure guard fails again.
|
||||
monkeypatch.delenv(ENV, raising = False)
|
||||
names = (
|
||||
"fake_modeling_rollback_gdn",
|
||||
"fake_other_family_modeling",
|
||||
"fake_unsloth_compiled_module_stale",
|
||||
)
|
||||
modeling, other, stale = (types.ModuleType(n) for n in names)
|
||||
for module in (modeling, other, stale):
|
||||
_install_kwargs_mixer(vars(module), _zoo_style_dispatch)
|
||||
stale.__name__ = other.__name__
|
||||
modeling.FakeKwargsGatedDeltaNet.__module__ = modeling.__name__
|
||||
for n, module in zip(names, (modeling, other, stale)):
|
||||
monkeypatch.setitem(sys.modules, n, module)
|
||||
before = {
|
||||
n: {k: vars(m)[k] for k in packing_module._KWARGS_KERNELS if k in vars(m)}
|
||||
for n, m in zip(names, (modeling, other, stale))
|
||||
}
|
||||
monkeypatch.setattr(packing_module, "_hybrid_exactness_probe", lambda target: "forced failure")
|
||||
assert (
|
||||
patch_hybrid_linear_attention_varlen(FakeHybrid(modeling.FakeKwargsGatedDeltaNet()))
|
||||
is False
|
||||
)
|
||||
for n, m in zip(names, (modeling, other, stale)):
|
||||
for k, fn in before[n].items():
|
||||
assert vars(m)[k] is fn, (n, k)
|
||||
|
|
|
|||
|
|
@ -206,10 +206,11 @@ def _route_fla_kernel():
|
|||
if not isinstance(space, dict) or space is adapter.__dict__:
|
||||
continue
|
||||
for key, value in list(space.items()):
|
||||
if (
|
||||
value is caller
|
||||
and isinstance(key, str)
|
||||
and key.endswith("chunk_gated_delta_rule")
|
||||
if value is not caller or not isinstance(key, str):
|
||||
continue
|
||||
# A packing varlen probe keeps the kernel as its `_fn` keyword default; route under it.
|
||||
if key.endswith("chunk_gated_delta_rule") or (
|
||||
key == "_fn" and str(space.get("_name", "")).endswith("chunk_gated_delta_rule")
|
||||
):
|
||||
space[key] = adapter
|
||||
routed += 1
|
||||
|
|
@ -268,12 +269,38 @@ def _namespaces(fn, depth = 3):
|
|||
return out
|
||||
|
||||
|
||||
def _alias_namespace(space):
|
||||
"""The namespace Dynamo resolves `space`'s globals through: an inlined function's globals are
|
||||
read as sys.modules[__name__], and Unsloth's compiled modules import transformers' __name__."""
|
||||
module = (
|
||||
sys.modules.get(space.get("__name__")) if isinstance(space.get("__name__"), str) else None
|
||||
)
|
||||
alias = getattr(module, "__dict__", None)
|
||||
return alias if isinstance(alias, dict) and alias is not space else None
|
||||
|
||||
|
||||
def _route_causal_conv(gdn):
|
||||
from ._gdn_op import _causal_depthwise_conv
|
||||
for space in _namespaces(type(gdn).forward):
|
||||
fn = space.get("causal_conv1d_fn")
|
||||
if callable(fn) and not getattr(fn, "_unsloth_block_adapter", False):
|
||||
if not callable(fn) or getattr(fn, "_unsloth_block_adapter", False):
|
||||
continue
|
||||
probe_defaults = (
|
||||
getattr(fn, "__kwdefaults__", None)
|
||||
if getattr(fn, "_unsloth_varlen_probe_of", None) is not None
|
||||
else None
|
||||
)
|
||||
if probe_defaults is not None:
|
||||
# Packing's varlen probe stays outermost so its boundary handshake still sees the call.
|
||||
inner = probe_defaults.get("_fn")
|
||||
if callable(inner) and not getattr(inner, "_unsloth_block_adapter", False):
|
||||
probe_defaults["_fn"] = _causal_conv1d_adapter(inner, _causal_depthwise_conv)
|
||||
else:
|
||||
space["causal_conv1d_fn"] = _causal_conv1d_adapter(fn, _causal_depthwise_conv)
|
||||
alias = _alias_namespace(space)
|
||||
if alias is not None and callable(alias.get("causal_conv1d_fn")):
|
||||
# Dynamo guards the alias binding: it must be the object the forward actually calls.
|
||||
alias["causal_conv1d_fn"] = space["causal_conv1d_fn"]
|
||||
|
||||
|
||||
def _route_sm_count(modules = None):
|
||||
|
|
|
|||
|
|
@ -767,8 +767,9 @@ def _varlen_seq_idx_applies(seq_idx, args, kwargs) -> bool:
|
|||
tensor = _varlen_input(args, kwargs)
|
||||
if tensor is None or tensor.dim() < 2 or seq_idx.shape[0] != tensor.shape[0]:
|
||||
return False
|
||||
# causal_conv1d_fn takes (batch, dim, seqlen), the others (batch, seqlen, ...).
|
||||
return total in (tensor.shape[1], tensor.shape[-1])
|
||||
# causal_conv1d_fn takes (batch, dim, seqlen), the others (batch, seqlen, ...). Compared with ==:
|
||||
# after a dynamic-shape recompile Dynamo evaluated `total in (...)` False for a matching row.
|
||||
return total == tensor.shape[1] or total == tensor.shape[-1]
|
||||
|
||||
|
||||
def _wrap_mamba2_seq_idx_call(orig, *, hit_attr = "_unsloth_varlen_fused_hit"):
|
||||
|
|
@ -931,6 +932,13 @@ def _kwargs_kernels_available(modules, recurrent_ids = ()) -> Optional[str]:
|
|||
return None
|
||||
|
||||
|
||||
def _kwargs_alias_namespace(namespace):
|
||||
"""The namespace Dynamo resolves `namespace`'s globals through (sys.modules[__name__]), when it differs."""
|
||||
name = namespace.get("__name__") if isinstance(namespace, dict) else None
|
||||
alias = getattr(sys.modules.get(name), "__dict__", None) if isinstance(name, str) else None
|
||||
return alias if isinstance(alias, dict) and alias is not namespace else None
|
||||
|
||||
|
||||
def _install_kwargs_probes(namespace) -> None:
|
||||
if not isinstance(namespace, dict):
|
||||
return
|
||||
|
|
@ -953,6 +961,33 @@ def _install_kwargs_probes(namespace) -> None:
|
|||
|
||||
probe._unsloth_varlen_probe_of = fn
|
||||
namespace[name] = probe
|
||||
# Dynamo reads an inlined function's globals through sys.modules[__name__]; Unsloth's compiled
|
||||
# modules import transformers' __name__, so both namespaces must hold the same probe or the
|
||||
# block compile guards one object and traces the other.
|
||||
alias = _kwargs_alias_namespace(namespace)
|
||||
if alias is not None and callable(alias.get(name)):
|
||||
alias[name] = probe
|
||||
|
||||
|
||||
_FOLDS_IS_EXPORTING = []
|
||||
|
||||
|
||||
def _dynamo_folds_is_exporting() -> bool:
|
||||
# torch 2.10 / 2.11 Dynamo constant-folds torch.compiler.is_exporting() to True inside any
|
||||
# compiled region; 2.12+ reads the real flag. Probed once, eagerly, outside any trace.
|
||||
if not _FOLDS_IS_EXPORTING:
|
||||
|
||||
def probe(x):
|
||||
return x + 1 if torch.compiler.is_exporting() else x
|
||||
|
||||
try:
|
||||
folds = bool(
|
||||
torch.compile(probe, backend = "eager", fullgraph = True)(torch.zeros(1)).item()
|
||||
)
|
||||
except Exception:
|
||||
folds = True # unknown: keep the packed mixer eager, which is always correct
|
||||
_FOLDS_IS_EXPORTING.append(folds)
|
||||
return _FOLDS_IS_EXPORTING[0]
|
||||
|
||||
|
||||
def _wrap_kwargs_mixer_forward(module) -> None:
|
||||
|
|
@ -963,6 +998,9 @@ def _wrap_kwargs_mixer_forward(module) -> None:
|
|||
varlen = getattr(module, "_unsloth_varlen", None)
|
||||
if varlen is None or not _varlen_seq_idx_applies(varlen[1], args, kwargs):
|
||||
return forward_orig(*args, **kwargs)
|
||||
return _packed_forward(varlen, args, kwargs)
|
||||
|
||||
def _packed_forward(varlen, args, kwargs):
|
||||
varlen = _varlen_on(varlen, args, kwargs)
|
||||
if kwargs.get("seq_idx") is None:
|
||||
kwargs["seq_idx"] = varlen[1]
|
||||
|
|
@ -978,6 +1016,13 @@ def _wrap_kwargs_mixer_forward(module) -> None:
|
|||
module._unsloth_varlen_kwargs_hit = True
|
||||
return out
|
||||
|
||||
if _dynamo_folds_is_exporting():
|
||||
# Eager even inside a compiled block: Dynamo folds torch.compiler.is_exporting() to True, so
|
||||
# the hub-kernel wrappers trace their torch reference, which drops seq_idx / cu_seqlens and
|
||||
# runs the packed documents into each other (Qwen3.5-0.8B per-document cosine 0.71). Newer
|
||||
# torch traces the real kernels, and the eager mixer costs 6-20% per step there.
|
||||
_packed_forward = torch.compiler.disable(_packed_forward)
|
||||
|
||||
module.forward = forward
|
||||
module._unsloth_varlen_kwargs_mode = True
|
||||
|
||||
|
|
@ -1442,9 +1487,14 @@ def patch_hybrid_linear_attention_varlen(model) -> bool:
|
|||
kwargs_namespaces = list(_iter_mamba2_install_namespaces(kwargs_modules + hub_short_convs))
|
||||
namespace_rollback = {}
|
||||
for ns in mask_namespaces + kwargs_namespaces:
|
||||
if isinstance(ns, dict) and id(ns) not in namespace_rollback:
|
||||
names = tuple(_MASK_BUILDERS) + tuple(_KWARGS_KERNELS)
|
||||
namespace_rollback[id(ns)] = (ns, {name: ns[name] for name in names if name in ns})
|
||||
# The alias namespace too: _install_kwargs_probes mirrors into it, and a failed probe must undo that.
|
||||
for space in (ns, _kwargs_alias_namespace(ns)):
|
||||
if isinstance(space, dict) and id(space) not in namespace_rollback:
|
||||
names = tuple(_MASK_BUILDERS) + tuple(_KWARGS_KERNELS)
|
||||
namespace_rollback[id(space)] = (
|
||||
space,
|
||||
{name: space[name] for name in names if name in space},
|
||||
)
|
||||
for ns in mask_namespaces:
|
||||
_install_packed_mask_positions(ns, lambda: _PACKED[0])
|
||||
for ns in kwargs_namespaces:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue