5 commits

Author	SHA1	Message	Date
Daniel Han	50b9deacb0	Trim comments in tests (#13844) ... Comment and docstring changes only, in test files, split out of #12968 so the test-only part can merge first.	2026-10-15 05:30:27 -07:00
Daniel Han	7fad6780f9	Probe Studio before reading the deadline in unsloth-studio-update (#11590) ... The health wait checked a whole-second deadline before its first probe. With a short UNSLOTH_STUDIO_UPDATE_HEALTH_WAIT the clock could step past it first, and the update rolled back a Studio it never asked. This is what failed Repo tests (CPU, studio) on main.	2026-09-22 15:12:56 -07:00
oobabooga	3277f0d967	Docker Studio: keep unsloth-studio-update --ref from leaving Studio dead (#10826) ... * Docker Studio: keep unsloth-studio-update --ref from leaving Studio dead * Docker Studio updater: stop on a failed frontend build, fail on a failed restart, restore on interrupt and after --with-deps * Docker Studio updater: recover from a killed run, keep pip on the home path, pin the CUDA stack under --with-deps, honour the npm lockfile and registry * Docker Studio updater: restore after an interrupted release install, and never restore twice or cut the cleanup short * docker: studio-update keeps the previous tree until Studio answers, one updater at a time - an exclusive flock on <src dir>/.src-update.lock for the whole run; a second updater exits 1 and changes nothing. The name shares the scratch prefix so the home linker never links it, and the startup sweep skips it. - the previous source tree and pins stay until the restarted service answers /api/health. A failed restart or a health timeout restores them, starts the previous install again and exits 1. HEALTH_WAIT=0 commits once the restart command succeeds; --no-restart and an unmanaged Studio commit after the import and frontend checks, as before. - restore reinstalls the recorded package identity with --force-reinstall, since pip takes a same-version editable tree as already satisfying the pin. - the oxc validator runtime follows the frontend's lockfile rule: npm ci when a package-lock.json ships with the ref, npm install otherwise, both with the registry override. - the rollback line for an editable install keeps the file:// URI pip recorded; the bare, percent-encoded path is not a valid editable requirement. Tests: lock refusal, restart failure and health timeout put the previous tree back and start it, wait 0 commits, force-reinstall argv, oxc npm ci with and without a lockfile, the editable URI in the rollback file. * Docker Studio updater: put back a previous tree a killed run never committed, and read the install record from the venv not the cwd * Docker Studio updater: record every --packages target for the rollback, and stop when the dependency snapshot fails * Docker Studio updater: keep the package record beside the tree so a killed run's restore can be finished, and say when a restore did not finish * Docker Studio updater: keep the lock file, and make the package record the one marker of an uncommitted update * Docker Studio updater: drop the unverified tree as soon as the previous one is back, and say when the recovery finished * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Docker Studio updater: keep the studio extra on a qualified unsloth spec, and take out what a --with-deps update pulled in on rollback * Docker Studio updater: a failed package listing fails the restore, recovery restarts Studio on the restored install, and the health probe bypasses proxies * Docker Studio updater: a signal after the restart puts the service back on the restored install; the test site carries its own package metadata * Docker Studio updater: --recover finishes a killed update's restore, and the container start runs it before Studio comes up * Docker Studio updater: shorter comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Docker Studio updater: run the repo formatter over the two test files, and unbreak a file that landed on main * Unbreak main: the new Windows idempotency job reinstalls every wheel each run * Unbreak main: the staged-root test anchors moved when the offline fast path took the plain if --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-09-14 05:13:30 -07:00
oobabooga	6d7fd47ede	Docker Studio: keep Studio's data on a volume without pinning its code (#10829) ... A volume on /opt/unsloth-studio kept the first image's code as well as the data, so later images silently ran the old Studio, while no volume lost accounts, chats and trained models on docker rm. The image now keeps Studio's code in /opt/unsloth-studio-app and links each code entry into the home; docker/studio_home.sh (unsloth-studio-home) relinks at build time and at every start, moves an older image's real directories aside to .unsloth-studio-legacy/ (--restore puts them back, UNSLOTH_STUDIO_KEEP_LEGACY=0 deletes), recovers a source tree a killed unsloth-studio-update left aside, and refuses a symlinked legacy path. UV_CACHE_DIR moves into the app dir so cache/ in the home stays data. run.sh mounts a named unsloth-studio volume by default (UNSLOTH_STUDIO_VOLUME). Studio's updater treats a component link that resolves into the app tree as its own install, and the transformers sidecar repair follows a link only into that tree. The Docker Hub page and README document the mount and the rollback path.	2026-09-13 21:28:12 -07:00
Daniel Han	58255747fe	Add Unsloth Docker images (base + Studio) for any NVIDIA GPU host, Ampere through Blackwell (#5748) ... * test_locally.sh: use docker buildx (or DOCKER_BUILDKIT=1) for the build The Dockerfile uses BuildKit-only features (the # syntax=docker/dockerfile:1.7 parser directive and RUN ... <<'PY' heredocs added in dockerfile 1.3+). The legacy builder rejects the --progress flag at the CLI level and would fail later at the heredocs anyway. Detect docker buildx and use it when available (preserves --progress=plain output). Otherwise fall back to plain `docker build` with DOCKER_BUILDKIT=1 exported, which gets the BuildKit features without buildx's nicer formatting. Reproduces the failure path seen on Docker 28.2.2 without buildx installed: unknown flag: --progress ERROR docker build exited 125 * test_locally.sh: require docker buildx, no legacy fallback Docker 28 removed the legacy image builder entirely. Setting DOCKER_BUILDKIT=1 no longer falls back to a builtin builder -- it delegates to buildx, which then errors out if buildx isn't installed: ERROR: BuildKit is enabled but the buildx component is missing or broken. The Ubuntu docker.io package omits buildx by default, so users on that path hit this immediately. Detect missing buildx up front and print exact install commands for apt / dnf / manual binary instead of attempting a fallback that cannot work. * test_locally.sh: pre-flight check for docker daemon connectivity If the user is not in the 'docker' group, every docker command after the pre-flight returns "permission denied while trying to connect to the Docker daemon socket at /var/run/docker.sock". This used to surface as a confusing buildx failure mid-Block-2, but the actual problem is a host permissions issue that's settable up front. Detect by running 'docker info' and checking its exit code (not just grep on its output -- a permission failure prints to stderr and returns non-zero, so the old grep-based check was a silent skip). Also clarify the nvidia-runtime WARN: on Docker 28+ with CDI mode this is a false positive most of the time. The real GPU-attach test is the smoke run in Block 3a, where the container entrypoint catches missing GPUs with an actionable message. * Dockerfile: drop system-python pip/uv bootstrap (PEP 668) Ubuntu 24.04 (noble) marks the system Python interpreter as externally-managed per PEP 668, so: curl get-pip.py | python python -m pip install -U pip uv fails inside the builder image with: error: externally-managed-environment This environment is externally managed The system-level pip and uv were never used: the very next RUN creates the venv at /opt/unsloth-venv, which bootstraps its own pip via the ensurepip module (provided by the python3.12-venv apt package). uv is then installed INTO the venv with the venv's pip, and used from there. Drop the two system-pip bootstrap lines. The venv path is unchanged. Reproduces on any Docker build of the unsloth-blackwell image against a noble base image (which our nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04 is). * smoke_test.py: import unsloth before unsloth_zoo / transformers / trl / peft unsloth_zoo/__init__.py guards against being imported standalone: if "UNSLOTH_IS_PRESENT" not in os.environ: raise ImportError("Please install Unsloth via `pip install unsloth`!") The env var is set by unsloth/__init__.py at import time, so importing unsloth must happen first. The old check_imports() imported xformers, bnb, transformers, trl, peft, then unsloth_zoo -- which fired the guard because unsloth had not been imported yet. Reorder check_imports() to import unsloth (and unsloth_zoo) first, then the rest. check_unsloth_import() becomes a thin re-import to keep the "FastLanguageModel reachable" banner in the output. Same fix the unsloth README has been recommending for years: "import unsloth at the top of your file, before transformers/trl/peft." * Dockerfile: install gcc + g++ + python3-dev in runtime stage Triton's nvidia backend lazily JIT-compiles a small C extension (CudaUtils, in triton/backends/nvidia/driver.py) on first GPU access. Without a C compiler and Python headers in the runtime image, the very first forward pass of any Unsloth model dies with: RuntimeError: Failed to find C compiler. Please specify via CC environment variable. The builder stage has build-essential and python3.12-dev so this worked during the build's verification step (no GPU = no Triton kernel call = no C extension build). But the runtime stage stripped those out for size, so the failure only surfaces when a real user runs training inside the container. Add gcc + g++ + python3.12-dev to the runtime stage. Increases the runtime image by ~250MB, which is the cost of letting Triton JIT correctly. Pre-compiling CudaUtils at build time would need a real CUDA device (the constructor calls cuda runtime functions), so shipping the toolchain is the right trade-off. * Expand arch list to every current x86_64 NVIDIA CC per developer.nvidia.com/cuda/gpus TORCH_CUDA_ARCH_LIST now covers the full set of compute capabilities NVIDIA publishes on https://developer.nvidia.com/cuda/gpus for x86_64 hardware, from Turing onward: sm_75 Turing T4, RTX 20-series, Quadro RTX sm_80 Ampere DC A100, A30 sm_86 Ampere A40, RTX A6000, RTX 30-series sm_89 Ada L4, L40, L40S, RTX 40-series sm_90 Hopper H100, H200, GH200 sm_100 Blackwell DC B100, B200, GB200 sm_103 Blackwell DC B300, GB300 sm_120 Blackwell RTX 50-series, RTX PRO 6000 Blackwell sm_121 Blackwell GB10 (DGX Spark) with +PTX on the highest entry so future arch revisions can JIT. Setting TORCH_CUDA_ARCH_LIST only affects nvcc invocations for any source build the user adds on top of this image (e.g. flash-attn, a custom CUDA op). The prebuilt cu128 wheels already include SASS for sm_70/75/80/86/90/100/120 (verified at build time via torch._C._cuda_getArchFlags()). Ada (sm_89), B300 (sm_103) and DGX Spark (sm_121) GPUs run via JIT-PTX from the nearest available arch. Jetson archs (sm_87 Orin, sm_110 Thor) are intentionally NOT included -- they require aarch64 wheels and this image is linux/amd64 only. Also lower the entrypoint's compute-capability gate from sm_80 to sm_75. Turing GPUs work, with the caveat that bfloat16 is unavailable; the entrypoint prints a NOTE in that case so Unsloth's fp16 fallback isn't a surprise. * Add docker/hf_{push,pull}.sh: simulate docker push/pull against HF Hub HF Hub does not act as a generic OCI registry for arbitrary Docker images -- the registry.hf.space endpoint only serves images that Spaces have built, not images pushed by `docker push`. So we cannot do `docker push huggingface.co/user/repo:tag` for an Unsloth image. For cross-host testing where we want one canonical place to pull from (and Docker Hub credentials are not yet configured), wrap the manual flow into push/pull-shaped commands: hf_push.sh: docker save | pigz | huggingface-cli upload hf_pull.sh: huggingface-cli download | gunzip | docker load This is approximation, not real OCI semantics -- every push uploads the full ~4 GB blob, no layer dedup, no manifest negotiation. Good for testing across A100 / H100 / RTX 6000 boxes; the real release should go through .github/workflows/docker-publish.yml to Docker Hub, which gets layer dedup, multi-arch manifest support, and standard `docker pull` UX for users. Usage: bash docker/hf_push.sh unsloth-blackwell:test danielhanchen/unsloth-blackwell-docker bash docker/hf_pull.sh danielhanchen/unsloth-blackwell-docker unsloth-blackwell-test.tar.gz unsloth-blackwell:test * hf_{push,pull}.sh: use new `hf` CLI, fall back to deprecated `huggingface-cli` In huggingface_hub >= 0.27 the `huggingface-cli` binary is deprecated and prints a "Use hf instead" notice then exits without doing the operation. The previous wrappers ran `huggingface-cli upload/download` silently, treated the deprecation exit as success, and uploaded nothing. Detect the new `hf` binary first and use that. If only the legacy `huggingface-cli` is on PATH (older installs), fall back with a WARN so users know the failure mode if anything goes sideways. Also: hf_pull.sh now asserts the downloaded file is non-empty (`test -s`) so we catch silent download failures before the `docker load` step. * test_locally.sh: use nbformat directly, drop fragile jupyter nbconvert call `jupyter nbconvert --to script nb.ipynb --output nb 2>/dev/null` was silently exiting 0 without producing the output file in some environments (likely because jupyter/jupyter_core wasn't on PATH or nbconvert's --output handling differed across versions). The 2>/dev/null hid the underlying error, and `set -e` did not catch the missing-output case because nbconvert itself returned 0. Switch to a direct nbformat-based conversion: pip install -q nbformat python -c "import nbformat; nb=nbformat.read('nb.ipynb', as_version=4); code='\n\n'.join(c.source for c in nb.cells if c.cell_type == 'code') open('nb.py','w').write(code + '\n')" Smaller dep set, no shell-out to a jupyter wrapper script, and an explicit `test -s nb.py` afterwards catches any silent failure before downstream steps try to read the file. Reproduces the failure on RTX PRO 6000 Blackwell (sm_120, docker 29.2.1, ubuntu 24.04) where nbconvert's CLI silently no-op'd. * test_locally.sh: skip notebook install cells, strip stray jupyter magic The previous nbformat-based conversion dumped raw cell.source for every code cell. The gpt-oss-20B notebook's first cell uses Jupyter !shell magic to install dependencies: !pip install --upgrade -qqq uv !uv pip install -qqq ... \ git+https://github.com/triton-lang/triton.git@0add68... ... Dumped verbatim, the `@0add68...` token tripped the Python parser with "SyntaxError: invalid decimal literal" before training could even start. The container already has unsloth, triton, transformers, etc. baked in, so we don't need the notebook's install cell. Skip any cell whose source contains pip/install markers, and comment out stray !cmd / %magic lines in any other cells. Then assert nb.py parses with ast.parse() before trying to run it -- catches conversion failures up front instead of at training time. Reproduces on RTX PRO 6000 Blackwell (sm_120, fresh Docker 29.2.1 host) where the previous conversion produced an invalid nb.py. * Add linux/arm64 (DGX Spark / Grace) support via QEMU at build time Make the docker image multi-arch so DGX Spark (GB10, sm_121, aarch64) and the Grace-Hopper / Grace-Blackwell SoCs (GH200 arm64, GB200 arm64) pull a natively-built arm64 child from the same manifest. Runtime emulation is NOT involved -- QEMU is used only for the cross-compile step on x86_64 CI runners; consumers on aarch64 hosts get a normal arm64 image and CUDA works as on any other host. Dockerfile: * ARG TARGETARCH; switch unsloth extras between cu128-ampere-torch2100 (amd64, with xformers) and huggingface (arm64, no xformers -- there is no cu128 aarch64 xformers wheel as of 0.0.34, so we fall back to Unsloth's native SDPA path; ~5-10% slowdown but functionally complete). * Build-time torch._C._cuda_getArchFlags() assertion: amd64 still requires sm_120, arm64 accepts sm_120 or sm_121. * Same TORCH_CUDA_ARCH_LIST on both arches; nvcc emits whatever's listed. docker/setup_qemu.sh (new): One-time host setup -- registers binfmt_misc handlers via tonistiigi/binfmt and creates a 'unsloth-multiarch' docker-container buildx builder. Required only on x86_64 build hosts targeting arm64. docker/test_locally.sh: --platform amd64|arm64 flag. Cross-builds verify QEMU is registered, then build through the in-image arch-flags assertion. Smoke + notebook blocks auto-skip when image arch != host arch (CUDA cannot run under user-space QEMU + nvidia-container-toolkit cannot bridge a QEMU guest to a real GPU). .github/workflows/docker-publish.yml: platforms: linux/amd64,linux/arm64 (single manifest, two children). Timeout bumped 60 -> 150 min for the slower arm64-under-QEMU leg. docker/setup-qemu-action@v3 with platforms: arm64 (was implicit before). * docker-publish.yml: native arm64 runner + per-arch digest merge GitHub announced free linux/arm64 hosted runners for public repos (GA Aug 2025) under labels `ubuntu-24.04-arm` / `ubuntu-22.04-arm`. Switching the arm64 leg from QEMU-on-amd64 to a native arm64 matrix runner is ~3x faster and avoids QEMU's occasional flakiness on long cu128 installs. The workflow now: * builds amd64 and arm64 in parallel on their native runners, pushing each as a single-arch image *by digest* (no tag) * stitches both digests into one multi-platform manifest in a follow-up `merge` job, using `docker buildx imagetools create` * keeps a separate buildx cache scope per platform to avoid cross-arch cache collisions Smoke-test job now needs `merge` (was `build`) so it only runs once the final manifest is published. Dockerfile header: replace the speculative aarch64 SASS list with the verified one from pytorch/pytorch v2.10.0 .ci/manywheel/build_cuda.sh (8.0;9.0;10.0;12.0 on aarch64), and note that sm_120 is forward-compatible to sm_121 per PyTorch maintainers -- which is what makes DGX Spark work without an explicit sm_121 SASS section in the wheel. setup_qemu.sh / test_locally.sh --platform stay in place: they're for the local-dev path on x86_64 boxes that don't have arm64 hardware. * Dockerfile: tighten arch-flag assertion + correct fat-binary claims Empirical reality (cuobjdump on the downloaded cu128 wheels): amd64: sm_70 sm_75 sm_80 sm_86 sm_90 sm_100 sm_120 arm64: sm_80 sm_90 sm_90a sm_100 sm_100a sm_120 sm_120a Earlier comments claimed sm_89 native and a "+PTX JIT to sm_121" fallback; both are wrong. cu128 wheels ship NO PTX. Ada (sm_89) runs on sm_86 SASS, B300/GB300 (sm_103) on sm_100, DGX Spark (sm_121) on sm_120 -- all forward-compat WITHIN a major architecture, which is the canonical CUDA rule and ptrblck (PyTorch maintainer) confirmed it directly: "the compatibility ... is also used for e.g. sm_89 with sm_86 and sm_80." Build-time assertion was `any(a in ("sm_120", "sm_121"))` on arm64. Since sm_121 is never in any cu128 wheel, the OR was misleading and could mask a real wheel regression. Tightened to just `assert "sm_120" in arches` on both arches. * Dockerfile: arm64 DGX Spark NVRTC + ptxas fix (cu13 alongside cu128) Empirically (cu128 wheel SASS list `sm_80;90;90a;100;100a;120;120a` on aarch64) the cu128 wheel covers DGX Spark sm_121 via sm_120 binary forward-compat. BUT two CPU-side compilers shipped at cu12.8 do not know sm_121 and need a cu13 swap: (1) torch's bundled libnvrtc.so.12 from CUDA 12.8 rejects sm_121 as a --gpu-architecture. Symlinks libnvrtc.so.13 over it. (2) Triton's nvidia backend runs ptxas. Wheels older than 3.6.0 bundled cu12.8 ptxas which silently downgrades sm_121 to sm_80 (see triton-lang/triton#8335). Bump pin triton>=3.6.0 (3.6 bundles cu13 ptxas) AND install cuda-nvcc-13-0 so the entrypoint can point TRITON_PTXAS_PATH at it as defense in depth. Both fixes are arm64-only (gated on TARGETARCH, ~400 MB on the arm64 image; amd64 is untouched, no sm_121 hardware exists on x86_64). Neither component talks to libcuda, so this does NOT bump the toolkit driver floor away from cu128's 570+. TRITON_PTXAS_PATH is set from the entrypoint (only when the cu13 ptxas actually exists in the image) rather than via a Dockerfile ENV, because ENV is unconditional and Triton errors out if TRITON_PTXAS_PATH points at a nonexistent file. Sources: martimramos/dgx-spark-ml-guide Challenge 14; triton-lang/triton issue #8335; ptrblck PyTorch forum thread on sm_121 fwd-compat from sm_120. * Dockerfile: switch runtime base cudnn-runtime -> base (~2.7 GB lighter) torch wheels ship their own cuDNN/cuBLAS/cuSPARSE/cuRAND/cuSOLVER/cuFFT/ NCCL/cuSparseLt inside torch/lib/, and libtorch_cuda.so's RPATH ($ORIGIN/../../nvidia/cudnn/lib:$ORIGIN/../../nvidia/cublas/lib:...) points at those wheel-bundled copies. The dynamic loader resolves through the wheel, never the system, so the libcudnn/libcublas in the system cudnn-runtime layer are unreachable code on every pull. Verified empirically via `readelf -d torch/lib/libtorch_cuda.so` and confirmed bitsandbytes' NEEDED list resolves against torch's bundled libcudart/libcublas/libcublasLt/libcusparse/libnvJitLink before bnb loads. Triton's .so files have zero CUDA NEEDED entries -- they dlopen through the host driver. Compressed image saving: ~2.7 GB (cudnn-runtime base 2.86 GB -> base 0.10 GB, on amd64; arm64 similar). Uncompressed: ~5 GB. Zero functional impact. Source: Fork 5 image-size audit, May 2026. * entrypoint.sh: correct driver-floor message (570+ unconditionally on cu128) The earlier message had per-arch driver minimums (525/535/555/570) that came from when each chip first got driver support. That's not how CUDA toolkit floors work -- cu128 imposes 570.26+ on EVERY GPU regardless of arch. Only B300 (sm_103) and DGX Spark (sm_121) need a newer driver (580+), and they ship factory with those drivers anyway. External HF README has the same correction applied in temp/hf_readme.md (updated separately when published). * Dockerfile: arm64 install cu13 nvrtc/nvcc directly without cuda-keyring deb The nvidia/cuda base image already registers the CUDA apt repo with its own Signed-By keyring. Installing cuda-keyring_1.1-1_all.deb on top adds a duplicate sources entry with a different Signed-By value, which makes `apt-get update` refuse the entire repo: E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/ The repo URL is monolithic (every CUDA version is served from the same path), so we can install cuda-nvrtc-13-0 + cuda-nvcc-13-0 directly without touching the keyring. Empirically reproduced on the ubuntu-24.04-arm GitHub Actions runner (staging-fork CI run 26360461375); fix verified via the same staging-fork after force-push. * Dockerfile: install vLLM nightly on amd64 for GRPO fast_inference Unsloth's GRPO notebooks (Qwen3_4B-GRPO.ipynb, Qwen3_8B_FP8_GRPO.ipynb, Llama_FP8_GRPO.ipynb, etc.) set `fast_inference=True` which requires vLLM to be importable in the same venv. Install vllm pre-release wheels from https://wheels.vllm.ai/nightly alongside the cu128 pytorch index, holding torch==2.10.0 fixed so uv refuses any vLLM build that would yank torch out from under unsloth. amd64 only -- vLLM does not publish aarch64 wheels yet (vllm-project/vllm#31128 is open). On arm64 the GRPO notebooks that need fast_inference will fail to import vllm; non-GRPO and fast_inference=False paths are unaffected. Gated by ARG INSTALL_VLLM=auto so the install can be disabled for contributors who want a smaller image or are blocked by vllm/torch resolve conflicts during iteration. * Dockerfile: arm64 build aborted by set -e in vLLM auto-gate The case-arm `auto) [ "${TARGETARCH}" = "amd64" ] && WANT_VLLM=1` exits 1 on arm64 (the [ test ] is false and nothing follows ||), which with `set -e` aborts the entire RUN. Replace with an explicit if/then/fi so each arch's auto branch returns 0. Caught by ubuntu-24.04-arm CI on the staging fork. * Dockerfile: re-upgrade numpy after vLLM install (2.2.6 wheel is broken) vLLM 0.19.1 pulls numpy down to 2.2.6 whose wheel ships numpy/_core/ without the tests/ subdir, but numpy/testing/_private/utils.py imports `from numpy._core.tests._natype import pd_NA`. Anything that hits `from numpy import *` (scipy._lib.array_api_compat does) then crashes. unsloth_zoo's gemma patch does `from transformers.processing_utils import Unpack` which touches that path, so `import unsloth` blew up on every GRPO notebook in the vLLM image. Bump numpy to >=2.4 right after the vllm install; vllm still imports fine on numpy 2.4.6 (verified locally). * Fix two upstream regressions surfaced by the Blackwell Docker validation 1. Inductor subprocess GPU invisibility in `--gpus '"device=N"'` containers. The NVIDIA container runtime sets NVIDIA_VISIBLE_DEVICES but not CUDA_VISIBLE_DEVICES; Inductor's compile worker subprocess pool then cannot enumerate the cgroup-pinned device and raises `Could not find an active GPU backend` from triton_helpers.set_driver_to_gpu. Force a single in-process compile thread on that exact fingerprint; opt out via UNSLOTH_FORCE_SINGLE_COMPILE_WORKER=0. Repros: nb/Mistral_v0.3_(7B)-CPT.ipynb, nb/gpt-oss-(20B)-Fine-tuning.ipynb. 2. unsloth_base_fast_generate injects `logits_to_keep` before transformers `_validate_model_kwargs` runs. transformers 5.0 auto-injects the same kwarg inside `GenerationMixin.generate` (utils.py:2527) AFTER the validator, so PEFT-wrapped GRPO models raise ValueError: The following `model_kwargs` are not used by the model: ['logits_to_keep'] Gate the legacy injection on transformers < 5.0 and defensively pop any leaked kwarg on 5.x. transformers 4.57.6 behaviour is preserved. Repros: nb/gpt-oss-(20B)-GRPO.ipynb, nb/gpt_oss_(20B)_RL_2048.ipynb. Both patches are gated and backwards-compatible (transformers 4.57.6 + 5.x, TRL 0.22.2 + 0.27.1 + 1.x, PEFT 0.19.x). * docker: add Dockerfile.studio extending the Blackwell image with Unsloth Studio The base unsloth-blackwell image ships the `unsloth` CLI but refuses to start `unsloth studio` until the dedicated Studio venv is laid down under UNSLOTH_STUDIO_HOME by install.sh. Build it once and commit the result as an opt-in companion tag (`:studio`) instead of bloating the base image. Build: docker buildx build --build-arg BASE_TAG=test \ -f docker/Dockerfile.studio -t unsloth-blackwell:studio docker/ Run: docker run --rm --gpus '"device=0"' -p 8888:8888 unsloth-blackwell:studio Open http://localhost:8888. Inference (llama.cpp CPU + GPU) and training are both available. First-boot admin password lands in container logs and at /opt/unsloth-studio/auth/.bootstrap_password. * Address reviewer findings on PR #5748: 4 release-path bugs Round-trip with the reviewer.py 12-persona pass surfaced four real issues. Fix all four in this PR so the new Docker release path is self-consistent. 1. docker/smoke_test.py used `import xformers` unconditionally, which guarantees a failure on arm64 (built with `[huggingface]` extras to skip xformers since it has no aarch64 cu128 wheel). Wrap the import in try/except so the same smoke script validates both arches. 2. unsloth/_gpu_init.py forced `TORCHINDUCTOR_COMPILE_THREADS=1` before `import unsloth_zoo`, but `patch_torch_compile` in unsloth_zoo main pops that env var in non-debug mode. After unsloth_zoo init the guard was effectively undone, so cgroup-pinned `docker --gpus '"device=N"'` containers still spawned the Inductor subprocess pool that cannot enumerate the GPU. Set `torch._inductor.config. compile_threads = 1` directly post-import-torch and re-populate the env var so `determine_compile_threads()` in the zoo options dict also returns 1, regardless of whether the zoo-side fix from PR #694 has shipped yet. 3. docker-publish.yml UNSLOTH_REF build-arg defaulted to `'main'` for tag pushes and scheduled runs, so a `v1.2.3` release image would contain whatever `main` happened to be at build time, not v1.2.3. Pick the tag's `github.ref_name` for tag events and `github.sha` for branch/schedule events. 4. The smoke-test job pulled `:latest` regardless of which tag the merge job had just published, so tag/schedule/sha publishes were never actually validated. Re-run docker/metadata-action with the same config the merge job used, then smoke-test the first tag from its output. All four changes are gated and backwards-compatible. * Remove async_task_outputs from repo (accidentally committed) The previous commit pulled in 51 transient async-task-output markdown files from the local workspace's reviewer.py runs. Drop them and add the directory to .gitignore so it cannot recur. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker/Dockerfile.studio: keep source for the editable install install.sh --local installs unsloth into the Studio venv as an editable package keyed to the just-cloned source tree. We were rm-rf'ing that tree in the same RUN; the resulting `unsloth_cli` import then failed at container start with `ModuleNotFoundError: No module named 'unsloth_cli'`. Clone the source directly under UNSLOTH_STUDIO_HOME/src so it persists in the image layer, and strip only .git to save ~120MB. * docker: add timm + addict to the base image Two vision-notebook deps that ship by reference rather than via unsloth extras: transformers' Gemma3N + TimmWrapperModel needs `timm`, and DeepSeek-OCR's dynamic modeling file requires `addict`. Both are tiny (~30MB combined). Including them in the base unified resolve avoids hitting `ImportError: TimmWrapperModel requires the timm library` or `ImportError: This modeling file requires the following packages that were not found in your environment: addict` after the user has already downloaded the model. Repros: nb/Gemma3N_(4B)-Vision.ipynb (timm), nb/Deepseek_OCR_(3B).ipynb (addict). * Address reviewer-2 findings on PR #5748 Round-2 of the 12-persona reviewer.py pass found 17 issues. Address the P1s + the regression-class P2s in this commit; the remaining nits are left for a follow-up cleanup pass. 1. unsloth/_gpu_init.py: the `NVIDIA_VISIBLE_DEVICES in os.environ` check triggered for every NVIDIA-runtime container including `--gpus all` (NVIDIA_VISIBLE_DEVICES=all is the default). Gate strictly on a non-special device list. Also drop the precondition that the env var was absent: if the user already pinned TORCHINDUCTOR_COMPILE_THREADS=1 we should still plant the UNSLOTH_FORCE_SINGLE_COMPILE_WORKER sentinel so the zoo-side patch knows to preserve the forcing. 2. unsloth/_gpu_init.py: after the post-`import unsloth_zoo` reassertion, monkey-patch `unsloth_zoo.temporary_patches.common.determine_compile_threads` to return 1, so any later `torch.compile` call that rebuilds the options dict still sees the single-worker forcing even if a downstream patch_torch_compile pops the env var again. 3. docker/Dockerfile: torchaudio==2.11.0 mismatched the torch==2.10.0 release pairing; pin to 2.10.0 so the ABI is correct and the audio stack matches torch/cu128. 4. docker/Dockerfile: drop `12.1+PTX` from TORCH_CUDA_ARCH_LIST. The cu128 toolkit compiler does not know about compute_121; the trailing PTX entry forced nvcc to emit a `sm_121` gencode that breaks any in-container source builds. 5. docker/smoke_test.py: the device-capability floor said `cap[0] < 8`, rejecting Turing (sm_75) while the Dockerfile + entrypoint advertise sm_75 as supported. Lower the smoke floor to sm_75 and print a hint that bf16 is not available on Turing. 6. docker/run.sh: `-it` is unconditional; CI / non-TTY invocations died with "the input device is not a TTY". Probe `[ -t 0 ] && [ -t 1 ]` first. Also remove `set -x` which echoed the forwarded HF_TOKEN / WANDB_API_KEY / UNSLOTH_LICENSE values to stdout. 7. docker/test_locally.sh: `-e HF_TOKEN="${HF_TOKEN:-}"` either pasted the secret verbatim into the process arg list or shadowed any in-container value with an empty string. Forward conditionally. 8. .github/workflows/docker-publish.yml: gate `latest` on default branch AND on `unsloth_ref` not being overridden via workflow_dispatch. Otherwise a maintainer testing a feature SHA from main could overwrite `:latest` with non-main source. 9. docker/Dockerfile.studio: add an `UNSLOTH_STUDIO_REF` build-arg so the Studio companion image is pinned to a known unsloth ref instead of cloning `main` whenever it builds. * Remove individual_reviews from repo (accidentally committed) reviewer.py drops the 12 raw per-persona review markdown files under individual_reviews/ in the current working tree. Drop them here and add the directory to .gitignore so it cannot recur. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests/studio: assert losses_per_step matches max_steps, not stale 7 PR #5537 bumped max_steps from 7 to 30 but the post-train assertion still hardcoded the old count, so every fresh run that reaches the post-train phase fails on `expected 7 logged steps, got [30 floats]`. Derive the expected count from `config.max_steps` and add a `train_result["train_steps"]` cross-check so the gate self-updates with future sweep changes. * Address 3 MAJOR review findings on the docker PR 1. Stop leaking secrets via docker run -e VAR=VALUE argv (run.sh, test_locally.sh) `docker run ... -e HF_TOKEN=hf_xxx ...` puts the literal token in the docker CLI's argv, which is visible to any user on the host via `ps auxe` / `/proc/<pid>/cmdline` for the lifetime of the process. Switch to the dash-only form `-e HF_TOKEN`, which tells docker to read the value from the parent shell's env and never appears in argv. Same fix for WANDB_API_KEY and UNSLOTH_LICENSE in run.sh and HF_TOKEN in test_locally.sh. 2. Stop stripping numpy/tests/ in the runtime layer (Dockerfile) The Dockerfile explicitly upgrades numpy >= 2.4 because numpy 2.2.6 shipped a stripped wheel where `from numpy._core.tests._natype import pd_NA` fails. Numpy 2.4 restores `numpy/_core/tests/`, then the existing `find ${VENV} -name tests -exec rm -rf {} +` deleted it again -- re-introducing the same broken-import state on the deployed image (the build-time verification at line 220 runs BEFORE the strip so it passed). Whitelist numpy's tests directories from the strip; keep stripping the rest. 3. Align :latest tag gate between merge and smoke-test jobs (.github/workflows/docker-publish.yml) merge job: enable = is-default-branch AND unsloth_ref == '' smoke-test job: enable = is_default_branch only On `workflow_dispatch`, `github.event.inputs.unsloth_ref` defaults to "main" (not ""), so the merge step skipped `:latest` but the smoke step still emitted `:latest` as tags[0]. The smoke step then `docker pull`-ed a prior `:latest` from Docker Hub instead of the image just merged -- so the smoke test verified the OLD image, not the new one. Copy the merge step's exact `enable=` expression into the smoke-test step so the two stay byte-identical and a workflow_ dispatch run validates whatever was actually merged. * docker: round-3 review fixes (concurrency, lockfile wording) - docker-publish.yml: add `concurrency: docker-publish-${{ github.ref }}` (cancel-in-progress: false) so two pushes to main never race the `:latest` retag. Don't cancel in-progress runs -- the build is expensive and a half-built image left around is worse than a stale :latest for a few minutes. - Dockerfile: soften the requirements.lock.txt comment. `pip freeze` captures versions but not wheel hashes, and several deps resolve from VCS / nightly indexes that float, so the file is not actually byte-reproducible. Reword as an "informational pin record". * docker-publish: pin UNSLOTH_ZOO_REF on tag pushes Previously, UNSLOTH_REF was pinned to the triggering tag (e.g. v2026.5.8) but UNSLOTH_ZOO_REF was hardcoded to main. That made release-tag images ship a zoo from whatever was on main at build time rather than the zoo release cut alongside that unsloth tag, so a 2026.5.8 tag image could install a zoo from days later. Mirror the tag branch of UNSLOTH_REF. SHA-based branch pushes still fall through to main because the unsloth SHA does not exist in the unsloth-zoo repo. workflow_dispatch still honours the unsloth_zoo_ref input. * docker/run.sh: translate UNSLOTH_GPUS index selectors to device= form The header docstring advertises UNSLOTH_GPUS values like "0" and "0,1" but Docker reads a bare integer for --gpus as a COUNT, not an INDEX. UNSLOTH_GPUS=0 was therefore exposing zero GPUs, and the entrypoint's GPU check refused to start. Wrap bare-int and comma-list inputs as "device=$GPUS" so the documented values do what they say; "all" and already-quoted device= selectors pass through unchanged. * docker/test_locally.sh: fail fast + pin notebook fetch to immutable SHA Two small fixes: 1. The fallback build-context refresh used `git pull --ff-only | tail`, which on this script (set -uo pipefail, no -e) silently masked any non-zero exit from pull. A failed refresh would then quietly build from a stale clone. Wrap both clone and pull in `if ! ...; then fail` so refresh failures abort the run with a clear message. 2. The gpt-oss-20B notebook was fetched from notebooks/main, which is mutable. Pin to the current immutable SHA (efe20c9) via NB_REPO_REF so reruns of this script don't silently change semantics when notebooks/main rolls forward. Override via env when you want to verify a newer notebook. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: ship Jupyter, Studio and prebuilt llama.cpp out of the box Base image (docker/Dockerfile): - Install JupyterLab + notebook + ipywidgets in a separate pure-Python uv pass so the cu128 pin set cannot move; EXPOSE 8888. - Bake the prebuilt llama.cpp bundle into /opt/unsloth/llama.cpp at the runtime stage using studio/install_llama_prebuilt.py from the same UNSLOTH_REF (sha256-verified, portable CUDA bundle since the build host has no GPU; arm64 resolves the linux-arm64-cuda13 bundle). Export UNSLOTH_LLAMA_CPP_PATH so unsloth_zoo's save_pretrained_gguf finds it and never reaches the interactive install prompt or a source build. - Optional github_token BuildKit secret for the resolver's API calls on shared CI runner IPs. Entrypoint: UNSLOTH_ALLOW_CPU=1 degrades a missing GPU to a warning so Docker Desktop on macOS / Windows-without-WSL2-GPU and plain CPU hosts can run Jupyter, GGUF tooling and Studio chat; with a GPU visible the normal pre-flight still runs. Full image (docker/Dockerfile.studio): now mirrors the production service set under supervisord - Studio on 8000, JupyterLab on 8888, key-only sshd on 22 (enabled only when PUBLIC_KEY/SSH_KEY is set). Points Studio's llama.cpp dir at the baked bundle to skip a duplicate download, accepts any git ref via fetch+checkout (CI passes commit SHAs), and FROMs a digest-pinned BASE_IMAGE. Publish workflow: base image moves to the base-* tag namespace; new build-studio/merge-studio jobs publish the full image as :latest (hub parity with the previous production image, which shipped Studio + Jupyter + SSH). Studio builds FROM the exact base manifest digest published by the same run. GPU smoke job now also boots the full image and probes Studio /api/health and Jupyter /api. run.sh: UNSLOTH_GPUS=none, UNSLOTH_ALLOW_CPU forwarding, UNSLOTH_PORTS publish flags, CPU-mode and Jupyter usage examples. * docker: pin the llama.cpp bake by target arch, add docker_confirm.sh The first bake attempt reused studio/install_llama_prebuilt.py, but that resolver selects a bundle for the CURRENT host: on a GPU build host /proc/driver/nvidia leaks into docker build and the resolver goes down the CUDA path with no readable driver runtime (chosen_asset=none, exit 2), while on a GPU-less CI runner it would resolve a CPU bundle instead. Both violate the image's build-host-independence rule. fetch_llama_prebuilt.py pins by build target only: amd64 takes the linux-x64-cuda12-portable bundle, arm64 the linux-arm64-cuda13-portable bundle (DGX Spark / Grace), both sha256-verified against the release's llama-prebuilt-sha256.json. convert_hf_to_gguf.py plus gguf-py/ are hydrated from the same release's source tarball so the converter's tensor mappings match the binaries, mirroring unsloth_zoo's _hydrate_converter_sources layout. LLAMA_PREBUILT_TAG build-arg overrides the pinned release. docker_confirm.sh: one-command confirmation script for any machine (Linux / WSL2 / macOS) following the staging confirm-script conventions: host + docker + GPU detection with CPU-mode auto-fallback, image pulls, in-container torch.cuda check, 5-step LoRA training smoke, baked llama.cpp verification, full-image boot probing Studio /api/health and JupyterLab /api, PASS/WARN/FAIL summary with RESULT line. * docker: whitelist new build-context files, add docker_confirm.ps1 The dockerignore uses an everything-out whitelist; fetch_llama_prebuilt.py (base bake) and supervisord.conf + studio_launch.sh (Dockerfile.studio) need explicit entries. docker_confirm.ps1 is the Windows Docker Desktop counterpart of docker_confirm.sh. * docker/studio: deterministic Studio install inside the image build Two failures from the first in-image Studio install, both rooted in install.sh probing the build host: 1. setup.sh aborted on the pre-linked llama.cpp dir: 'already exists and is not marked as a Studio-owned llama.cpp install'. The dir is the image's baked prebuilt, provisioned exclusively for Studio, so write the .unsloth-studio-owned marker next to the binaries. 2. With no GPU and no nvidia-smi in the build container, install.sh fell back to cu126 torch wheels for the Studio venv (and would pick cpu wheels on a CI runner without /proc/driver/nvidia), so the published image's Studio venv would depend on which host built it and could not train on Blackwell. get_torch_index_url now honours an explicit UNSLOTH_TORCH_INDEX_FAMILY override naming the index leaf (cu128, cu130, rocm7.2, cpu, ...). The resolved family flows into UNSLOTH_TORCH_BACKEND, which install_python_stack.py already consumes, so the whole downstream chain follows the pin. Dockerfile.studio sets cu128 on amd64 and cu130 on arm64 (DGX Spark / Grace). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: mirror the llama.cpp bake into build/bin so Studio setup reuses it Studio's setup.sh provisioning runs install_llama_prebuilt.py, whose host-probing cannot succeed inside an image build, so it fell back to a CPU-only llama.cpp source build layered over the baked CUDA bundle. setup.sh skips that fallback when build/bin/llama-server and build/bin/llama-quantize are executable, so hardlink the installed bundle into build/bin: zero extra bytes, $ORIGIN rpath still resolves, and no symlink cycle when setup.sh later relinks the root quantizer to build/bin/llama-quantize. * docker_confirm.sh: rename unused poll counter for shellcheck SC2034 * docker: review fixes from the 8-reviewer pass and staging CI entrypoint.sh: a container started without a GPU request has no nvidia-smi at all (the toolkit injects it), so the old check 1 reported 'CUDA runtime in this image is broken, re-pull' for the most common user error. Fold the missing-binary case into the actionable 'No GPU visible' message and document the CPU-only option (UNSLOTH_ALLOW_CPU=1). run.sh / test_locally.sh: guard empty-array expansions with the ${arr[@]+...} form; bash 3.2 (macOS /bin/bash) treats "${empty[@]}" as unbound under set -u, which broke the documented macOS CPU path. studio_launch.sh: exclude *_TOKEN, *_API_KEY, *_PASSWORD, *_SECRET, *_LICENSE from the env snapshot written for SSH sessions; secrets stay in process env only, never on disk. supervisord.conf / Dockerfile.studio: pin HOME=/root for the studio and jupyter programs (jupyter would silently fall back to token auth if HOME were unset), default JUPYTER_PORT and UNSLOTH_ENABLE_SSHD at the image level so a direct supervisord invocation cannot hit a bad %(ENV_*)s expansion, and document the root-services decision (non-root parity with the previous production image is a tracked follow-up). docker_confirm.ps1: mirror the bash script's GPU selector translation so GPUS=0 / 0,1 select devices instead of silently using all GPUs. docker-publish.yml: studio cache scope moves to mode=min; a mode=max cache of a ~24GB image would evict everything else in the 10GB GHA quota for no hit-rate gain. * docker: second review pass fixes - Dockerfile: lift numba past vllm's 0.61.2 pin after the numpy>=2.4 re-upgrade; 0.61.2 refuses numpy 2.3+ at import time and the stack cannot move numpy down. Verified numba 0.65 + numpy 2.4.6 + vllm import cleanly together. - docker-publish.yml: resolve UNSLOTH_ZOO_REF in a step that mirrors the pushed tag only when the tag exists in unsloth-zoo (the zoo currently cuts no tags, so blind mirroring broke every tag publish); falls back to main. - Dockerfile.studio: Studio venv stays on cu128 for arm64 too, matching the base venv (cu130 wheels would lift the driver floor to 580+), and gets the same NVRTC cu13 swap for DGX Spark / GB10 sm_121 support. - docker_confirm.sh: do not drop to CPU mode when docker info lacks a nvidia runtime entry; CDI installs and Docker Desktop WSL2 expose GPUs without one. The phase 3 --gpus probe is now the authority. - docker_confirm.ps1: GPU selector built as an args array; comma device lists get version-aware CSV quoting (native arg passing changed in PowerShell 7.3). - studio_launch.sh: no fixed Jupyter default password; generate a random one and print it when JUPYTER_PASSWORD is unset. Env snapshot for SSH sessions now written via shlex.quote instead of sed so values with quotes or command substitution cannot break or inject into /etc/profile.d. - install.ps1: honour UNSLOTH_TORCH_INDEX_FAMILY like install.sh does. * docker ci: aggressive runner disk reclaim before image builds A staging run of the studio image build died with ENOSPC during the Studio venv install: the hosted runners' default free space does not fit the base image plus buildkit state plus the Studio layer. Drop all unused preinstalled toolchains and the runner's preloaded docker images in both build jobs. * gpu_init: satisfy the import-hoist lint in the compile-thread patch The checker does not count attribute assignment on an aliased module import as a use and flagged _zoo_common as added-but-unused. Set the attribute through importlib.import_module instead; importlib is already a module-level import here. Behaviour unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: zstd + matplotlib for out-of-the-box notebook coverage Running the published unslothai/notebooks set inside the image surfaced two gaps: the Ollama export notebook installs ollama in-container and that installer needs zstd for extraction, and DeepSeek-OCR's trust_remote_code modeling file imports matplotlib unconditionally (plotting is also simply expected in a Jupyter image). * docker: audio decode out of the box (ffmpeg + matched torchcodec bake) The TTS/STT notebooks decode datasets Audio features through torchcodec, which fails three different ways on a fresh image: the PyPI wheel pairs with the cu13 torch line and dlopens libnvrtc.so.13; builds newer than 0.10 reference torch 2.11+ symbols; and the matching +cu128 build dlopens torch and NVIDIA runtime libraries that live inside the venv where the loader cannot see them. Bake ffmpeg, torchcodec==0.10.0 from the cu128 channel, nvidia-npp-cu12, and register the venv lib dirs via ld.so.conf.d (not LD_LIBRARY_PATH, so the llama.cpp bundle keeps winning through its own RUNPATH). Verified in-container: AudioDecoder imports and llama-server still resolves its bundled libraries. * docker: split the torchcodec bake across build stages The wheel install belongs in the builder (the venv copy carries it), but the ld.so.conf.d registration and the import check belong in the runtime stage: the conf file does not survive the stage copy and the import needs ffmpeg, which only the runtime stage installs. * docker: install vLLM on the arm64 leg too and probe it in the confirm scripts PyPI has shipped aarch64 abi3 wheels for every vLLM release since 0.17, so the arm64 skip rested on a stale premise. With torch held at 2.10.0 the resolver lands on vllm 0.19.1 (the release pinning torch==2.10.0) on both arches; verified by cross-resolving the exact index set for aarch64-unknown-linux-gnu. amd64 keeps fail-loud semantics. arm64 is fail-soft because the aarch64 wheels are newer and their GPU kernels get validated on Spark hardware via docker_confirm.sh rather than in CI; on failure the fallback uninstalls vllm and restores the numpy/numba floor so a partial install cannot break import unsloth (numpy 2.2.6 ships a broken numpy.testing). The install steps form an explicit && chain instead of a set -e subshell: POSIX shells disable errexit inside condition contexts (verified on dash), so a (set -e; ...) condition would mask failures. Both confirm scripts gain a 5b vLLM phase: ok on import, bad if missing on x86_64, warn on other arches where fast_inference=True is best-effort. * docker: add wget to the runtime image Notebooks fetch sample assets with !wget; without the binary the shell prints not-found to stderr, the cell still exits zero from Jupyter's perspective, and the next cell crashes confusingly on the missing file. The Whisper notebook died exactly this way in the validation matrix. * ci: retrigger after bulk-cancelled runs * docker: unbreak standalone vllm serve (ninja-build + flashinfer-jit-cache) The notebook validation matrix caught the synthetic-data notebook dying because the vllm server SyntheticDataKit launches never came up. Two layers to the failure: 1. flashinfer's cpp_ext JIT shells out to ninja. The pip ninja lives in the venv bin, which subprocesses like vllm serve do not always inherit on PATH, so the JIT failed with exit 127. Install ninja-build so the binary is reachable from any PATH. 2. With ninja present the JIT still cannot succeed for device code: the runtime image deliberately ships no nvcc. Bake flashinfer-jit-cache (cu128) so ops missing from the cubin package (fmha_gen on sm_100a was the repro) come precompiled. In-process GRPO never hit this because unsloth-zoo blocks the FlashInfer JIT path; standalone vllm serve gets no zoo patches. Fail-soft on the jit-cache for arches without a wheel; the vLLM chain itself stays fail-loud on amd64. * dataprep: detect vllm 0.19 server readiness (renamed log line, stderr) SyntheticDataKit.from_pretrained waits for 'Starting vLLM API server on' in the child's stdout before declaring the server up. vLLM 0.19 renamed the line to 'Starting vLLM server on ...', so the regex never matched, the 1200 s readiness timeout expired with a perfectly healthy server, and the launcher tore it down; every downstream synthetic-data-kit step then failed on missing files. Accept both wordings, watch stderr too (vLLM has moved its logging between pipes across versions), and bail out of the wait early if the child exits. With this plus the ninja-build and flashinfer-jit-cache image fixes the Meta synthetic data notebook goes from a 21 min timeout-and-fail to a 3 min pass inside the container. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: honor UNSLOTH_TORCH_INDEX_FAMILY in CUDA repair path, assert torch CUDA family at studio image build _detect_cuda_torch_index_url now respects the explicit family override before probing nvidia-smi, matching install.sh get_torch_index_url and install.ps1 Get-TorchIndexUrl. Without it, a GPU-less environment falls back to cu126 wheels which lack sm_100/sm_120 kernels and break training on Blackwell. ROCm repair path is intentionally unchanged. Dockerfile.studio now fails the build if the Studio venv torch local version tag does not match the pinned TORCH_FAMILY, so a studio ref whose installer ignores the override can never ship a silently wrong image. Metadata-only check so QEMU arm64 builds do not need to load torch. * Fix import on driverless hosts under UNSLOTH_ALLOW_CPU=1 unsloth_zoo.device_type.get_device_type() deliberately returns cuda when UNSLOTH_ALLOW_CPU=1 and no accelerator exists (CPU CI, Docker Desktop without GPU passthrough), but the DEVICE_TYPE == cuda import paths probed torch.cuda.get_device_capability() unconditionally and raised RuntimeError: Found no NVIDIA driver. Guard the module level probes with torch.cuda.is_available(); bf16 stays enabled in the degrade branch since CPU bf16 kernels exist while fp16 ones largely do not. Healthy GPU hosts take the original branches unchanged. Found by the cross platform CPU mode validation of the Docker images. * docker: mirror soname symlinks into llama.cpp build/bin, assert the relinked quantizer executes The build/bin hardlink mirror skipped symlinks, so the soname links (libllama-common.so.0 and friends) never reached build/bin. Studio's setup.sh relinks the root llama-quantize to build/bin/llama-quantize, whose RUNPATH is $ORIGIN, so the loader failed with libllama-common.so.0 not found and GGUF export from Studio died with No working quantizer found, then hit the interactive source-build prompt in a non-TTY export subprocess (EOFError). Mirror same-directory soname symlinks into build/bin and extend the bake sanity check to execute llama-quantize from both the install root and build/bin. Dockerfile.studio now also runs the studio-visible quantizer after install.sh so a regression fails the image build instead of runtime exports. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker/studio: make the quantizer build assertion content based llama-quantize exits nonzero on --help/--version while still printing usage, so a bare invocation fails the build even when the binary is healthy. Grep for the usage banner instead; a loader failure prints error while loading shared libraries and no usage text. * docker_confirm: accept locally built images when pull fails A locally built tag (test_locally.sh or docker build) is not on a registry, so the pull phase reported hard failures on a machine that was actually fine. Degrade to a warn when the image is present locally; missing images still fail. * docker: ship cuda-nvcc and cudart-dev in the runtime image flash-linear-attention's TileLang backend JIT-compiles CUDA kernels via nvcc at runtime for gated-delta-rule models (Qwen3.5 family). The -base image only ships runtime libraries, so Studio vision training of unsloth/Qwen3.5-2B died on the first backward pass with [Errno 2] No such file or directory: /usr/local/cuda/bin/nvcc. Install cuda-nvcc and cuda-cudart-dev matching the image CUDA version and assert nvcc is executable at build time. Found by driving a real Qwen3.5-2B training run through the Studio UI in the image. * docker: bake soundfile, evaluate, tensorboard for notebook deps The TTS notebooks (Sesame CSM, Orpheus) read audio via soundfile, the Whisper notebook computes WER via evaluate, and TrainingArguments defaults report_to to tensorboard. These are declared by notebook pip cells that the in-image notebook runner neutralises (deps are meant to be prebaked), so without them those notebooks die on import. All are pure-Python or self-contained wheels and never name torch, so the cu128 pin set is undisturbed. * docker: add jiwer, langid, easydict, protobuf to baked notebook deps Continuation of the notebook-dep prebaking: the in-image notebook runner neutralises pip cells, so declared deps must be prebaked. evaluate's WER metric imports jiwer (Whisper), DeepSeek-R1 GRPO's reward uses langid, some vision trust_remote_code files need easydict, and sentencepiece tokenizer conversion needs protobuf. All pure-Python; torch pin intact. * docker: notebook deps, image size cuts, per-notebook transformers Notebook dependency coverage (base Dockerfile): - Bake omegaconf, einx, librosa, decord, ftfy so the TTS/STT and vision notebooks stop dying on a silent No module named X. Installed in the notebook-deps layer (after the torch/vLLM resolve) with an assertion that the resolve did not move torch 2.10.0 / numpy>=2.3 / numba>=0.65. Image size (no functional change): - Base: prune npp to the two libs torchcodec actually dlopens (libnppicc + libnppc), drop link-time-only .a archives and the nvshmem device bitcode. Headers (torch/include etc) are kept so causal-conv1d / mamba-ssm still build at notebook time with --no-build-isolation. - Studio: pin the Studio venv to Python 3.12 (matches base) so its nvidia-*-cu12 wheels are byte-identical to the base venv's, then symlink the heavy arch-independent CUDA libs (cudnn/cublas/nccl/...) into the base venv copy. cuda_nvrtc and cuda_runtime are excluded (the arm64 nvrtc swap mutates nvrtc in place). Also remove the build-only frontend node_modules (runtime serves the committed dist). Studio image drops ~4.8GB. Per-notebook transformers version, run notebooks unchanged: - Bake coherent transformers sidecars (4.57.6 default + 5.3.0/5.5.0/5.10.2), each transformers==X with its matched huggingface_hub/tokenizers/ safetensors installed --no-deps into its own dir. Companion versions are resolved at build time so they satisfy each transformers' requirements. - unsloth_nb_compat.py: pick the sidecar from the notebook's pin or the model name and activate it (prepend to sys.path) before any ML import, without touching the base cu128 torch/vLLM/unsloth stack. - pip/uv shim on PATH: a notebook install cell becomes safe and idempotent inside a kernel (keeps the baked stack, records the requested transformers for its sidecar); passthrough to the real tool everywhere else. - IPython startup hook for manual JupyterLab, and unsloth-run for the headless driven path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: pre-load unslothai/notebooks into JupyterLab, edit-safe refresh JupyterLab now opens with the unslothai/notebooks collection already present, so people can open and run a notebook directly without a git clone or wget. - Bake the repo into the image as a read-only template at /opt/unsloth-notebooks (~206MB, .git stripped, build commit recorded). Inherited by the studio image. - On boot the entrypoint populates /workspace/unsloth-notebooks from the template (instant, works offline) and best-effort refreshes from GitHub, but only when upstream has actually advanced (cheap git ls-remote gate, no download otherwise). - The user's edits always win. We record the content hash of every file we write; on refresh a file whose hash differs from what we last wrote is treated as user-modified and is left untouched, so the refresh only updates files the user has not changed and adds new ones. It never overwrites an edited notebook and never produces merge conflicts. Verified: an edited notebook stays the user's version across repeated upstream changes. - Fully best-effort and gated: UNSLOTH_SKIP_NOTEBOOK_SYNC=1 disables it, UNSLOTH_SKIP_NOTEBOOK_REFRESH=1 keeps the baked copy and never hits the network. Offline boots keep what is there and never error. base 18.45 -> 18.67GB, studio 24.88 -> 25.10GB (+~206MB baked notebooks). * docker: notebook refresh ignores header/footer-only upstream changes The boot-time refresh now compares only the tutorial body (the non-boilerplate cells) when deciding whether to update an untouched notebook. If only the install header, announcements, or footer moved upstream, the user's file is left as-is so it is not churned. Notebooks the user has edited or run are still kept untouched, and non-notebook files keep the whole-file refresh. Adds unsloth_nb_content_sig.py to segment head/middle/tail and bakes it into the image. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: heal deleted notebooks on boot + fix notebooks helper dockerignore The boot-time notebook sync now restores notebooks the user deleted, on every boot, from the baked template (offline, even when upstream has not advanced). It only restores files that are missing, so it never resurrects or overwrites an edited notebook, and the GitHub refresh still bumps a restored file to the latest upstream. Opt out with UNSLOTH_KEEP_DELETED_NOTEBOOKS=1. Also add unsloth_nb_content_sig.py to docker/.dockerignore's allowlist; it was referenced by the Dockerfile COPY but excluded from the build context, which broke the image build. * docker: track latest llama.cpp release + show its update banner in Studio Two related changes to the baked llama.cpp prebuilt. 1. Dynamically follow the newest unslothai/llama.cpp release. build.sh resolves the latest release tag (following the /releases/latest redirect, no API token) to a concrete tag and passes it as LLAMA_PREBUILT_TAG, so the layer cache busts only when upstream publishes. The Dockerfile default is now "latest" and fetch_llama_prebuilt.py resolves it the same way, so a plain `docker build .` also tracks latest. Pin LLAMA_PREBUILT_TAG to a concrete tag for a reproducible, frozen build. 2. Make the in-app "newer llama.cpp available" banner work inside the image. Studio's freshness check (utils.llama_cpp_freshness.check_prebuilt_freshness) keys off tag / release_tag / published_repo in UNSLOTH_PREBUILT_INFO.json -- the schema install_llama_prebuilt.py writes. The image bakes the bundle directly, so the marker was the release tarball's own, which only carries upstream_tag / source_repo; the freshness check then bailed with installed_tag=None and could never report "behind", hiding the banner. fetch_llama_prebuilt.py now augments the baked marker with those keys (setdefault, no build timestamp so the layer stays byte-identical). A fresh build is on latest -> no banner; once upstream publishes a newer release the banner appears, as verified against the real freshness backend. * docker: add unsloth-studio-update for in-place Studio updates Updating Studio in the container previously meant pulling a fresh ~25GB image (or at best the ~6GB fused Studio layer) for what is usually a small Python/UI change. Add a baked helper so a running container can update in place: docker exec <container> unsloth-studio-update It updates only the Studio packages -- the backend code and the pre-built frontend, which ships inside the unsloth wheel -- with `pip install -U --no-deps unsloth unsloth_zoo`, then restarts just the studio service via supervisor. The torch/CUDA stack is left untouched, so it is safe in both GPU and CPU-only containers. This deliberately avoids `unsloth studio update`, which re-runs the full installer and re-probes the GPU to pick torch wheels -- in a container started without --gpus that finds no GPU and can downgrade torch to CPU/cu126. Options: --ref <branch|tag|sha> installs from git (track main) instead of the latest PyPI release; --with-deps also updates dependencies; --no-restart defers the restart. After the swap the helper smoke-imports studio.backend.main and, if a transitive dep is now missing, points the user at --with-deps. The update lands in the container's writable layer (survives docker restart); mount -v unsloth_studio_home:/opt/unsloth-studio to keep it across a recreate. * docker: add unsloth-llama-update for in-place llama.cpp prebuilt updates Parity with unsloth-studio-update: update the baked llama.cpp prebuilt in a running container without pulling a new image. docker exec <container> unsloth-llama-update # latest release docker exec <container> unsloth-llama-update --check # report only It reuses the build-time fetcher (fetch_llama_prebuilt.py, now baked at /usr/local/lib/unsloth) rather than the host-probing installer behind the in-app banner. The fetcher resolves the latest release via the GitHub /releases/latest redirect (no API token, not rate-limited) and installs the portable CUDA bundle that runs on CPU and every supported GPU, so it works the same in a CPU-only or a --gpus container. The installer path, by contrast, scans the GitHub API (rate-limited to 403 in practice) and probes the host GPU, which falls back to a slow source build in a container started without --gpus. The fetch lands in a sibling temp dir on the same filesystem and is swapped in with an atomic rename; on any failure the existing install is left untouched. The Studio ownership marker is preserved across the swap. Verified end to end in a CPU-only container: b9596-mix-e6f2453 -> b9773-mix-1f1aaa4. * docker: optional Cloudflare tunnel for JupyterLab (UNSLOTH_JUPYTER_CLOUDFLARE) Mirror the public-link convenience Studio already has for its own UI, for JupyterLab. Off by default; opt in two ways: docker run -e UNSLOTH_JUPYTER_CLOUDFLARE=1 ... unsloth/unsloth docker exec <container> unsloth-jupyter-tunnel --force unsloth-jupyter-tunnel waits for JupyterLab, reuses a cached cloudflared (or fetches the static binary for the arch, no account needed), and starts a quick tunnel to the Jupyter port; the https://<name>.trycloudflare.com URL is printed to docker logs. supervisord runs it as the jupyter-cloudflare program, autostarted only when UNSLOTH_JUPYTER_CLOUDFLARE=1 (studio_launch.sh exports a 0 default so the autostart gate expands, matching the sshd pattern). JupyterLab still enforces its password, so the tunnel is not an open door. Verified: the helper fetches cloudflared and mints a working trycloudflare URL that reaches JupyterLab (HTTP 200) inside a running container. * docker-publish: clean build-args + add least-privilege default permissions - Move the explanatory prose out of the two `build-args:` blocks. docker/build-push-action forwards every non-empty line verbatim, so a leading-# line is passed as a bogus --build-arg; the comments now live above each block. This workflow has not run yet, so the issue was latent. - Add a top-level `permissions: contents: read` default so every job (including smoke-test, which had none) limits the GITHUB_TOKEN. The merge jobs keep their own `packages: write` blocks. Addresses the CodeQL "workflow does not contain permissions" findings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: fix notebook pip-shim drops, first-boot overwrite, base :latest tag, arm64 decord - pip shim: count editable/local/url/vcs targets (-e ., ., git+https, wheel URLs) as install targets, not just canonical package names, so they are no longer silently skipped inside notebooks - notebook sync: never overwrite a pre-existing user notebook on first boot (match the refresh path's ownership rule); skip .unsloth_sync_state.tmp when recording state so it is not tracked as a managed file - docker-publish: set flavor latest=false on the base image metadata so a v* tag push cannot publish :latest from the base image (the Studio image owns it) - notebook deps: pin to tested versions and install decord on its own, hard on amd64 and fail-soft on arm64 (no aarch64 wheel) so the arm64 base build works * docker-publish: pin one llama.cpp prebuilt release across both arch legs The base build-args never passed LLAMA_PREBUILT_TAG, so the Dockerfile fell back to latest and each matrix leg resolved whatever unslothai/llama.cpp release was current at its own build time. If latest moved between the amd64 and arm64 legs, one published manifest could carry different GGUF binaries per arch. Resolve the release once in a new prepare job (explicit llama_prebuilt_tag dispatch input for a frozen build, else follow the /releases/latest redirect to a concrete tag, mirroring docker/build.sh) and pass that single tag to both legs. * docker: address review follow-ups (pip-shim flags, sync ownership, tags, zoo ref, arch list) - pip shim: do not treat the value of an index-url / find-links / constraint flag as an install target. A cell like 'pip install --extra-index-url <url> torch' now no-ops after keeping the baked stack instead of exec'ing a bare 'pip install --extra-index-url <url>' that fails. Positional . / url / vcs and -r/--requirement files still count as targets. - notebook sync: on first boot, record only files we actually wrote (or that are byte-identical to the template), never a kept pre-existing user file; and on the GitHub refresh, treat a file present in DEST but absent from the sync state as user-owned and keep it. Previously a bind-mounted notebook was recorded as managed and then overwritten by upstream. - docker-publish: add flavor latest=false to the Studio metadata steps too, so a v* tag push cannot emit an implicit :latest via metadata-action's latest=auto; :latest stays default-branch-only, and the smoke test pulls the published tag. - unsloth-studio-update: resolve the unsloth-zoo ref independently of --ref (new --zoo-ref, else use the ref only when the zoo repo has it, else fall back to main) so 'update --ref <unsloth-tag/sha>' does not fail on a missing zoo ref. - Dockerfile: drop 10.3 (compute_103) from TORCH_CUDA_ARCH_LIST in both the builder and runtime stages. B300 runs sm_100 SASS, and the bundled CUDA 12.8 nvcc cannot compile compute_103 (added in 12.9), which broke arch-list-honoring source / JIT builds. * docker: address review round 3 (notebook -r filter, studio zoo ref, pinned notebooks commit) - unsloth_pip_shim.py: filter protected packages out of a notebook `pip install -r requirements.txt`. The -r value was passed to the real pip unchanged, so torch / transformers / vLLM / nvidia pins inside the file could overwrite the baked cu128 stack or push transformers into the base venv. _filter_requirements_file() applies the same _KEEP / transformers-sidecar rules per line, writes the survivors to a temp file, keeps comments, option lines, nested includes and urls verbatim, and records a pinned transformers version for the sidecar. - install.sh + Dockerfile.studio + docker-publish.yml: forward the resolved unsloth-zoo ref into the Studio build. install.sh --local overlaid unsloth-zoo from git main regardless of the operator-requested or base-image ref, so the full image could run a different zoo than the base. install.sh now honors UNSLOTH_ZOO_REF across all four --local overlays, Dockerfile.studio passes UNSLOTH_STUDIO_ZOO_REF through to it, and the workflow resolves one zoo ref in the prepare job and shares it with both the base and Studio builds. - Dockerfile + docker-publish.yml: pin unslothai/notebooks to one resolved commit. Each arch leg cloned HEAD independently, so the same tag could seed different baked templates and .unsloth_template_commit depending on the pulled platform. The prepare job freezes notebooks to one sha (like the llama.cpp prebuilt tag) and the Dockerfile fetches that single ref at depth 1. * docker: pip-shim catches direct-reference protected installs + --opt=value req files Two more notebook-shim gaps from review: - A quoted PEP 508 direct reference for a protected package, e.g. `pip install "torch @ https://.../torch.whl"` or `"unsloth @ git+https://..."`, bypassed _KEEP: _canon hit the url guard and returned None before pulling the distribution name, so the token was treated as a real target and reinstalled into the base venv. _canon now extracts the name from the `name [extras] @ url` form first, so a protected package pinned through a URL/VCS is still dropped; a non-protected direct reference returns its name and is kept exactly as before. - The `--requirement=reqs.txt` equals-form (pip accepts `--option=value` for any value-taking flag) was not recognized: the token starts with `-`, so it was kept as an opaque option, the file was never filtered, and has_target stayed false -- a cell whose only target was that file silently no-op'd. The scan now splits `--flag=value`, filters the requirements file for `-r`/`--requirement`, and counts it as a target; other inline-value options stay options. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: address review round 4 (jupyter probe, CPU messaging, llama EXDEV, %pip shim) - docker-publish smoke + docker_confirm.sh probe Jupyter /login, not /api: the launcher always configures a password hash so /api returns 403 and curl -f would never flip the health flag (false build failure). - entrypoint.sh CPU messaging: CPU mode covers Jupyter, GGUF tooling and llama.cpp (GGUF) Studio chat; training AND loading an Unsloth model (FastLanguageModel) still need a GPU, since from_pretrained runs CUDA probes. - install_llama_prebuilt.py: rollback/activation moves used bare os.replace, which fails with EXDEV across overlayfs in a Docker build and fell back to a broken source build (no nvcc). Add is_cross_device_error + move_install_dir_aside (os.replace fast path, copy+remove on EXDEV; busy errors still re-raise). - notebooks: %pip / %uv line magics and the `!python -m pip` form bypassed the PATH pip/uv shim and could overwrite the baked cu128 torch/vLLM stack. Add unsloth_nb_pip_magic.py to re-point them at the shim, wired via the IPython startup hook and installed into the venv site-packages. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker_confirm.ps1: probe JupyterLab /login, not /api A Jupyter password hash is always configured, so /api returns 403; the Windows confirmation reported a healthy full image as a hard failure. Matches the fix already in docker_confirm.sh and docker-publish.yml. * Docker notebook safety hardening and vLLM startup timeout fix pip shim (docker/unsloth_pip_shim.py): - Drop protected packages named via a VCS/URL #egg=NAME fragment so git+... #egg=torch no longer reinstalls into the baked venv. - Filter constraint files (-c/--constraint) through the same protected package filter as requirement files, so a pinned torch/transformers in a constraint cannot downgrade the baked stack during resolution. - Recursively filter nested -r/-c includes and absolutise their paths so the filtered /tmp copy still resolves them and no protected spec deep in the include tree slips past the keep list. - Remove an unused subprocess import. Notebook environment: - Scope the transformers-request marker per kernel (UNSLOTH_NB_TF_MARKER keyed on the kernel connection-file id) so concurrent notebooks no longer read each other's pin. - Install the IPython startup hook under IPYTHONDIR (set via ENV) so it loads for any uid, including docker run --user, not just root. - unsloth_nb_content_sig.py: only treat a %%capture / %%bash cell as install boilerplate when it carries an install command, so substantive captured/bash cells are hashed and upstream changes are not skipped. - unsloth_run.py: clean up the temp dir used to materialise a downloaded notebook. - unsloth_sync_notebooks.sh: honor UNSLOTH_KEEP_DELETED_NOTEBOOKS across GitHub refreshes so a deleted notebook is not restored when upstream advances. install_python_stack.py: the --local unsloth-zoo overlay now honors UNSLOTH_ZOO_REF (default main), matching the install.sh overlay. synthetic.py: preserve the timeout=None unbounded vLLM startup wait instead of coercing it to 1200s. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker-publish: freeze the unsloth-zoo ref to a concrete sha before fan-out The zoo_ref prepare step emitted the bare branch name (main) on the normal push/schedule path, and both arch matrix legs plus the Studio build pass that to pip install unsloth-zoo @ git+...@REF. If unsloth-zoo advanced mid-build a single multi-arch tag could bake different zoo code across architectures or between the base and Studio venvs. Resolve a branch/tag to its current sha via ls-remote here (mirroring the notebooks step), so the whole matrix pins one immutable commit. A 40-char sha input stays frozen; a lookup miss falls back to the bare ref so the build can still fetch by name. * docker/run.sh: forward Studio service env to the container The bundled launcher only forwarded HF/W&B/license/CPU vars, so the documented Studio service config read by studio_launch.sh was silently dropped when running the full image through this wrapper: JUPYTER_PASSWORD fell back to a random password, PUBLIC_KEY/SSH_KEY never enabled sshd, and UNSLOTH_JUPYTER_CLOUDFLARE never started the tunnel. Forward them with the same dash-only -e VAR form as the secrets above, so the value is read from the parent env and never lands in argv. * docker: move the image torch stack to 2.11.0 and document the amd64 sm_103 JIT limit Bump the base image torch triplet to torch==2.11.0 / torchvision==0.26.0 / torchaudio==2.11.0 and the paired torchcodec to 0.11.0, and hold torch at 2.11.0 during the vLLM resolve so uv lands on the vLLM 0.20+ line that pins torch 2.11.0 (the split-install rationale already anticipated the bump). Update the build-time self-test assertion, its status line, and the test_locally.sh log grep to match, plus the FA2 wheel note. Also clarify the advertised architecture support: forward-compatible SASS covers precompiled kernels on sm_103 (B300/GB300), but runtime Triton/NVRTC JIT targets the actual device cap and the bundled cu12.8 ptxas/NVRTC cannot emit compute_103. arm64 sm_121 is handled by the cu13 NVRTC/ptxas override; amd64 sm_103 has no cu13 override yet, so JIT-heavy paths there can fail until it lands. Precompiled SASS still runs on sm_103 via sm_100 forward-compat. * Normalize kwarg spacing in loader.py after the main merge Post-merge ruff-format-with-kwargs pass (the pre-commit.ci hook) on the merged loader.py; whitespace only, no logic change. * docker: Colab-grade JupyterLab and Studio UX for the Unsloth image (#6681) * docker: Colab-grade JupyterLab and Studio UX for the Blackwell image Stacks a Colab-like JupyterLab and Studio experience on top of the existing Blackwell image. Additive only: the training stack, CUDA/torch pinning, and the Studio/JupyterLab/sshd service trio are unchanged. JupyterLab labextension (prebuilt in a throwaway builder stage, so the runtime image stays Node-free): - Unsloth Dark (Monokai) theme, adaptive light/dark by system preference - Colab-style ArrowDown/Up cell navigation - top-bar Unsloth logo (stock Jupyter logo disabled and locked) - #@title lines render as collapsible Heading-2 form bars - Ctrl+A in a cell output selects only that output, not the whole notebook (the old behaviour ran notebook:select-all and was laggy) - right activity bar hidden by default - overrides.json: per-cell run button without auto-advance, labeled Restart and Run All, windowing off so collapsing an output does not snap to the cell top, news/update prompts suppressed Studio and login branding: Unsloth favicon, page logo, and a dark Unsloth login page that rotates through the curated Studio sloth stickers (fail-soft to the logo). Notebook organization and Colab compatibility (base image): - categorized folder view built from relative symlinks mirroring the README sections, rebuilt each boot; real .ipynb files never moved, and the symlink tree is invisible to the sync state machine - AMD-* notebooks shown only on an AMD/HIP host (autodetected) - Docker-only strip of the Colab "Run all on Colab" intro sentence from unedited notebooks (upstream notebooks unchanged) - hoist %%capture above a leading #@title form so the cell runs - the per-cell transformers-sidecar log is silent unless UNSLOTH_ENABLE_LOGGING=1 Dependency pinning and naming: the curated notebook extras are pinned to their resolved versions for reproducible rebuilds; decord is split into its own fail-soft install (no aarch64 wheel). The lean base image is renamed from :base to :core. Adds tests/validate_studio_features.py, a static self-test for the labextension plugins, overrides keys, and branding wiring. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: address review feedback on the JupyterLab/Studio UX - unsloth_nb_view.py: rebuilding the categorized view no longer deletes user files. The view is also JupyterLab's landing dir, so a user may save real notebooks there; _clear_view now unlinks only the symlinks we own and removes only folders that end up empty, leaving regular files in place. It also tests islink before isdir, so a view that is itself a symlink to a directory is unlinked instead of being walked into (which would have wiped the symlink target). - studio_launch.sh: derive the landing URL and preferred_dir from UNSLOTH_NOTEBOOKS_VIEW_DIR / UNSLOTH_SKIP_NOTEBOOK_VIEW, the same env the sync script uses, instead of hard-coding /workspace/Unsloth Notebooks. A relocated or disabled view no longer opens JupyterLab on a missing folder; it falls back to the default /lab over /workspace. - Dockerfile.studio: the labext-builder stage now installs Node 20 from NodeSource. Ubuntu 24.04's distro nodejs is 18, below JupyterLab 4.6's declared Node >=20 engine. Node stays confined to the throwaway builder stage, so the runtime image is unchanged. - .dockerignore: explicitly allowlist jupyter/install_sloth_stickers.py alongside its sibling jupyter assets, rather than relying on the directory re-inclusion. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: publish lean image as :core and full image as :studio Complete the base->core (and "studio as studio") tag rename so the publish workflow matches the user-facing helpers and the Dockerfile.studio header. - The lean training image now publishes as :core (core-<tag>, core-nightly, core-sha-*); run.sh / docker_confirm.* already told users to pull :core, but docker-publish.yml still tagged it :base, so that pull would have 404'd. The per-arch digest artifacts are renamed to match. - The full Studio image keeps :latest and gains a stable :studio alias, matching the Dockerfile.studio header. Both the merge and post-publish smoke-test metadata blocks are updated together. Internal "base image" wording (the layer Studio builds FROM) is left as-is. * docker: address second-round review feedback on the JupyterLab/Studio UX - studio_launch.sh: also gate the categorized-view landing URL on UNSLOTH_SKIP_NOTEBOOK_SYNC (the entrypoint skips building the view entirely in that mode), not just UNSLOTH_SKIP_NOTEBOOK_VIEW, so a no-sync container does not land on a missing folder. - Dockerfile.studio: scope the sticker-install "|| echo" fallback to only the sticker step via a { ...; } group. It was attached to the whole branding && chain, so a failure in a REQUIRED step (JS resolve, favicon/logo/login copy) was swallowed and the build continued with broken branding. - unsloth_nb_view.py: when creating the categorized symlinks, only replace our own stale symlinks; if a real user file already occupies that name, keep it and skip the link instead of os.remove-ing it. - overrides.json: drop doNotDisturbMode (it silenced ALL JupyterLab toasts, including kernel-restart / connection-drop feedback). The news/update prompts are already off via fetchNews / checkForUpdates. - Dockerfile: keep decord mandatory on amd64 (fail the build on a missing or incompatible wheel) and only fail-soft on arm64/other arches that have no wheel. - cellNav.ts: do not hijack ArrowUp/Down when focus is in an interactive output widget / form control, or while a completion popup is open, so ipywidgets controls and autocomplete at cell boundaries keep working. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: keep Studio branding RUN free of comments inside the line continuation Move the sloth-sticker fail-soft explanation above the RUN so no comment line sits between backslash-continued commands. BuildKit strips such comments, but keeping the RUN body a plain && chain removes the ambiguity for non-BuildKit builders and static linters. The { ...; } fail-soft scoping is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: AGPLv3 attribution + integrity guard for the Studio/JupyterLab image Make it obvious the image is built by Unsloth and hard to white-label out with a shallow find-and-replace, and surface the AGPLv3 license + copyright in the UI. Visible attribution (labextension): - Help > "About Unsloth Docker Studio" dialog (about.ts): Unsloth logo, the AGPLv3 notice, "Copyright 2026-Present the Unsloth team", and source/website/ license links. Added to the Help menu and the command palette. - The JupyterLab loading splash is replaced with a spinning Unsloth logo (splash.ts, provides ISplashScreen; honors prefers-reduced-motion). The stock @jupyterlab/apputils-extension:splash is disabled+locked at build time, like the stock logo. - AGPLv3 footer (license + copyright + links) on the branded login page. - Labextension relicensed AGPL-3.0-only; SPDX headers on every source file. Anti-tamper (no encoded/obfuscated strings -- plain readable text only; the one data URI is the logo image): - A canonical, plain-text attribution set lives in unsloth_branding.py with a TypeScript mirror (branding.ts) bundled verbatim into the labextension, so the phrase, copyright, links and plugin ids are spread across independent layers. - unsloth_branding.py verifies all of these across the installed files (AGPLv3 text, login footer, theme, labextension package + built bundle strings, logo, favicon) and fails loudly if any are missing. It runs at three layers: build time (fails the image build), the whole-container launcher (studio_launch.sh refuses to start), and as a jupyter_server extension (refuses to serve JupyterLab). - tests/studio/test_branding_guard.py: positive + per-marker negative coverage, plus a check that no base64/decoder obfuscation crept into the attribution. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: address #6681 review round 2 (colab magics, output select, branding guard) - unsloth_colab_compat.py: only hoist a leading `%%` cell magic above the Colab `#@title` form for magics whose body runs as code (capture/time/bash/python/ ...). Content magics (%%writefile, %%html, %%latex, ...) are left untouched so the form comment is never injected into the written file / rendered output. - outputSelect.ts: stop trusting the text selection anchor to decide ownership of Ctrl/Cmd+A. A stale selection inside an output survives a click onto a command-mode cell or the file browser, which made select-all keep re-selecting the old output. Gate on the keystroke target or the last pointer-down (reset to null on any click outside an output) instead. - unsloth_branding.py: also reject page_config.json that disables the Unsloth labextension or any of its plugin ids via disabledExtensions (dict or list form); that leaves the bundle on disk so the prior checks passed while the logo/About/splash attribution was stripped at load. Lock unsloth-jupyterlab in Dockerfile.studio as well (defense in depth), and add guard tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * labext: pin JupyterLab extension deps; confirm.ps1 /login probe Pin the unsloth-jupyterlab npm deps to exact versions matching the baked jupyterlab==4.6.0 (builder stays 4.5.9, its newest release) instead of floating ^/~ ranges, so the same commit always builds the same labextension bundle. Also probe JupyterLab /login (not /api, which 403s behind a password hash) in the Windows confirmation script. * docker: categorize AMD/domain notebooks and wire the feature validation into CI unsloth_nb_view.parse_readme only reset the folder section on level-3 (###) headings. The notebooks README carries level-1 domain headers (# AMD Notebooks, # Kaggle Notebooks) with their own nb/*.ipynb link tables and no intervening ###, so those notebooks were mis-filed under the previous stale section (all 148 AMD notebooks landed in Other Notebooks on an --amd build). Reset on any heading level and strip a leading emoji/symbol run so the domain notebooks get their own clean folder. Also run tests/validate_studio_features.py explicitly in the repo CPU job. It is named validate_* (not test_*) so pytest never collected it, which meant a regression in the notebook view, Colab compat, strip, JupyterLab defaults or login branding failed CI only when run by hand. * labext: use caret ranges so jlpm dedups JupyterLab/Lumino singletons The exact pins introduced earlier (@jupyterlab/* 4.6.0, @lumino/widgets 2.8.0, @jupyterlab/builder 4.5.9) break the Dockerfile.studio labext-builder stage. Exact-pinning the framework packages defeats jlpm's (yarn classic) hoisting: transitive @jupyterlab deps request caret ranges that resolve to newer patch releases (e.g. @jupyterlab/ notebook pulls @jupyterlab/cells ^4.6.0 -> a newer patch), so jlpm installs a second nested copy alongside the exact top-level one. Two copies of @jupyterlab/cells and @lumino/widgets in the tree produce TS2345 "not assignable" errors (protected-member/identity mismatch) and the build fails. Caret ranges let jlpm collapse every @jupyterlab and @lumino package to a single hoisted copy, which is required for a JupyterLab prebuilt (federated) extension: at runtime those packages are shared singletons provided by the host JupyterLab, so the build-time versions only need to type-check against one consistent tree, not match an exact runtime patch. This is the version set the published image was built and validated with end to end. Verified by building the labext in isolation against the base image (Node 20 + bundled jlpm): caret ranges build clean (webpack compiled successfully); the exact pins fail with the duplicate-package TS errors. * ci(studio-backend): trigger on docker/** so the JupyterLab feature validation guards docker-only changes The 'Docker JupyterLab/notebook feature validation' step runs tests/validate_studio_features.py, which checks docker/jupyter (the labextension, overrides.json, login branding) and the docker notebook helpers. The pull_request paths filter listed studio/unsloth/tests but not docker/**, so a PR that only touches docker/ would skip that step and a regression in those files could pass CI. Add docker/** so the validation runs whenever the files it checks change. * jupyter: center the login card and place the attribution below it #site was a flex container using the default row direction with two children (the login card and the AGPLv3 attribution), so they rendered side by side: the card sat left of centre and the attribution floated up to the top-right. Stack them in a column so the card is horizontally centred and the attribution sits below it as a footer, matching the intended single-column layout. * jupyter: refresh Studio attribution, About dialog and loading splash - Attribution now reads 'Built by the Unsloth team' with a single Apache 2.0 / AGPLv3 license link (to the repo license section) on the login page and in the About dialog, replacing the plain 'Built by Unsloth. Licensed under the GNU AGPLv3.' line. The integrity guard, its canonical PHRASE and the branding tests are updated to match. - About dialog: left-align the link rows so the labels line up instead of each row centering independently; add an 'Unsloth Reference' link to the docs, and a Licenses section listing Unsloth Studio (AGPLv3) and Unsloth Core (Apache 2.0) alongside the full license link. - Loading splash now reads 'Loading Unsloth Docker' instead of the attribution label, via a dedicated SPLASH_LABEL constant. * docker: document the branding attribution as an AGPLv3 Section 7 notice Add docker/NOTICE and docker/jupyter/BRANDING.md so the Unsloth attribution that unsloth_branding.py enforces is also a written license condition, not only a build check. docker/NOTICE designates the attribution (the "Built by the Unsloth team" label, the copyright line, the license notice, the logo and theme, and the Help > About links) as required Appropriate Legal Notices under AGPLv3 Section 7(b), referencing /studio/LICENSE.AGPL-3.0 and /LICENSE. BRANDING.md is a human-readable note next to the guard describing what must stay, where it lives and how it is enforced. * ci(studio-backend): restore docker/** trigger path The docker/** pull_request path added in b558bc7d was dropped by a later rebase, so the "Docker JupyterLab/notebook feature validation" step (which runs tests/validate_studio_features.py against docker/jupyter branding and notebook helpers) no longer ran on PRs that only touch docker/. Re-add docker/** so a docker-only change is validated on the PR rather than only after merge to main. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * docker: finish the torch 2.11.0 move and extend the cu13 JIT override to amd64 Base image (torch 2.11.0): - amd64 unsloth extra: cu128-ampere-torch2100 -> cu128-ampere-torch2110. The old extra pulls xformers 0.0.34, which hard-pins torch==2.10.0 and conflicts with the torch==2.11.0 held throughout the build; the torch2110 family pulls xformers 0.0.35 (no torch pin) and resolves cleanly. This needs an unsloth carrying the torch2110 CUDA extras on main, so merge the torch2110 extras PR first (default UNSLOTH_REF=main). - notebook-deps assertion: startswith('2.10.0') -> '2.11.0' so the layer actually verifies the torch it now installs. - refresh the torch2100/xformers 0.0.34 references in the surrounding comments to the torch2110/0.0.35 line. sm_103 (B300/GB300) JIT override (Codex item): The cu13 NVRTC/ptxas override was arm64-only (sm_121), and its comment claimed triton 3.6.0 bundles cu13 ptxas and set TRITON_PTXAS_PATH -- neither was true: triton 3.6.0's bundled ptxas is CUDA 12.8 (V12.8.93, tops out at sm_120) and TRITON_PTXAS_PATH was never set. So sm_103 (amd64) and even sm_121 (arm64) Triton JIT were unfixed. Run the cu13 install on both arches and actually wire the ptxas override: - NVRTC swap (cu13 libnvrtc.so.13 over torch's bundled cu12.8 .so.12) now runs on amd64 too. - ENV TRITON_PTXAS_PATH=/usr/local/cuda-13.0/bin/ptxas routes every Triton JIT through the cu13 ptxas. Global rather than per-arch is safe: cu13.0 ptxas spans sm_70..sm_121 (verified: Volta/Turing/Ampere/Hopper through Blackwell), so no regression for the older GPUs in the arch list. Verified on amd64 in the built base image: cuda-nvrtc-13-0/cuda-nvcc-13-0 install cleanly from the base's CUDA repo, ptxas lands at /usr/local/cuda-13.0/bin/ptxas (V13.0.88) and libnvrtc.so.13 at /usr/local/cuda-13.0/lib64/. The sm_103/sm_121 runtime path itself is not hardware-tested (no such GPU on hand); precompiled SASS still covers both via sm_100/sm_120 forward-compat, so only JIT-heavy paths rely on this. * docker/studio: assert the Studio venv torch exactly matches the base before CUDA dedup The Studio build symlinks the Studio venv's CUDA libs onto the base venv's copies to reclaim ~3.7GB. That is only safe when both venvs run the same torch, but the pre-dedup guard only checked the CUDA family (endswith('+cu128')). A Studio venv that installed torch 2.10.0+cu128 (an installer capped below the base's 2.11.0, or a build-time nvidia-smi fallback) would pass that check yet mismatch the base's 2.11.0+cu128, and the dedup would link incompatible libs. Capture the base venv's torch from its metadata and assert the Studio venv torch equals it exactly (version and family) before the dedup runs, so a mismatch fails the build loudly instead of silently linking skewed CUDA libs. Comparing to the base venv also avoids hardcoding the version here. The Studio venv reaches torch 2.11.0+cu128 via the installer's UNSLOTH_TORCH_INDEX_FAMILY=cu128 handling and its CUDA torch spec allowing 2.11.x. * docker: address review round 2 (CI ref freeze, Studio NVRTC amd64, pip-shim edges) docker-publish.yml: freeze the requested unsloth ref to one sha in the prepare job before the matrix fans out. UNSLOTH_REF / UNSLOTH_STUDIO_REF were raw expressions re-evaluated per base arch leg and in the Studio build, so a mutable branch (the workflow_dispatch default unsloth_ref=main) advancing during the run could bake different unsloth commits under one manifest. Resolve once (same precedence: dispatch input, else pushed tag, else triggering sha, else main; ls-remote a branch/tag to a sha, mirroring the zoo/notebooks steps) and read needs.prepare.outputs.unsloth_ref everywhere. Dockerfile.studio: run the Studio venv NVRTC cu13 swap on both arches, not arm64 only. amd64 sm_103 (B300/GB300) needs cu13 NVRTC just as arm64 sm_121 does, and the CUDA dedup never touches cuda_nvrtc, so an amd64 Studio venv would otherwise keep its bundled cu12.8 libnvrtc and fail NVRTC/jiterator JIT on compute_103. The base cu13 layer installs cuda-nvrtc-13-0 on both arches, so the target .so.13 exists here regardless of TARGETARCH. unsloth_pip_shim.py: close three ways a protected package slipped past _KEEP. Treat -e/--editable as a value-taking flag paired with its target and drop both when the target is protected (was leaving a dangling -e that failed the cell); filter -P/--upgrade-package values through _KEEP (a named baked package could be refreshed while installing another target); and parse the PEP 427 distribution name out of a wheel URL/path so a bare `pip install https://.../torch-...whl` drops instead of reinstalling the baked torch. Non-protected editables, upgrade selectors, and wheels are unchanged. Adds tests/python/test_unsloth_pip_shim.py (18 regression tests, exec captured via a patched os.execv). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: address review round 3 (requirement-file shim edges + device-gate cu13 JIT tools) unsloth_pip_shim.py: close three more ways a protected package slipped past _KEEP. An editable line (-e/--editable <target>) inside a -r requirements file is a real install target, so a protected editable there is now classified and dropped like the command-line case (new _parse_editable). pip/uv accept the attached short forms -rreqs.txt / -cconstraints.txt / -epath / -Pname as one token; these were falling through as opaque options (so an attached -r-only cell no-op'd and an attached -c/-e/-P value bypassed _KEEP), so the 2-char flag is now split from its value and routed through the separated-form handling. And a nested -c constraint inside a -r file no longer records its transformers pin as an install request (a constraint is not a request; mirrors the top-level -c path). entrypoint.sh / Dockerfile: gate the CUDA 13 ptxas + NVRTC to sm_103 / sm_121 at runtime instead of a global build-time default. A cu13 cubin needs a >= 580 driver to LOAD even when it targets an older arch (CUDA has forward, not backward, cross-major driver compatibility), but the image supports Turing.. sm_120 on a 570+ driver, so the previous global TRITON_PTXAS_PATH ENV + cu13 NVRTC symlink would break ordinary Triton/NVRTC JIT on 570-579 driver hosts. The build still bakes cu13 (saving the cu12.8 NVRTC as .cu128.orig); a new select_cuda_jit_tools() in the entrypoint reads the device compute_cap and only activates cu13 for sm_103/sm_121 (which ship >= 580 drivers), otherwise leaving Triton on its bundled cu12.8 ptxas and restoring the cu12.8 NVRTC in both the base and Studio venvs. The base ENTRYPOINT runs for the Studio image too. Adds 9 pip-shim regression tests and tests/sh/test_select_cuda_jit_tools.sh (7 device-gating cases); registers the latter in CI and tests/run_all.sh. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: close notebook pip-shim bypasses and scan all GPUs for cu13 Notebook pip/uv shim (docker/unsloth_pip_shim.py), all active only under UNSLOTH_NB_SHIM=1: - Parse a bare wheel filename (torch-*.whl in the CWD, no ./ or / prefix) so it is matched against _KEEP instead of passing through as an opaque positional and reinstalling the baked torch. - Infer the distribution from an egg-less VCS URL by repo basename (git+https://github.com/huggingface/transformers.git -> transformers) so the egg-less form the repo itself recommends cannot clobber the baked stack. - Refuse remote (URL) -r/-c requirement/constraint files -- top-level and nested includes -- since their pins cannot be inspected before the real tool would fetch and install them. - Strip resolver-wide reinstall/ignore-installed switches (--force-reinstall, --ignore-installed, -I, uv --reinstall) so they cannot rebuild already-satisfied baked deps pulled in by a kept target. - Route uv --reinstall-package through the same _KEEP handling as -P/--upgrade-package (both attached and separated forms; no dangling flag). Entrypoint (docker/entrypoint.sh): select_cuda_jit_tools() now scans every visible GPU's compute_cap instead of only the first, so a datacenter Blackwell (sm_103/sm_121) behind an H100/B200 still enables the cu13 JIT tools it needs. Adds regression tests for each case (tests/python/test_unsloth_pip_shim.py, tests/sh/test_select_cuda_jit_tools.sh). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: close more pip-shim bypasses and make cu12.8 NVRTC the default Notebook pip/uv shim (docker/unsloth_pip_shim.py, active only under UNSLOTH_NB_SHIM=1): - Parse protected source archives (sdist/zip) by basename too, e.g. `pip install https://.../unsloth-2026.7.1.tar.gz` or `./torch-2.11.0.tar.gz`, mirroring the wheel-basename handling. A first-hyphen-before-digit split keeps hyphenated names like flashinfer-python intact. - Recognise uv's PLURAL long flags --requirements / --constraints, so those files go through the same protected-package filter as the singular names. - Drop --upgrade-strategy eager in shim mode so a kept target cannot eagerly rebuild already-satisfied baked deps (falls back to pip's only-if-needed). NVRTC default (docker/Dockerfile, docker/Dockerfile.studio, docker/entrypoint.sh): - Make cu12.8 the immutable baked default (libnvrtc.so.12 -> .cu128.orig) with a staged .cu13 alias, and have select_cuda_jit_tools retarget to cu13 ONLY for sm_103/sm_121. Previously cu13 was baked as the default and restored to cu12.8 at runtime, so a non-root `docker run --user` container that cannot rewrite the symlink stayed on cu13 NVRTC and emitted cubins a 570-579 driver cannot load. The safe default now needs no runtime write. Adds regression tests for each case (tests/python/test_unsloth_pip_shim.py, tests/sh/test_select_cuda_jit_tools.sh). * docker: harden rollback, publish, shim, and view-cleanup paths Ten verified fixes from a 12-reviewer audit of the image tooling, each reproduced before fixing: 1. install_llama_prebuilt.py move_install_dir_aside: the EXDEV fallback copied straight into the rollback path, so a copy that died halfway (ENOSPC, I/O error) left a partial tree that activation recovery would later restore over the intact install while deleting the good copy. Copy to a temp sibling and publish with one atomic rename; dst.exists() is now a truthful complete-tree signal. 2. unsloth_run.py --out truncated the existing output before nbconvert ran, so a timeout, missing kernel, or failed cell irreversibly destroyed the previous result. The input copy and executed result are staged as temp files next to the destination and published with os.replace only on exit code 0. 3. unsloth_nb_view.py cleanup treated every symlink in the view as its own: user-created links (and an operator's view-root routing symlink) were deleted on every rebuild. Cleanup now removes only links that resolve into the notebooks tree it links from, and builds inside a view-root symlink's target instead of unlinking it. 4. unsloth_llama_update.sh: the unconditional EXIT trap deleted the .old backup even when it was the only remaining copy (signal between the two renames, or a failed swap whose restore also failed). The handler now restores the backup first when the install dir is missing and removes it only after the new tree is verifiably active; HUP/INT/TERM route through the same handler. 5. unsloth_pip_shim.py: transitive dependencies could replace the baked torch stack (reproduced with a wheel requiring torch==99.0). Every forwarded install now carries a constraints file pinning the installed protected set, turning the swap into ResolutionImpossible. 6. unsloth_pip_shim.py: ${UPPER} env references in requirements files were classified before pip expanded them, bypassing the protected-package filter; the shim now expands with pip's exact regex first. 7. unsloth_pip_shim.py: a failure writing the filtered requirements copy returned the ORIGINAL file, forwarding exactly the protected pins it had detected; it now fails closed. 8. docker-publish.yml: workflow_dispatch defaulted unsloth_ref to 'main' while the stable-tag gates require '', so UI-default manual runs could never advance :core/:latest/:studio; the default is now empty. 9. entrypoint.sh: the sm_103/sm_121 branch rewrote libnvrtc.so.12 to the CUDA-13 build but the ordinary-GPU branch never restored it, so a container moved to an older GPU kept the stale link; it is now reversed when it points exactly at the .cu13 target. Rejected after verification (no code change): timeout=0 semantics are documented at the site with no zero callers, TORCHINDUCTOR_COMPILE_THREADS override is deliberate, fetchNews is a string enum per JupyterLab's schema, :base tag appears in no in-tree doc, install-cell digest exclusion is the module's stated contract, transformers ceiling semantics are documented, and the cloudflared download mirrors the pre-existing Studio downloader (Cloudflare publishes no checksum asset). The UNSLOTH_ALLOW_CPU import crash lives in unsloth_zoo (compiler.py / loss_utils.py capability probes), not in this diff; the image consumes the zoo fix automatically once merged there. Tests: shim suite extended to 63 (constraints, env expansion, fail-closed), jit-selector suite to 14 (NVRTC reversal transitions), plus staged-publish and ownership repros; wider studio install suite green except failures reproduced at the unmodified head. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: close pip-shim bypasses and warn on arm64 cu13 llama.cpp mismatch Four follow-ups to the shim/entrypoint audit fixes: 1. unsloth_pip_shim.py let a local project directory install through: `pip install ./transformers` / `-e ./unsloth` is not a requirement spec, so _canon returned None and both the arg filter and the constraints file (which only rejects a version MISMATCH) passed it, letting a same-version local build silently replace the baked wheel. _canon now resolves the project name from pyproject [project].name, then setup.cfg, then the directory basename when it is an installable project, so a local checkout of a protected package is dropped like every other artifact form. Names match exactly after normalization, so a user dir named my-torch-utils is untouched, and a metadata-less directory still passes through. 2. unsloth_nb_pip_magic.py only rewrote literal `!python -m pip`, so the `!{sys.executable} -m pip ...` form notebooks use to target the running kernel (and absolute interpreter paths) bypassed the PATH shim entirely. Input transformers see the raw cell text before IPython expands the braces, so the matcher now also covers {sys.executable} (quoted or bare) and quoted/bare interpreter paths ending in python[0-9.]*(.exe) before -m pip|uv. 3. unsloth_pip_shim.py did not strip uv's --exact, which performs an exact sync that removes every installed package outside the kept target's closure (vLLM, bitsandbytes, the NVIDIA libs); `uv pip install --exact peft` would strip the baked stack after the filter kept it. --exact now joins the resolver-wide destructive flags dropped in shim mode. 4. entrypoint.sh: the arm64 image bakes a CUDA 13 llama.cpp because upstream (unslothai/llama.cpp) publishes no CUDA 12 arm64 asset, while the torch stack (cu128) runs on a 570-series driver. A CUDA 13 cubin cannot load on a 570-579 driver, so on GH200/GB200 hosts below 580 GGUF export and Studio chat fail while training works. The entrypoint now warns up front on aarch64 + driver < 580 instead of letting llama-server fail later. Tests: shim + nb-pip-magic suites at 81 (18 new, including local-project name resolution, the executable/brace forms, and --exact stripping). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: tighten comments * docker: add the AGPL-3.0 SPDX header to the new Python files Every new .py this PR adds now carries the same two-line SPDX header the other new files in the branch already use (docker/jupyter/unsloth_branding.py), with the shebang kept first where present. Matches the licensing laid out in docker/NOTICE: the image bundles Studio (AGPL-3.0) while Unsloth Core stays Apache-2.0. * docker: drop the local dev harnesses from the image PR Remove seven dev-only scripts that never reach the image or CI: the .dockerignore whitelist excludes them from the build context, docker-publish.yml runs smoke_test.py via buildx with native arm64 runners (no QEMU setup script), and nothing else references them beyond a few comments. test_locally.sh, docker_confirm.sh/.ps1, setup_qemu.sh, hf_pull.sh, hf_push.sh and freeze.sh can return in a follow-up dev-tooling PR; this PR stays the image itself. Cuts 1105 lines and 7 files from the diff. * docker: dedupe repeated rationale comments and parametrize the pip-shim tests Comment-only consolidation: the sm_103/sm_121 + cu13 JIT story and the xformers-aarch64 note were each told four times across docker/Dockerfile; keep the header telling canonical and cross-reference it elsewhere (same for the workflow's six retellings of the resolve-refs-once rationale and Dockerfile.studio's NVRTC block). Comments that pointed at the removed dev scripts now name the underlying command or artifact instead. Non-comment lines of both Dockerfiles and the workflow are byte-identical. unsloth_sync_notebooks.sh folds the three copies of the override -> PATH -> sibling helper resolution into one resolve_helper(), behavior verified for all four modes including graceful absence under set -u. unsloth_pip_shim.py collapses an if/else whose branches were identical and merges the structurally duplicate _parse_include/_parse_editable into one _parse_flag_line. The test suite folds 35 near-duplicate tests into 8 parametrized groups with exact case-count parity (69 collected before and after, 81 passing including the nb-pip-magic suite). Cuts another 144 lines with zero behavior change outside the two refactors. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: preflight every GPU, classify all uv/pip value flags, volume-safe llama update Preflight (entrypoint.sh) now scans every visible device: an unsupported device 0 stays fatal as before, an unsupported secondary GPU (mixed rig) warns at startup with its index and the CUDA_VISIBLE_DEVICES remedy, instead of surfacing only when a job pins to it or a multi-GPU launch fans out. The pip shim's _VALUE_FLAGS now covers every value-taking flag of uv pip install and pip install (generated from both tools' --help). The separated form `uv pip install --torch-backend cu128 torch` used to drop the protected torch but exec uv with no install target at all (uv hard-errors) instead of no-oping like the attached `=` form, and `--extra torch peft` misread the extra name as a protected target, leaving a dangling --extra that swallowed peft. Adds parametrized regressions plus help-derived drift guards so a future uv/pip value flag cannot silently reintroduce the misparse. unsloth-llama-update now detects when the install dir is itself a mount point (the documented -v unsloth_llama:/opt/unsloth/llama.cpp persistence recipe, where rename(2) fails EBUSY) and swaps the bundle CONTENTS inside the mounted tree, so the update lands in the volume and stays persistent. Work and backup dirs live under the mount (same-fs renames), the abort trap restores the old contents even mid-swap, and the non-mounted path keeps the whole-dir rename. Verified: in-place swap preserves the dir inode and ownership marker, failed fetch leaves the install untouched, simulated mid-swap abort restores fully. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: strip VCS refs before the basename, bake the value-flag drift check into the build A VCS @ref can itself contain a slash (@feature/foo), and the shim split the last path segment BEFORE dropping the ref, so git+https://github.com/unslothai/unsloth.git@feature/foo canonicalized as "foo" and a protected repo installed from a branch dodged _KEEP. The ref is now stripped from the path portion first (after the authority, so an SSH userinfo @ is never mistaken for the ref separator, matching pip's own last-@ parsing), with regressions for slash refs, SSH userinfo, plain tags and the no-ref form. The help-derived value-flag drift guards were version-sensitive: repo CI runs whatever pip/uv are current, so the next tool release turned unrelated PRs red (pip 26 added --all-releases/--only-final/--requirements-from-script/ --uploaded-prior-to, uv added --no-editable-package/--upgrade-group; all six now classified). The guards are opt-in for local runs (UNSLOTH_SHIM_FLAG_DRIFT_CHECK=1) and the authoritative check now runs at image build time via a new --unsloth-selfcheck-value-flags mode wired into the Dockerfile verify step, where the baked pip/uv are exactly the tools the shim fronts, so a flag added by a future baked-tool bump fails the build instead of a user's notebook cell. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: pin xformers explicitly, forward the llama tag to the Studio build The amd64 base install named the cu128-ampere-torch2110 extra, which does not exist on main yet (the CUDA extras stop at torch2100): pip/uv only warn on an unknown extra, so plain unsloth installed without xformers and the required- package check failed the build. Both arches now take the plain huggingface extra and amd64 pins xformers==0.0.35 explicitly in the same resolve (it requires torch>=2.10 without an exact pin, pairing with the baked 2.11.0; verified on PyPI, x86_64 wheels only, matching the arm64 exclusion). This decouples the base image from the pending extras PR. The Studio build now receives the SAME llama.cpp tag the base image baked: Dockerfile.studio grows a LLAMA_PREBUILT_TAG arg exported as UNSLOTH_LLAMA_TAG to install.sh (setup.sh honours it; the "latest" default is byte-identical to setup.sh's own default for local builds), and the publish workflow forwards the prepare job's resolved tag in the studio build-args. Without the pin a dispatch override or an upstream release landing between the two jobs let the no-GPU Studio build re-resolve "latest" and replace the pinned CUDA bundle. The Studio venv-match assertion also needs installer support for torch 2.11 on the CUDA path; that lands in a separate installer PR and is now declared as a merge-order dependency in the PR description (the publish workflow only runs on main pushes, so nothing builds before both are merged). * docker: gate stable tags on every overridable baked input The :core/:latest/:studio gates only checked unsloth_ref, so a default-branch dispatch overriding unsloth_zoo_ref, notebooks_ref, or llama_prebuilt_tag still published stable tags carrying non-standard bits; an earlier review round asked for this and only the unsloth_ref half landed. All six gate sites (merge + byte-identical smoke-test copies) now also require zoo and notebooks refs to be blank or their 'main' default and the llama tag to be blank. push/schedule events leave inputs null, which GitHub coerces to '', so automated publishes are unaffected; verified the full event matrix (push, default dispatch, each single override) against the expression semantics. * docker: pin /usr/local/cuda to 12.8 and quote native multi-device selectors Installing cuda-nvcc-13-0 for the sm_103/sm_121 JIT tools also flips the update-alternatives-managed /usr/local/cuda link to cuda-13.0: the package hard-depends on cuda-toolkit-13-0-config-common, whose postinst registers priority 130 over 12.8's 128 (reproduced in a clean nvidia/cuda:12.8.1-base-ubuntu24.04 container; --no-install-recommends does not help against hard Depends). TileLang JIT and torch.utils.cpp_extension resolve nvcc through /usr/local/cuda, so on the 570-driver hosts this image supports they would emit cu13 cubins that need driver 580 and fail at load. Pin the alternative back to 12.8 right after the cu13 install; the cu13 tools stay reachable by absolute path, which is exactly how the entrypoint activates them, and manual mode prevents future apt flips. run.sh accepted the native --gpus device=0,1 form through an unquoted passthrough, but docker requires the comma-carrying value to be quoted (daemon rejects it with 'cannot set both Count and DeviceIDs'; reproduced against a live daemon, and the docker GPU docs call the quoting out explicitly). A native multi-device selector is now wrapped in the same embedded quotes the other comma paths already use; single-device and pre-quoted forms pass through unchanged. All eight selector forms verified through the case block. * docker: tighten comments across the Blackwell image and helpers Condense the verbose explanatory comments added by this branch to their essential points without dropping any load-bearing rationale. Touches comments and docstrings only, no code changes. Leaves the stable-tag gate rationale, the byte-identical enable= sync notes, and the update-alternatives pin comment as is. * docker: per-run transformers marker, exact vLLM deadline, non-root workspace Three fixes from review: unsloth-run now gives each invocation its own UNSLOTH_NB_TF_MARKER (a temp file, cleaned up afterwards) unless the caller pinned one. The shared default marker leaked one run's transformers pin into later or concurrent runs in the same container: a notebook pinned to 4.57.6 left the marker behind and the next unpinned run's kernel activated the stale sidecar. An empty marker reads as no pin, so pre-creating the file is safe. The vLLM startup wait in dataprep/synthetic.py capped every poll at a full second regardless of the remaining budget, so a fractional timeout could overshoot by up to a second. The final wait is now clamped to the remaining time; verified empirically (timeout=1.1 elapses 1.10s). /workspace and the default HF/Triton cache dirs were root-owned, so docker run --user without a bind mount could not sync notebooks or populate caches. They are now world-writable (a+rwX), matching the documented non-root use the /opt prebuilt placement already supports. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: tighten comments * tests: track the moved pass-through inheritance in the gguf order check Main moved the llama_extra_args pass-through inheritance out of the GGUF branch into _resolve_inherited_extra_args, which runs before it, so the source-order assertion's "if request.llama_extra_args is None" anchor no longer exists inside the branch and the check failed after the main merge. The test now asserts the same property in the current shape: inheritance before the GGUF branch (a carried --no-mmproj still shapes the hub guard's companion requirement), and marker, hub guard, unload in order within the branch. Full file passes (32 tests). * tests: anchor the inheritance order check on the call, not the definition source.index("_resolve_inherited_extra_args(") matched the function definition, which always precedes the endpoint, so the ordering assertion was vacuously true. Anchoring on "= _resolve_inherited_ extra_args(" pins the first call site inside the load endpoint (line 4505), which is the statement whose position relative to the GGUF branch the test is meant to guard. 32 tests pass. * tests: align the gguf order test with main Main fixed the stale ordering assertion in PR 7252; adopting its version verbatim removes this file from the branch diff entirely and avoids a conflict on the next main merge. 32 tests pass. * docker: detach the notebook network refresh from container startup The GitHub refresh phase ran synchronously in the entrypoint's notebook sync, so an offline or slow network could hold container startup for up to two fetch timeouts (ls-remote + clone, about two minutes at the defaults) despite the sync being described as non-blocking. The local template populate and the categorized view still run in the foreground; the refresh now re-enters itself as a detached child (guarded by a flag so it forks once), whose phase-1 pass no-ops via the hash state and whose finalize is idempotent. Verified with an unreachable remote and an 8 second timeout: the parent returns in under a second with the notebooks populated while the child owns the waiting. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: give llama.cpp its libcublas so GGUF stops running on the CPU The portable llama.cpp bundle loads libggml-cuda.so with dlopen, links it against libcublas, and does not ship libcublas. The CUDA runtime base image only carries libcudart, and the only libcublas in the image is torch's wheel copy under site-packages/nvidia/cublas/lib, which was not on the loader path. So the CUDA backend failed to load, and llama.cpp said nothing about it: `--list-devices` printed an empty list and every GGUF request ran on the CPU. Measured in the built image on a B200 with gemma-4-E2B-it UD-Q4_K_XL: 1.6 tok/s from llama-cli and 4.2 tok/s from llama-server. With the fix, the same image and model report `CUDA0: NVIDIA B200` and run at 229 tok/s and 193 tok/s. Studio's GGUF chat and the GGUF export path go through the same bundle, so both were affected. The venv loader config already existed for torchcodec, so cublas/lib joins it there rather than on LD_LIBRARY_PATH: ld.so.conf.d is consulted after DT_RUNPATH, which keeps llama.cpp resolving its own $ORIGIN libs first. cu13/lib comes along for the arm64 bundle's layout. A silent 140x slowdown deserves a build-time gate, so the layer after the fetch runs ldd over libggml-cuda.so, installs the cublas major the bundle actually asks for when it is missing, and fails the build on anything still unresolved. The amd64 bundle wants libcublas.so.12 and torch already provides it; the arm64 bundle is CUDA 13, and deriving the major from ldd keeps that leg honest without hardcoding either. libcuda.so.1 is exempt: nvidia-container-toolkit injects the driver stub at `docker run --gpus`, so it is never resolvable at build time. ldd needs no GPU, so the build stays host-independent. tests/python/test_docker_llama_cuda_backend.py pins the loader entry, the guard, the driver-stub exemption and the ordering. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * vision: keep a caller's logits_to_keep on transformers 5 The v5 branch popped logits_to_keep and num_logits_to_keep unconditionally, so an explicit caller value was discarded before generate() ever saw it. v5 injects logits_to_keep=1 itself, but that injection is guarded by `"logits_to_keep" not in model_kwargs`, which makes it a default rather than an override: a value the caller passed is honored and must not be dropped. The cost of dropping it is not a no-op. Measured on a LoRA Qwen2-VL under transformers 5.14.1, with the model's forward hooked so the validator still sees the real signature: asking for logits_to_keep=0 (the whole sequence) reached forward as 1 and returned logits of shape (1, 1, 151936); with the value preserved it reached forward as 0 and returned (1, 88, 151936). Deleting the pops outright would be wrong in the other direction. An explicit num_logits_to_keep raises from _validate_model_kwargs on plain, PEFT, text and vision models alike, because v5 renamed it away, and logits_to_keep raises on the 12 of 79 image-text-to-text architectures whose top-level forward does not take it. So each key is now stripped only when _unsloth_generate_accepts_kwarg says this model would reject it, which is the same predicate the validator uses. Under PEFT, self inside the wrapper is the object the validator later runs against, so the check has no false negatives there. Also softened the comment above the branch. Unsloth 2026.7.5 does pre-inject on a LoRA Qwen2-VL under 5.14.1 and generation succeeds, so "pre-injecting makes the strict validator raise on PEFT models" overstates it. Skipping the injection on v5 is still right: it is redundant, and the arch walk can select a key the top-level model rejects. tests/test_generate_kwarg_gate.py gains four cases covering a preserved supported value, a stripped unsupported one, untouched neighbours, and the absence of the unconditional pop. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: fix the unsloth CLI and the vLLM engine in the image Two defects found by running the built image rather than reading it. 1. Every unsloth_cli subcommand that touches the studio backend died on import. `unsloth list-checkpoints` on the published image: ModuleNotFoundError: No module named 'structlog' and the same for train / export / chat, since all four import studio.backend.core.*. structlog is a studio backend requirement, not an unsloth[huggingface] one, so nothing in the base install pulled it in. Added it to the base venv, and added a build-time `from studio.backend.core.export import ExportBackend` so a future missing dependency in that closure fails the build instead of the user's first CLI invocation. That guard has to live in the LAST builder verification block: the closure also needs starlette, which only arrives with vLLM two stages later. 2. flashinfer-jit-cache was pinned to a literal 0.6.6 while vLLM 0.26.0 resolves flashinfer-python 0.6.14. flashinfer raises at import when the two disagree, and that exception is thrown inside the vLLM EngineCore, so Unsloth's GRPO fast_inference path fails at engine start with no earlier warning. A literal pin drifts again on the next vLLM bump, so the version is now read back from the resolved flashinfer-python, and the build proves `import flashinfer` works. Verified on the rebuilt image: flashinfer-python 0.6.14 with flashinfer-jit-cache 0.6.14+cu128, structlog 26.1.0, the export backend importable, and `unsloth list-checkpoints` exiting 0. tests/python/test_docker_llama_cuda_backend.py gains two static cases pinning both: the jit-cache version must be derived rather than literal and the build must import flashinfer, and the base venv must ask for structlog with the CLI reachability guard present. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: trim redundant comments in the image build files Comment-only pass over the PR's own files. No executable line changes. - Dockerfile / Dockerfile.studio: drop the decorative stage banner rules, the stale "5)" / "6)" step numbering, and the entrypoint pre-flight list that restated (and had drifted from) entrypoint.sh's own accurate header. Cut the llama.cpp asset bullet list that repeats fetch_llama_prebuilt.py's docstring and the structlog rationale already spelled out at the install site. - entrypoint.sh / studio_launch.sh: fold the section banners into the explanation lines that follow them. - docker-publish.yml: remove the comment rule lines around the job headers. - validate_studio_features.py: same for the numbered section headers. - smoke_test.py: drop the stale "~125M params" note on a 1B model. - unsloth_branding.py, unsloth_nb_view.py, unsloth_nb_pip_magic.py, colabTitle.ts: remove comments that restate the adjacent line. * docker: derive build.sh's arch-list banner from the Dockerfile The banner printed before the build hardcoded a second copy of the CUDA arch list, and it had already drifted: it showed 8.0;8.6;8.9;9.0;10.0;12.0+PTX while the Dockerfile builds with 7.5;8.0;8.6;8.9;9.0;10.0;12.0+PTX so anyone reading build.sh's output was told Turing is not covered when in fact it is. The echo does not feed the build, so no image was ever wrong; only the report was. Read the value back out of the Dockerfile instead of repeating it. The sed anchors on an optional-leading-whitespace assignment, so the Dockerfile's explanatory comment mentioning the same variable is not matched, and head -n1 takes the builder-stage ENV. Verified to yield 7.5;8.0;8.6;8.9;9.0;10.0;12.0+PTX against the current Dockerfile. * docker: keep transformers sidecar selection inside what the baked vLLM can import The image runs unslothai/notebooks unchanged by refusing a notebook's transformers pin and activating a baked sidecar on sys.path instead. Selection was a pure ceiling (smallest baked version >= the request) and ignored that vLLM is version-locked to transformers, so two of the four baked sidecars could not be imported by the baked vLLM 0.26.0 at all: 4.57.6 ImportError: Support for Transformers v4 is deprecated and was removed in vLLM v0.24.0 5.3.0 ImportError: cannot import name 'ALLOWED_LAYER_TYPES' from transformers.configuration_utils Those two are exactly the ones the common pins select. 241 notebooks pin 4.48/4.52.3/4.55.4/4.56.1/4.56.2/4.57.x and land on the 4.57.6 sidecar, 13 pin 5.2.0/5.3.0 and land on the 5.3.0 sidecar. All 254 died at `from unsloth import FastModel`, before the first model cell. Pointing UNSLOTH_TF_SIDECAR_ROOT at an empty directory and changing nothing else turned Gemma3 (270M) and Gemma3 (1B) GRPO into clean 22/22 and 25/25 passes. Put a floor in front of the ceiling. Which versions clear the floor is measured, not hardcoded: the build imports vllm.transformers_utils.config under every candidate sidecar, deletes the ones that raise, and records the lowest survivor. That is the vLLM module which reads the transformers API, it reproduces both failures, and it imports without a GPU, which matters because the build host has none. A request below the floor is clamped up to the lowest eligible sidecar, the closest version to the notebook's pin this image can actually run; a request above every sidecar still falls through to the baked transformers. Measured on the rebuilt image: sidecars 5.5.0 and 5.10.2 survive, floor 5.5.0, tf-sidecars drops from 250M to 123M, and all 13 distinct transformers pins found across the 433 shipped notebooks now reach `from unsloth import FastModel`. Gemma3 (270M) runs end to end exactly as shipped, 22 of 22 cells, loss 4.09 down to 0.85 over 10 steps. * docker: protect the tested training stack from notebook install cells The pip shim fronts pip/uv inside the notebook kernel so an install cell cannot replace the baked cu128 stack, but _KEEP only covered torch/vLLM/unsloth. Across the 433 shipped notebooks that left the training half wide open: trl 382 pin an older release, 378 of them ending the install cell with `pip install --no-deps trl==0.22.2`, against a baked trl 0.24.0 torchao 273 reinstall it and 2 pin 0.15.0, replacing 0.17.0+cu128 torchcodec 92 reinstall it and 26 pin 0.5 or 0.7.0, replacing the 0.11.0+cu128 wheel the Dockerfile pairs with torch 2.11 datasets 254 reinstall it, observed falling from 4.3.0 to 3.0.0 peft 225 reinstall it, observed falling from 0.19.1 to 0.14.0 accelerate 225 reinstall it hf hub 240 reinstall it and tokenizers 64, both version-locked to transformers and shipped in matched copies inside every sidecar So every notebook run mutated the stack the image was validated with, while the shim printed that it was keeping the baked versions. The membership criterion is "replacing this invalidates the tested stack or breaks unsloth", not "a notebook mentions it": snac, causal-conv1d, mamba-ssm, omegaconf, protobuf, sentencepiece and the rest still install normally. Verified in the rebuilt image by running the Gemma3 (270M) install cell verbatim: trl 0.24.0, peft 0.19.1, datasets 4.3.0, accelerate 1.14.0, torchao 0.17.0+cu128, transformers 5.14.1 and huggingface_hub 1.24.0 are all unchanged afterwards, the requested transformers pin is still recorded for the sidecar, and a package the image does not bake still installs. The existing shim tests used peft as their "unprotected package" sentinel, so they move to snac. * docker: fix the notebook sync race and widen the Colab intro strip Two bugs that compound. The sync backgrounds a GitHub refresh child and the parent exits immediately, firing its `trap finalize EXIT` (Colab intro strip plus categorized view rebuild) while the child is concurrently cp -a'ing refreshed notebooks into the same tree and rewriting the same state file. Six identical fresh-container boots reported cleaned 337/311/316/277/297/360 notebooks, and one of them published a categorized view holding 176 of 359 notebooks because both processes tore down and rebuilt the symlink farm at once. The lost writes are permanent: 222 to 309 recorded hashes no longer matched the file on disk, so those notebooks were treated as user-edited and skipped by every later strip, which is where 10 of the 23 notebooks still carrying the Colab intro came from. Keep the refresh detached, which is the whole point of it, and fix the ordering instead. One exclusive flock covers a whole invocation so the child cannot start until the parent has exited, the parent runs the finalize explicitly before it forks so the order holds even where flock is missing, the finalize is run-once, and the child re-arms it only when the refresh actually copied something. The strip itself only inspected cells[0], which missed 23 of the 433 shipped notebooks: 21 put the Colab badge in cells[0] and the sentence in cells[1] (Advanced_Llama3_2_(3B)_GRPO_LoRA, Falcon_H1-Alpaca, gpt-oss-(20B)-GRPO and friends), and 2 (NeMo-Gym-*) wrap the sentence in a single-line HTML comment. Scan the leading markdown block instead, stopping at the first code cell so it can never reach prose between code cells, and match the closed single-line comment form. The strip stays idempotent and leaves the content signature of all 433 notebooks unchanged, so the boot refresh does not re-copy and re-strip them forever. Measured on the rebuilt image: ten consecutive fresh-container boots all report cleaned 536 notebook(s) and view 359 notebooks in 26 folders, 0 of 433 notebooks retain the Colab intro (was 23), 0 recorded hashes mismatch (was 222 to 309), and a second boot on the same volume is a no-op. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: fix build.sh's arch-list read for relative invocations a05c58b6b read the arch list back out of the Dockerfile with "$(dirname "$0")/Dockerfile", but build.sh already does cd "$(dirname "$0")" near the top. The dirname is therefore applied twice, so every invocation by a path other than ./build.sh from inside docker/ died before reaching docker build: $ bash wt_r5748/docker/build.sh sed: can't read wt_r5748/docker/Dockerfile: No such file or directory EXIT=2 set -euo pipefail turns the sed failure into an abort, so this broke the whole script rather than just the banner it was meant to print. Use a bare filename, which is what the rest of the script already does (the docker build context below is a bare "."). Verified from the workspace root, from an absolute path, and from docker/ itself: all three now print arch list 7.5;8.0;8.6;8.9;9.0;10.0;12.0+PTX * tests: fix two environment-dependent failures found by the wider CI matrix Both surfaced only once the staging matrix ran these suites on runners the org queue does not cover. Neither is a product defect; both are tests asserting something their environment cannot supply. test_unsloth_pip_shim.py::test_forwarded_install_carries_protected_constraints reads the ambient environment through importlib.metadata.distributions. _protected_constraints_file correctly returns None when no protected package is installed, so no --constraint pair is appended, and the test then indexed execd[-2] unconditionally: E IndexError: list index out of range 1 failed, 115 passed, 2 skipped It failed on all four docker-test legs and in any bare venv, and passed upstream only because studio-backend-ci installs torch and transformers first. Its own sibling at line 93 already guards with len(execd) >= 2. Guarding the index alone would have left the test measuring whatever happened to be installed, so distributions() is now stubbed and the test asserts the real contract deterministically. A second case covers the other half of that contract, which is what a bare venv actually hits: with nothing protected installed the install must still be forwarded, just without the pair. test_select_cuda_jit_tools.sh stages libnvrtc as symlinks and asserts through readlink, because retargeting that symlink is what the function under test does. git-bash copies instead of symlinking unless MSYS=winsymlinks:nativestrict and the user is elevated, so readlink comes back empty and all 14 assertions fail on both Windows runners, taking tests/run_all.sh down with them for any Windows contributor. The code only ever runs inside a Linux container, so probe for real symlink support and skip when it is absent rather than assert something the filesystem cannot represent. Verified: the shim suite is 87 passed / 2 skipped in both a bare venv and a full one; the shell suite still reports 14 passed on Linux and skips with exit 0 under a simulated no-symlink filesystem. * docker: close five failure paths the review found Notebook sync, in-place publish. entrypoint.sh runs sync_notebooks and then execs the container command, so the detached refresh child is still copying while JupyterLab serves the same tree. cp -a writes through the destination inode, so a reader can catch half-written JSON and a save made after the recorded-hash check is destroyed and then recorded as pristine. Publish through a same-dir dot-prefixed temp plus an atomic rename, and re-read the hash once the staging copy is complete (the earlier check sits before middle_unchanged, a python subprocess, so the window was most of the loop). A single-file bind mount cannot be renamed over, so that path falls back to the previous copy. Notebook sync, first boot. A pre-existing file whose bytes already match the baked template fell through to cp -a, which is --preserve=all: as root that stamps root:root, the baked mode and the build mtime onto a bind-mounted host file and locks its owner out of editing it. Record it as managed instead. The hash is identical, so the state file is byte-for-byte what the copy wrote. unsloth-studio-update. The post-update import check only warned, then the default restart replaced a process that was serving fine with one known not to import. supervisord retries startretries times, lands in FATAL and never leaves it on its own, so the container serves nothing until someone execs in. Keep the running service and exit non-zero with the remedy. unsloth-llama-update --check. resolve_latest swallows every failure into an empty string, which fell into the "up to date" branch and exited 0, so the command reported a state it could not observe. Report UNKNOWN and fail. unsloth-llama-update rollback. The in-place restore iterates the backup's entries, so a file the new release introduced survives it and the restored tree is mixed-version; ggml dlopens every libggml-*.so next to the binaries. Clear the install dir before restoring, gated on the drain having completed, because before that an entry there can still be the only copy of an old file. docker-publish ref freeze. git ls-remote exits 0 whether or not a ref matched, so a non-zero exit means the remote was never reached. That exit was lost twice over: first element of a pipeline, and a run step with no explicit shell runs under bash -e without pipefail. The step exited 0 and published ref=main, which the amd64, arm64 and Studio builds each resolve again, so one multi-arch tag could carry different revisions. Fail the prepare job instead, keeping the passthrough for the reachable-but-no-match case it was written for. Jupyter output select. lastPointerOutput was only replaced by another pointer-down, but J/K/arrow cell navigation fires none, so Ctrl/Cmd+A on a later cell selected the previously clicked output and suppressed notebook:select-all; after a re-run the node is detached and the chord did nothing at all. Revalidate the remembered output (still in the document, still in the active cell) before using it as the fallback. Tests: four static guards in test_docker_nb_sync_race.py, a new behavioural test_docker_update_helpers.py driving both helpers with stub pip, supervisorctl and mv, a new test_docker_publish_ref_freeze.py that executes each resolver step under bash -e with a failing ls-remote, and a source check in validate_studio_features.py. Each fails against the code before this change; the interrupted-drain case also fails against the unconditional form of the rollback fix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docker: close four holes the review found docker-publish.yml: the llama.cpp tag resolver was the one step in the prepare job still using `curl | sed` without pipefail. The runner's default `bash -e` shell takes sed's exit status, so an unreachable github.com left TAG empty and the step published the mutable `latest`. Both arch legs re-resolve that through fetch_llama_prebuilt.py and Dockerfile.studio resolves it a third time, so a release cut mid-run can put different llama.cpp bundles under one manifest. Capture the redirect first and fail the job when it is missing or does not land on a release tag, matching the three ref resolvers below it. unsloth_nb_strip_colab.py: strip_notebook read, parsed and then unconditionally os.replace'd. The refresh child re-arms finalize after the entrypoint has execed the container command, so JupyterLab is already serving the tree and a save landing in that window was destroyed, after which migrate recorded the cleaned hash and marked the notebook pristine forever. Re-read the hash once the staged copy is complete and drop it when the file moved, the same rule the refresh publish in unsloth_sync_notebooks.sh already follows. unsloth_nb_view.py: ownership for the view teardown accepted any symlink target under DEST, but every link the tool creates points at DEST/nb. A shortcut the user made in the landing dir to their own file elsewhere in the checkout was therefore classified as ours and deleted on the next boot. Key ownership on DEST/nb instead. cellNav.ts: the edit-mode boundary test compared the cursor line against editor.lineCount, both logical, while JupyterLab wraps markdown and raw editors by default (StaticNotebook.defaultEditorConfig). A one-line markdown header renders as several visual rows, so every arrow left the cell and the wrapped rows could not be reached. Ask CodeMirror whether it can still move one visual line (EditorView.moveVertically, compared by coordsAtPos top) and keep the logical test as the fallback for a non-CodeMirror editor. New tests: 12 passed / 8 failed before, 20 passed / 0 failed after. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record what the refresh published, and let an install-cell fix reach the container Three review findings from the latest round. Record the staged hash, not a re-read of the published file. rename(2) is atomic, but the hash taken after it is a second, unprotected read, and JupyterLab is already serving the tree while the refresh child runs. A save landing in that window was recorded as the sync-owned pristine version, so the next refresh was allowed to overwrite the user's work. unsloth_sync_notebooks.sh now hashes the dot-prefixed per-PID staging copy before the rename, and strip_notebook hands migrate() the hash of the bytes it wrote instead of rehashing the live file. Keep package specifications in the refresh signature. unsloth_nb_content_sig counted the whole generated install cell as boilerplate, so an upstream fix that only bumps a transformers pin or adds a dependency reported SAME. SAME does not just skip the copy, it re-records the old hash, so the notebook never converged and the image kept selecting the wrong transformers sidecar. The cell is now hashed with only its cosmetics normalized away, so comment and spacing churn still reports SAME while a changed spec refreshes. A #fragment glued to a token is treated as data, so git+https://host/repo#subdirectory=pkg keeps selecting the package. Fail the studio update when the zoo lookup never reached the remote. git ls-remote --exit-code reserves status 2 for "reached the remote, no matching ref"; 128 means the lookup did not happen. Conflating them installed the requested unsloth revision with unsloth-zoo main as soon as the network recovered for the pip call, and the two share a private API. Same rule the publish workflow already applies when it freezes refs. Regression tests: test_docker_nb_sync_race.py grows a static guard plus three behavioural cases that drive the real refresh child against a local git remote with an mv shim landing the save inside the window (3 failed before, 19 pass after); test_docker_nb_strip_colab_race.py grows the post-replace window case (1 failed before, 5 pass after); test_docker_nb_install_cell_sig.py is new (4 failed before, 7 pass after); test_docker_update_helpers.py grows the three zoo-ref cases (1 failed before, 10 pass after). * Reconcile three guards that merged cleanly but asserted against each other Nothing here conflicted textually. Each is a case where one side wrote a rule and the other side wrote code the rule rejects, so the merged tree parses and fails. tests/studio/install/test_install_llama_prebuilt_logic.py Main added test_activate_install_tree_keeps_existing_install_when_aside_move_fails, which injects EXDEV into os.replace and expects the aside-move to fail and leave the previous install in place. EXDEV was picked as a convenient stand-in for a generic move failure, but on this branch it is the one rename failure the aside-move deliberately completes by copy: a Docker studio build moves the base image's llama.cpp onto a different overlay layer, and rename cannot cross it. Switched the injected error to EIO, which is still non-busy and still exercises exactly what the test is named for, so main's rule holds unchanged for every move failure that really is one. Added three tests for the behaviour that displaced it, since the merged aside-move path had no coverage of its own: the EXDEV aside-move completes and the activation goes through; a copy that dies partway leaves no partial tree at the destination and does not touch the source, which matters because callers read dst.exists() as proof of a complete tree; and a linked install is refused rather than dereferenced, matching the refusal main already makes on the rollback-restore copy. .github/workflows/docker-publish.yml Main added tests/studio/test_main_runs_survive_merge_bursts.py, which requires every workflow triggered by a push to main to put each commit in its own concurrency group. docker-publish.yml, added by this branch, used one group per ref to serialise runs so two pushes could not both retag :latest. That serialisation does not exist: GitHub cancels a PENDING run as soon as a newer one is queued, and cancel-in-progress only spares runs already executing, so the shared group would have dropped the earlier main push rather than queuing it. Appended the per-commit suffix the guard asks for, which leaves tag and dispatch runs sharing a per-ref group as before, and rewrote the comment to record the trade: two main commits can now race to retag a mutable tag, where previously one of them published nothing at all. Encoding on reads of checked-in files Main added tests/test_source_read_encoding.py, which requires every read of a checked-in file from the test trees to name an encoding so it does not break on Windows the moment that file gains a non-ASCII byte. Five reads in this branch's new tests did not: the Dockerfile reads in test_docker_llama_cuda_backend.py and test_docker_tf_sidecar_vllm_floor.py, the sync-script read in test_docker_nb_sync_race.py, the compat-module read in test_docker_tf_sidecar_vllm_floor.py, and the vision.py read in test_generate_kwarg_gate.py. All five now pass encoding = "utf-8". Verified against bare origin/main as well as the pre-merge branch head: after these three fixes the merged tree's failure set contains nothing that is not already failing on one or both of them. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Classify uv's --prerelease-package as a value-taking flag The image build fails at [runtime 12/16] on the pip shim's own selfcheck: [unsloth-nb] value flags missing from _VALUE_FLAGS: {'uv': ['--prerelease-package']} unsloth_pip_shim.py fronts pip/pip3/uv inside notebooks and has to know which flags consume the token that follows them. _VALUE_FLAGS is that table, and _selfcheck_value_flags compares it against the baked tools' own --help at image build time, so a toolchain bump fails the build rather than a user's cell. The trigger is external drift rather than anything in the shim. The Dockerfile installs uv unpinned (pip install uv), and uv 0.12 added --prerelease-package alongside the long-classified --prerelease. The table already carries the rest of that per-package family (--config-settings-package, --exclude-newer-package, --no-build-isolation-package, --no-editable-package, --no-sources-package, --refresh-package, --reinstall-package, --upgrade-package), so this is a plain omission of one member. The image that last built shipped uv 0.11.32, which predates the flag. Left unclassified, the scanner reads the flag's VALUE as an install target, so uv pip install --prerelease-package foo bar treats foo as a package to install and leaves a dangling flag. That is the same shape as the --torch-backend and --extra misreads this table was introduced to prevent. The selfcheck enumerates every gap for both tools in one pass rather than stopping at the first miss, so this single flag is the complete change. Confirmed against the image venv with uv 0.12.9 and pip 26.2.1, the current releases: the unpatched table reports exactly this flag and nothing else, and the patched table passes for pip and uv alike. pip's entries are not stale. tests: cover --prerelease-package in the value-flag no-op parametrisation, and assert the whole per-package family is classified so the next sibling uv grows fails in the suite instead of at image build time. The existing help-derived drift guards stay opt-in, since repo CI runs whatever pip and uv are current. * Docker: bake the llama.cpp marker with the same tag split the installer uses Studio's prebuilt marker schema, written by studio/install_llama_prebuilt.py, splits the installed version across two fields: "tag" holds the normalized upstream build (b10715) and "release_tag" the full fork release (b10715-mix-86bd2d3). The freshness check reads them deliberately opposite ways, base for display and full for the up-to-date comparison, so a mix release does not render as a permanent pending update. docker/fetch_llama_prebuilt.py ignored that split and baked the full release tag into both fields. The image and the in-app updater therefore disagreed about what "tag" means, and Studio's installed-version display changed shape depending on which of the two had run last in a container. Fixed on the writer side rather than at display time. The alternative, flipping the freshness display fallback to prefer "release_tag", would have papered over the disagreement without resolving it: the field would still carry a different kind of value depending on the installer, every other reader of it would still have to guess, and the display change would contradict the base-tag rendering that the freshness and update-status suites pin. The marker "tag" now takes the bundle's own upstream_tag when the tarball carries one, else the base build parsed off the release tag, mirroring llama_cpp_freshness.parse_base_build. setdefault semantics are unchanged, so a tarball that already populates the field is still left alone, and no timestamp is written, so layers stay identical. unsloth-llama-update reads the marker back and compares it against the release feed, and it preferred "tag". Now that the baked marker records a base build there, and on any container where the in-app updater had already written one, a raw comparison of b10715 against the latest tag_name b10715-mix-86bd2d3 reports an update that applying it can never clear. It now reads "release_tag" first, comparing the full release identity against the full tag_name on both sides. Tests: tests/python/test_docker_update_helpers.py gains a --check regression guard for a split marker (up to date, and the full tag reported as installed), a companion asserting a genuinely newer release is still offered, and base build normalization cases for the fetcher. 16 passed there, and the Studio suites that pin the display and comparison behaviour are untouched: 221 passed across test_llama_cpp_freshness, test_llama_cpp_update, test_combined_update, test_llama_route, test_llama_backend_switch, test_inference_status_route and test_whisper_cpp_freshness. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gate the logits_to_keep injection on the plain 5.0.0 sentinel Two fixes to the transformers v5 gate this branch added to unsloth_base_fast_generate. 1. The sentinel was wrong for the release it exists for. Version() here is unsloth_zoo's wrapper, not packaging's class: it keeps the leading numeric run and appends ".1" whenever the string carried a suffix, so Version("5.0.0.dev0") is 5.0.0.1. transformers 5.0.0 FINAL therefore sorted below the sentinel and took the legacy 4.x branch, which is the exact release the gate was written to skip. Measured against the installed wrapper: transformers Version(tf) < Version("5.0.0.dev0") < Version("5.0.0") 4.57.6 True True 5.0.0.dev0 False False 5.0.0rc1 False False 5.0.0 True False 5.0.1 False False The plain sentinel still catches the prereleases, since 5.0.0.dev0 and 5.0.0rc1 both normalize to 5.0.0.1. It is also the form every other v5 gate in the tree uses (import_fixes.py, models/loader.py, models/rl.py, models/_utils.py). tests/test_generate_kwarg_gate.py pins it. 2. tests/test_fa2_fast_generate_bypass.py rebuilds unsloth_base_fast_generate from source with ast.parse + exec into a hand-built globals dict, so the function's free variables have to be listed there by hand. The gate put Version and transformers_version on the pre-delegate path for the first time, and both FlashAttention tests died on "NameError: name 'Version' is not defined" before reaching what they assert. Seed the two names, the way tests/python/test_fast_language_model_ text_only.py and tests/test_new_mapper_fetched_fp8.py already do. packaging and importlib.metadata rather than unsloth_zoo.utils and transformers: importing unsloth_zoo pulls in bitsandbytes and CUDA, which is why the file rebuilds the function from source in the first place. The exercised path does not depend on the value, only on the names resolving, so this does not make the file transformers-version sensitive. tests/test_fa2_fast_generate_bypass.py 12 passed 2 failed -> 14 passed. tests/test_generate_kwarg_gate.py 5 passed -> 6 passed. * Fix four docker publish and shim defects found in review Pass constraints before the end-of-options marker. The pip shim appended --constraint after the kept arguments, so a cell running "pip install -- pkg" handed the real tool "-- pkg --constraint <file>". Both pip and uv stop parsing options at a bare --, so they read --constraint as a package name and the cell died. Insert the pair ahead of the terminator; with no -- present the arguments are unchanged. Publish --out with usable permissions. The staging file came from mkstemp, which is 0600, and nbconvert truncates that same inode, so os.replace carried 0600 onto the destination. Under the documented bind-mount layout the container is root and the host user is not, so the executed notebook was unreadable, and overwriting an existing output dropped its mode and owner. Reuse the destination's metadata, or the umask-derived mode a plain write would have produced. Read the base manifest digest through this run's own tag. metadata-action sorts tags by priority, so tags[0] on a main push is the mutable :core, and main runs are deliberately not serialised. A second run retagging :core between the manifest creation and the inspection would export its digest, and build-studio would layer this run's Studio over that commit's base. Select the core-sha- tag, keeping tags[0] as a fallback. Stop the zoo tag probe conflating an unreachable remote with a missing tag. git documents ls-remote --exit-code status 2 as "reached the remote, no matching ref"; every other non-zero means the lookup never happened. Swallowing both let a transport blip pair a released unsloth tag with zoo main, which the freeze step below then resolved cleanly once the network recovered. Only status 2 may fall through, matching the three resolvers alongside it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Smoke-test the image this run published, not a mutable tag The manifest digest export was moved onto this run's own tag, but the two smoke test steps still resolved their image with tags[0]. metadata-action sorts tags by priority, so on a main push tags[0] is the mutable :core for the base image and :latest for the Studio image, and main runs are deliberately not serialised. An overlapping run can retag either between the publish and these pulls, so the smoke test and the Studio probe could validate a different run's image while reporting on this one. The comment above the base step asserted the opposite. Select the run-specific sha tag in both steps, keeping tags[0] and the existing empty-metadata fallbacks. Confirmed the selector against representative metadata: the base list resolves core-sha- rather than :core, the Studio list resolves sha- rather than :latest, a tag push still resolves its sha tag, and empty metadata still yields an empty string so the defensive branch below is unchanged. Also anchored the digest export's selector on the colon so all three read alike. * Take the notebook transformers pin from install lines only _scan matched the pin regex against the whole source of every code cell, and the pin outranks the model tier, so any prose mention won. A commented out legacy line such as "# !pip install transformers==4.57.6" above a gemma-4-12b notebook made 4.57.6 the requested version, which clamps up to the lowest eligible sidecar rather than the tier the model needs, and the kernel was launched on it. Nothing installs in that case, so the pip shim never gets a chance to correct the marker or PYTHONPATH. The module docstring already said "its install-cell pin", and the sibling pip shim already reads real argv tokens rather than raw text. Match the invocation instead: pip, pip3, uv pip, an interpreter with -m pip, with or without a ! or % magic, at any indent, ignoring commented lines and trailing comments while keeping backslash continuations, since real install cells split one command over several lines. Checked against all 561 notebooks in unslothai/notebooks: 430 pins before, 430 after, zero differences, so no shipped notebook changes tier. A commented pin now resolves to no pin while a genuine multi-line uv pip install still resolves to its own. * Scan only live code for the pin and the model, and normalise names per PEP 503 The pin scan was restricted to install invocations, but the model scan still read the whole raw cell source and the first match wins, so a commented out or triple quoted model reference above the real call decided the sidecar tier. Swapping models by commenting the previous line out is the most routine notebook edit there is. Two shipped notebooks are affected today: Openenv_wordle_grpo and its AMD variant both carry a commented model_name for gemma-3-1b-it directly above the live Qwen3-4B argument, and the scan picked the commented one. Both happen to resolve to the same tier, so nothing is broken in practice there, but the detection was wrong. Blank triple quoted bodies and comments once, then scan that for both the pin and the model. Newlines are preserved so the backslash continuation logic is unaffected, and single quoted strings are left intact because the model name lives inside one. This also covers install text embedded in a docstring or an assigned script, which the previous line wise match treated as a real install. Verified across all 561 notebooks in unslothai/notebooks: no pin changes, and the only two model changes are the two notebooks above, where the new answer is the correct one. Separately, _canon collapsed only underscores, so a distribution spelled with a dot or a repeated separator, such as unsloth.zoo or nvidia.cublas-cu12, did not match the protected entries and could replace a baked wheel. Normalise any run of hyphen, underscore or dot to a single hyphen, which is what PEP 503 defines and what the lowercasing here was already reaching for. * Normalise every distribution name through one PEP 503 helper The previous commit fixed only the final fallthrough in _canon. The function has five earlier returns, for a PEP 508 direct reference, an egg fragment, a wheel filename, a source archive name and a VCS basename, and _sdist_name, _local_project_name and the protected-constraints builder each carried their own copy of the same partial rule. All of them collapsed underscores only, so a protected package spelled with a dot still missed the keep list: measured at the previous head, five of six spellings of unsloth.zoo were not protected and only the plain version spec was. Route all ten sites through one helper that collapses any run of hyphen, underscore or dot to a single hyphen. The constraints builder keeps the raw metadata name in the pin it writes and normalises only the key it matches on. Regression tests cover each branch, the prefix match, and that torch-directml, torch_tensorrt and torchsde are still not confused with torch. They fail at the previous head. * Keep install-cell indentation in the signature, and the notebook owner on refresh Two defects in the notebook refresh path. The install-cell signature collapsed each line with a whitespace join, which dropped leading indentation. Upstream generates that cell as an if/else on whether COLAB_ is in the environment, so a line moving out of the Colab only branch starts running on every runtime, which is the branch the image takes. Measured at the previous head, dedenting the torchao install hashed identically to leaving it in, and the unchanged verdict re-records the old hash, so that notebook would never refresh. Keep the indent, expanding tabs so a tab or space rewrite at the same depth still reads as cosmetic, and skip lines that were only a comment so bare indentation cannot resurrect comment churn. Publishing a refreshed notebook renamed a staging copy over the destination. Rename swaps the directory entry, so the staged inode carried the freshly cloned root ownership and mode onto a bind mounted file the host user owns, and they could no longer edit their own notebook. Such a file is reachable because the first boot populate deliberately records a bind mount matching the baked template as managed without copying it, precisely to avoid stamping root on it. Apply the destination metadata to the staging file first, and drop preserve all from the single file bind mount fallback so it stops chowning an existing destination. Both are best effort, mirroring unsloth_run.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Forward dependency-group installs, and keep the owner when cleaning notebooks A flag-only install such as "pip install --group test" scanned to no target, so the shim returned without invoking pip while printing that everything was ok. pip documents --group as installing a named dependency-group and --requirements-from-script as installing a script PEP 723 dependencies, so each is the install target with no package on the command line. Mark them as such, in both the spaced and inline value forms. A cell carrying only option flags, or only baked packages, still correctly no-ops. The Colab cleanup replaced the notebook from a staged temporary file without carrying the destination metadata across, which is the third site of this on the branch after the run helper and the sync script. Since the sync script records a bind mounted notebook matching the baked template as managed without copying it, precisely so the host user keeps owning it, cleaning that notebook handed it back root owned. Apply the destination owner and mode to the staged file first, best effort as elsewhere. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Give new outputs the host owner, and keep a failed sync retryable The --out publish helper only set a umask derived mode when the destination did not exist yet, so a new output written by the root container into the bind mounted workspace stayed owned by root and the host user could read it but not edit it, unlike an existing output whose owner is preserved. Take the owner of the directory it lands in, which under the documented layout is the host user. The refresh advanced the sync marker even when a notebook could not be written. Both the staging copy and the publish share a directory with the destination, so a read only mount fails them, and each failure dropped that path from the state file while the marker still recorded the upstream commit as fully synced. The next start then exited on remote equals last and never retried, and the missing record made a later refresh read the stale destination as user owned and keep it indefinitely. Carry the previous record forward on either failure, count them, and only stamp the marker when none occurred. The staging half is the one a read only directory actually hits; the publish half is covered for the same reason. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rewrite the options dicts the single-compile-worker guard cannot reach The guard set torch._inductor.config.compile_threads = 1, exported TORCHINDUCTOR_COMPILE_THREADS=1 and replaced unsloth_zoo's determine_compile_threads, then found that none of it reached the compile sites it exists to protect. It runs after `import unsloth_zoo`, and that import chain (unsloth_zoo/__init__ to temporary_patches to gpt_oss to common) has already built its module-level Inductor options dicts by calling determine_compile_threads and storing the integer. Nine live snapshots hold the original count. They are passed to torch.compile as `options`, and torch wraps them in _TorchCompileInductorWrapper, which hands them to compile_fx as config_patches applied with config.patch, so `options` outranks both the environment variable and the config value set above. Measured against the torch in these images: with config.compile_threads = 1 and options {'compile_threads': 32}, get_compile_threads() inside the patch reports 32. TORCHINDUCTOR_COMPILE_THREADS is popped again later by the zoo, so the env belt was gone too. Rewriting the dicts in place rather than rebinding them is what keeps this small: the same dict object is re-exported by loss_utils, rl_replacements, cross_entropy_loss and temporary_patches.utils, and closed over by the functools.partial in common.torch_compile, so one write reaches all of them. Dicts built after this point still go through the patched function. The test extracts the guard block verbatim from the shipped _gpu_init.py and execs it against a synthetic unsloth_zoo module graph with the real shapes: one shared re-exported dict, two per-model dicts, the partial, and a non-zoo module that must be left alone. No docker, no GPU, no real compile. It fails at the previous head and passes here. * Do not claim work that failed: incomplete populate, unpublishable output Three fixes, two of them the same shape: a step reports success for work it did not do, and the state it writes then stops anything from retrying. unsloth_sync_notebooks.sh first-boot populate. A cp -a that fails, a destination subpath that is momentarily not a writable directory or a full disk, is skipped silently and gets no state entry, so the every-boot restore in 1b never brings it back. Stamping the template commit unconditionally then told the refresh in 2) that this commit was already synced, so it exited early too and the notebook was gone for good. The loop now counts failures, which needed the pipeline turned into a process substitution to survive it, and leaves the sync marker off plus a .unsloth_sync_partial marker that re-runs populate on the next start. That is the offline half of the retry: 2) needs a clone. The marker is in record_state's skip list so it is never recorded as a notebook. unsloth_run.py --out. Binding the OUTPUT FILE itself, -v $PWD/out.ipynb: /workspace/out.ipynb, makes the destination a mount point, and rename(2) onto one returns EBUSY even though the file is writable. The finally block then deleted the staged executed notebook, so a run that had already finished was lost outright. It now falls back to writing through the existing inode, which is what such a mount needs since the host tracks the inode and not the directory entry, and if even that fails it keeps the staged result and prints where it is. Not atomic, unlike the rename, so it stays the fallback. docker/run.sh usage. With the default unsloth/unsloth:latest, no command means Dockerfile.studio's CMD, the Studio launcher, not a python REPL. The line promising a REPL contradicted the paragraph twelve lines below it describing the same invocation as starting Studio and JupyterLab. Corrected the usage line and added the explicit `bash docker/run.sh python` form rather than changing the default, which the ports examples below depend on. Tests: a behavioural populate-retry file driven against the real script offline with a blocked path that fails for root too, and two publish tests raising the errno the kernel raises for a mount point, since a real bind mount needs root. All new assertions fail at the previous head. 1668 pass in tests/python, with the one pre-existing unrelated failure in test_grpo_ddp_model_config.py unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Own the directories we create, and do not read a loader failure as a version Output directory ownership. mkdir(2) gives a new directory the CALLER's uid; only the setgid bit carries anything down from the parent. The container runs as root while -v $PWD:/workspace is the host user's own tree, so an --out pointing into a subdirectory that does not exist yet left them a root-owned directory they cannot write, and _stage_metadata then derived the OUTPUT's owner from that same just-created directory, so both ends of the path came out unusable. Every level created is now chowned to the nearest EXISTING ancestor, outermost first, which is the right anchor because a user can own /workspace/host/projectA without owning everything above it. An existing directory is untouched. llama-server probe. The sanity check ran each binary with --version and asserted a substring, ignoring the exit code. For llama-server the substring is "version", and that is the word the dynamic loader itself uses when it refuses to start a binary: `./llama-server: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by ./llama-server)`. The program never reaches main and exits nonzero, but the check passed, so the image could ship a server that cannot run, and Studio chat depends on it. llama-server does have --version, so a healthy run exits 0 and that is now required. The two llama-quantize probes keep the banner check alone, because an unknown flag is not required to exit 0 across builds while a loader failure never prints "usage". The checks moved into sanity_check_binaries() at module scope so the regression test can drive them against stub binaries. The alternative was exec'ing a slice of the file out of main(), which scripts/lint_exec_literals rightly refuses. Tests: two new files' worth of cases, chown recorded rather than observed because chowning to another uid needs root, and the loader message reproduced through a quoted heredoc because it contains a backtick. All seven fail at the previous head. 1676 pass in tests/python with the one pre-existing unrelated failure in test_grpo_ddp_model_config.py unchanged, and both lint gates are clean. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip the chmod-based publish failure under root, where chmod means nothing test_a_failed_publish_does_not_claim_the_commit_is_synced makes the destination directory unwritable with chmod 0500. Root bypasses that outright, since CAP_DAC_OVERRIDE is documented as "bypass file read, write, and execute permission checks", so inside a root container the publish succeeds, the sync marker advances and the assertion fails deterministically rather than catching anything. Skipped under uid 0 rather than rebuilt, because the lane that runs it, studio-backend-ci's auto-discovered CPU job on ubuntu-latest, is not root, so the coverage stays where it works and the skip only fires for someone running the suite inside a root container. The docstring says which capability is responsible and points at the sibling that keeps the same "do not stamp a commit whose work failed" property root-proof: test_docker_nb_populate_retry blocks the path with a regular file where a directory has to be, which is ENOTDIR for everyone. * Give created notebook directories the host owner too, at all three sites Sibling sweep of the output-directory fix in unsloth_run.py, found by looking for the same shape rather than waiting for it to be reported. unsloth_sync_notebooks.sh already works hard to keep a published FILE host-owned: stage_metadata chowns the staged inode from the destination before the rename, and cp_keep_meta writes through an existing inode rather than replacing it. New DIRECTORIES had none of that. mkdir(2) gives the new directory the caller's uid and only the setgid bit carries anything down from the parent, so a category folder upstream adds landed root:root in the middle of a tree whose files those two helpers are careful about, and the host user could not write into it. mkdir_keep_owner is the shell twin of unsloth_run.py's _makedirs_as_host: it walks up to the nearest EXISTING directory, creates the path, then chowns each level it created from that ancestor, outermost first so a partial failure still fixes what it can. The ancestor rather than $DEST, because a bind mount can be nested inside the notebooks tree. Best effort like its neighbours, since a filesystem that refuses chown must not cost the user their refresh. All three sites route through it: first-boot populate, the every-boot restore of deleted notebooks, and the refresh publish. Fixing one and leaving the other two is how this keeps coming back, so there is a test that fails if a bare mkdir on a $DEST path reappears anywhere below the helper. Tests drive the real function out of the shipped script with chown replaced by a recorder on PATH, so the per-level calls and their reference are asserted without needing root. All three fail at the previous head; 1679 pass in tests/python with the one pre-existing unrelated failure unchanged, and shellcheck is clean. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim comments across the Docker image work Cut the explanatory comments back to what a reader cannot get from the code itself: invariants the code does not state, citations for non-obvious choices, and the reasons a tempting alternative is wrong. Deleted rather than reworded: restatements of the adjacent statement, section banners, spacer lines, type and signature echoes, and measured numbers that belong in the PR description. Kept deliberately: - The usage and environment-variable tables in run.sh, entrypoint.sh, studio_launch.sh, unsloth_studio_update.sh and unsloth_sync_notebooks.sh. These are the user-facing documentation for those entry points. - Lines 2 to 21 of unsloth_llama_update.sh, which usage() prints verbatim with sed -n '2,21p'. - The "--- refresh child ---" and "1) First-boot populate" markers in unsloth_sync_notebooks.sh, which test_docker_nb_sync_race.py locates by string index. - The gating rationale in .github/workflows/docker-publish.yml, which is the only record of why several steps exist. - The CSS comments inside the labextension template literals, which are string content rather than comments. Note that lines 2 to 21 of unsloth_llama_update.sh are program output rather than commentary: usage() prints them. The same applies to the option tables read by run.sh and unsloth_sync_notebooks.sh. No functional change. Verified comment-only against the previous head with an AST comparison over every changed Python and TypeScript file, bash -n on every changed shell script, and a ruff report identical to the one before the pass. Test comparison, run twice with the same command, once on this tree and once on a fresh detached worktree at the previous head a46dd38d4: 43 failed, 1404 passed, 123 skipped, 11 errors Both runs produced those identical counts, so the failures predate this commit and come from the local torch and torchaudio install. Four files error during collection for the same reason and were excluded from both runs: tests/python/test_construct_chat_template_validation.py tests/python/test_get_chat_template_escaping.py tests/python/test_grpo_logit_transform_families.py tests/python/test_rocm_bf16_capability.py The 58 scripts in tests/sh all pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Preserve symlinks in the cross-device copy, and quote the Jupyter view paths Two review items on the same theme: a value that survives one code path and not its neighbour. install_llama_prebuilt's EXDEV fallback copied the existing install aside with shutil.copytree and the default symlinks = False, which DEREFERENCES every link inside the tree. A Linux llama.cpp bundle is full of soname links (libllama.so.0 -> libllama.so), so the aside copy duplicated each shared library, and restoring it after a failed activation replaced the install's link topology with independent files. Measured on a three-entry probe: default gives three real files, symlinks = True gives one file and two links. The rollback-restore copy below already passed symlinks = True, so this was one site fixed and its sibling missed. The three copytree calls in this file that pass dirs_exist_ok are deliberately left alone, and the comment says why: combined with symlinks = True, copytree raises FileExistsError on any name the destination already holds, because CPython calls os.symlink without unlinking first. Adding the flag there would turn a re-install over an existing tree into a crash. Their cost is duplicated bytes rather than a broken install, so they want an unlink-first helper as a follow-up. studio_launch interpolated UNSLOTH_NOTEBOOKS_VIEW_DIR straight into Python string literals in a heredoc. A double quote in the path closed the literal and made jupyter_lab_config.py a SyntaxError, so the documented override stopped JupyterLab starting, and a backslash silently produced a different path since Python reads \t in a literal as a tab. Both are legal in a POSIX path. The three settings are now emitted through python repr(), with the values arriving via the environment the way the password block above already does. The other two heredocs in that file were checked and are safe: one is quoted, and the other reads os.environ rather than interpolating. Eight tests drive the real generator extracted from the shipped script, compile and exec the result the way Jupyter loads it, cover five awkward path shapes and a newline injection attempt, and pin the premise by reproducing the old heredoc form and asserting it really does raise SyntaxError. 5047 pass across tests/python and tests/studio/install with only the pre-existing unrelated failure. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hand a brand-new notebook to the host user, in all three places it is created stage_metadata returned early when the destination did not exist, on the reasoning that there was nothing to inherit from. That was the defect rather than the justification: with no destination, the staged file keeps the clone's root:root 0644 and the rename publishes it, so a notebook upstream had just added arrived in the host user's own bind-mounted tree owned by root and they could not edit it. The two `cp -a` copies in first-boot populate and in the restore of a deleted notebook have the same shape, since -a is --preserve=all and the template is equally root-owned. unsloth_run.py's _stage_metadata has had a new-file branch for exactly this since the earlier ownership fix on this branch: take the mode a plain write would have produced, 0666 & ~umask, and the owner of the directory the file lands in, which under -v $PWD:/workspace is the host user's. own_like_dir is that branch, and all three sites now call it. The existing-destination path is untouched and still copies from the FILE, which is what preserves a mode the user chose themselves. One existing assertion had to change rather than being satisfied: it asserted `[ -e "$2" ] || return 0` was present, with the comment that a brand-new notebook has no destination metadata to inherit. That encoded the behaviour being fixed, so it now asserts the fall-through instead, and the comment says why it flipped. Tests drive the real shell functions out of the shipped script with chown replaced by a recorder, so the reference argument is asserted without needing root, plus a sibling guard that fails if either template copy stops handing the file over. All four fail at the previous head. 1690 pass in tests/python with only the pre-existing unrelated failure, and shellcheck is clean. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip a baked package only when the image actually has it The pip shim dropped every _KEEP name unconditionally, so a notebook cell recovering from a bake the Dockerfile is allowed to skip turned into a silent success. Measured on the shipped shim with torchcodec absent from the venv: pip install torchcodec [unsloth-nb] kept baked versions, skipped: torchcodec [unsloth-nb] nothing to install after keeping the baked stack; ok. Nothing was installed and nothing was baked, so the message is untrue and the import stays broken. Both of the affected bakes are deliberate: docker/Dockerfile line 249 ends the torchcodec install with `|| echo ">> torchcodec bake skipped (no matching wheel for ...)"`, and the vLLM block falls through to "vLLM skipped on ${TARGETARCH}: install or import check failed (fail-soft on non-amd64)" with a `uv pip uninstall vllm` behind it. An arm64 image can therefore ship with no vLLM at all while the shim reports it as kept, leaving the GRPO fast_inference path unavailable. An absent package is nothing to protect. Every drop decision now goes through one predicate, _is_protected, which asks importlib.metadata whether the distribution is really installed; if the scan cannot be read it keeps the old, stricter answer rather than opening the stack up. Forwarding instead is safe because _protected_constraints_file already pins the installed protected set for the resolver, so the recovery install either lands under the baked pins or fails loudly, and both beat a false success. Routed all three drop sites through the predicate rather than the one the review pointed at: the command line scan, _filter_requirements_file for -r and -c, and _classify_flag_target for -e and -P. test_every_drop_decision_goes_through_the_ one_predicate keeps them from drifting apart again. Both shim test fixtures now pin the installed set to a fully baked image. They were reading the ambient venv, which has no torchcodec, so test_cuda_matched_wheels_are_not_replaced_by_pypi_builds would have passed or failed on the runner rather than on the shim. New coverage in tests/python/test_docker_pip_shim_training_stack.py: the absent package forwards on all three sites, a present one is still skipped, an unreadable metadata scan falls back to skipping, _installed_names really reads a venv rather than returning an empty set, and a premise pin on the Dockerfile fail-soft branches so the absence path cannot quietly become dead code. Neutering the installed check turns the four absence tests red; inverting the metadata fallback turns its test red. * Restore the accepted-cost note on the per-sha concurrency group The comment trim cut the sentence that recorded why main runs are deliberately not serialised, leaving the per-sha group looking like an oversight. It is the only record of the trade-off: a shared group would not have serialised a main burst, it would have discarded the earlier commit and published nothing for it, so the race to retag was accepted in exchange. Also states the part that keeps the race harmless for image contents, which was only visible from the two consumer sites: the digest handoff and the smoke test both read the immutable :core-sha- tag rather than tags[0], so a Studio image is always layered on its own run's base and the smoke test never inspects another run's image. What can go stale is the tag pointer, not the artifact. * Read the pin out of the cell about to run, not just the shim marker The pre_run_cell hook only consulted the marker file, which is the pip shim's record of a PREVIOUS cell. The notebooks pin a new model by installing and importing in one cell, and the shim writes that marker from a child process partway through the very cell the hook has already returned from, so the import landed on the base transformers and every later cell was answered with "transformers already imported; cannot switch". Reproduced end to end against a real IPython shell with fake sidecars: as shipped the combined cell resolved to the base venv and the same notebook split across two cells resolved to the sidecar; with the fix both resolve to the sidecar. Thirteen notebooks in unslothai/notebooks have exactly that shape, all in the Muse Glimmer family, and they are baked into the image and opened in JupyterLab. The headless unsloth-run path was never exposed: it front-loads the scan over the whole notebook and puts the sidecar on the kernel process's PYTHONPATH before the kernel starts, so cell order cannot matter there. The scan itself is moved rather than copied. unsloth_nb_compat is the file the image installs into site-packages, so it is the only one a kernel can import; unsloth_run now reads the same regexes and helpers back out of it. A second copy would let the headless kernel and the in-notebook hook disagree about what an install line is and pick different sidecars. That reuse is what keeps the dead code rules intact here: a commented-out install, a mention inside a triple-quoted block and a bare string literal must not supply a pin, because a wrong sidecar is unrecoverable once transformers is imported while no sidecar merely leaves the base venv. Also guarded the PYTHONPATH prepend in activate the way the sys.path insert next to it already was. The hook runs on every cell until transformers is imported, so the unconditional prepend added one copy of the sidecar per cell and every child process inherited the pile. tests/python/test_docker_nb_compat_pre_run_cell.py is new and the hook had no coverage at all before: the combined cell, six shapes that install nothing, six real install shapes, the marker still winning, silence once transformers is imported, the PYTHONPATH count, a premise pin that IPython really supplies info.raw_cell, and an identity check that unsloth_run holds the same function objects. Restoring the old hook body turns seven of them red. The first test in the file also failed on its own until the fixture dropped transformers from sys.modules at setup: the session had already imported it, and each test was otherwise passing on the teardown of the one before it. One thing this does not address, because it is a separate gap: the image bakes sidecars for 4.57.6, 5.3.0, 5.5.0 and 5.10.2, while those thirteen notebooks pin 5.15.0, which resolves to no sidecar at all and then asserts its own version. * Hand the categorized view directories to the host user too The view is built as root and its default location is /workspace/Unsloth Notebooks, directly under the host bind mount, so os.makedirs left the view root and every "NN Section" folder root:root with the usual 0755. The host user could then neither drop their own link into a category nor delete the generated view without escalating. This is the sibling site of a fix already made twice. mkdir(2) gives the new directory the caller's uid and only setgid carries down, which is why unsloth_sync_notebooks.sh grew mkdir_keep_owner and unsloth_run has _makedirs_as_host; the shell helper's own comment even says "a new category folder would land root:root", but the category folders are created here, in Python, and that site was missed. Imported unsloth_run's helper rather than writing a third copy of the same walk. __file__ is the /usr/local/bin/unsloth-nb-view symlink in the image, so the directory is derived from the resolved path, and the sys.path entry is removed again afterwards. A test asserts the helper resolves and comes from docker/unsloth_run.py, so a broken import fails CI instead of silently degrading back to root-owned folders. The symlinks inside those folders stay root-owned and that is fine: permission to add or remove a name comes from write and execute on the directory, not from owning the entry, and the view root is not sticky. Tests record os.chown rather than inspecting st_uid, because the suite runs unprivileged where chowning to the anchor's own uid is a no-op and invisible in the result. Covered: every directory the run creates is re-owned to the nearest existing ancestor, a directory that already existed is left alone, and the helper is the shared one. Reverting to plain os.makedirs turns the first red. * Flatten only the install commands, not the whole cell around them Install-cell detection was a substring search over the entire cell text, so any code cell that merely mentioned "pip install" was classified as one and then run through _normalize_install, which strips comments and collapses intra-line whitespace. A false SAME is sticky: the refresh keeps the old file and re-records its old hash, and reports it as "kept (only header/footer changed upstream)". Measured against the real corpus at unslothai/notebooks, 561 notebooks and 11448 code cells, using the shipped functions: 725 cells classified as install cells 4 of them contain no install invocation at all, only a prose comment saying `uv pip install` (the NeMo-Gym family, each a 200-line executable Python cell that clones a repo, writes a YAML config and spawns a server) 555 of them, 76 percent, carry executable Python or shell besides the install The four are not hypothetical. NeMo-Gym-Multi-Environment.ipynb cell 8 builds a YAML config out of string literals, and collapsing intra-line whitespace flattens its nesting levels to one space each. Re-nesting a key in the real file, two different ways, still hashes SAME today; both are DIFF after this change. Detection is now per line: an install command is a `!` line, a %pip, %uv or %conda magic, or any line of a %%bash cell. Normalization follows the same rule. A non-install line keeps its bytes unless it is blank or a whole-line comment, or it contains no quote and no `#` at all and therefore has nowhere for whitespace to be data. That last clause is what keeps `import os, re` cosmetic, which the existing indentation-churn test pins. All ten existing tests in this file pass unchanged. Added: the comment-only mention is not an install cell, YAML nesting inside a string literal is not cosmetic, a `#` and a run of spaces inside a string literal beside a real install are both compared, a %%bash cell still treats every line as a command, six real invocation shapes still classify, and four mentions do not. Eight of them go red against the previous implementation. One thing deliberately left alone, with a test saying so: "pip3 install" is not in _INSTALL_MARKERS and was not before. That miss errs the safe way, since an unrecognised install cell is compared byte for byte and costs one extra refresh rather than hiding a spec change, so widening the marker set is its own decision. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the managed set across a populate retry, and drop what upstream removed Two state-file defects in the notebook sync, both of which silently and permanently un-manage a notebook while the commit marker is stamped anyway, so the tree looks converged. First, the retry branch rebuilt the state from the baked template alone. Between a partial populate and the next start the refresh publishes newer bytes for template files and adds notebooks that exist only upstream, and the populate loop reads both as user files: the newer ones differ from the template so they take the "kept existing user file" branch, and the upstream-only ones are never walked at all. Reproduced by blocking one path with a plain file where a directory has to be, exactly as the existing retry test does: after the second boot the records for the refreshed file and the upstream-only notebook were gone, the files stayed on disk, and every later refresh reported them as "kept (your edits)". The retry now merges the records it did not produce, keyed on the path, so its own entry always wins. Inert on a true first boot, where there is no state to carry. A partial populate is not exotic. First boot copies 208 MB and 1241 files into the bind mount, so ENOSPC lands squarely in the middle of it, and the state file is written by a same-filesystem rename that still succeeds on a full disk. One unwritable path under the mount is worse, because the partial marker then never clears and the truncation repeats on every boot. Second, the refresh walks only the clone, so a notebook deleted or renamed upstream is never visited, and the state file is then replaced by what that walk recorded. The published copy stays on disk with no record, the next refresh reads it as user-owned, and the categorized view lists it under "Other Notebooks" indefinitely; a rename leaves a duplicate pair. Over the last year upstream deleted 10 and renamed 7 notebooks under nb/, and the sync mirrors the whole repository, so 128 files in total. Prior state entries absent from the clone are now reconciled. The reconcile deletes, so it is fenced. Only a file that still hashes to what the sync itself wrote is removed, which is what keeps an edit safe; verified with a user-edited copy of a notebook upstream deleted, which survives while its pristine sibling goes. A case-only rename is skipped, because the clone is case-sensitive while the destination may be a macOS or Windows bind mount where the old path resolves to the file just published. rmdir is one level and never $DEST, since rmdir -p would climb out of it. UNSLOTH_KEEP_REMOVED_NOTEBOOKS=1 turns it off, and the summary line now reports the count. The re-arm at the end is widened to cover removals: the categorized view is built before the child forks, so a deletion with no accompanying update would otherwise leave a link to a notebook that is gone. test_the_child_re_arms_the_finalize_ only_after_it_copies is renamed and rewritten for that, with a sibling asserting the re-arm is still conditional at all. Six new tests. Removing either fix turns exactly one of them red. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let the cell's own pin outrank the marker, and bound a URL fetch Two things on the notebook helpers. The pre_run_cell hook consulted the marker before the cell it is about to run. The marker records an install that has ALREADY happened, so it is the weaker of the two: within one notebook a later cell can pin a different transformers version while the marker still holds the earlier one, and the marker path falls back to pid-<pid> when the ipykernel connection file cannot be read, with nothing ever sweeping /tmp/unsloth_nb, so a recycled pid inherits a stranger's pin. The cell about to run is the better authority in both cases. A cell that pins nothing of its own still falls back to the marker, which is the shape the hook has always handled, and that is asserted rather than assumed. unsloth-run fetched a URL with no timeout at all. --timeout is nbconvert's whole-notebook budget, an hour by default, and it never reached the download, so a host that accepted the connection and then went quiet hung the run before a single cell executed. There is now a --fetch-timeout, defaulting to 60 seconds and to UNSLOTH_NOTEBOOK_FETCH_TIMEOUT if set, which is the same knob and the same default the notebook sync script already uses for its network operations. This bounds each blocking socket operation rather than the total transfer, which is what the stalled-host case needs; a server trickling bytes forever is a different problem and the docstring says so rather than implying it is covered. Four tests. The default and the override both reach urlopen, the stall limit is asserted to be separate from and smaller than the execution budget so the two cannot be collapsed later, a local path still opens with no timeout argument, and the stale-marker case picks the cell's sidecar rather than the marker's. Dropping either fix turns one of them red. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record each cleaned notebook before publishing it The Colab-intro migration published every notebook and then wrote the state file once at the end, inside a bare `except OSError: pass`, with the success message outside the try. Lose that single write and every cleaned notebook no longer matches its record. The refresh compares the byte hash first and continues before the body-aware comparison is ever reached, carrying the stale record forward, so those notebooks are read as user edits and upstream updates stop reaching them permanently, while the migration prints "cleaned N notebook(s)" and exits 0. The caller discards the exit code anyway. Measured on the real corpus, 445 notebooks staged with their recorded hashes, running unprivileged with the barrier asserted to be real before the run: with the destination root unwritable the shipped migration reports "cleaned 445", exits 0, and leaves 445 orphaned records. Driving the sync script afterwards over two upstream releases shows the frozen arm reporting "1 kept (your edits)" and never receiving either fix, while the arm with a correct record takes both. It does not self-heal: re-running the migration skips those notebooks because their hash no longer matches, so nothing ever puts it right. The trigger does not need a disk to fill. The whole set is cleaned in one pass on the first boot after this ships, which is exactly when a docker stop or an OOM kill is most likely, and a SIGKILL in that window strands all 445 with no code left running to report anything. That variant is why failing loudly is not the fix; only the ordering is. So the record now goes down first, per notebook: stage the cleaned copy, write the state, then publish. If the state write fails nothing has been published and the disk still matches its record, so the next start simply tries again. The inverted window is far narrower and self-correcting: a publish that fails costs one notebook, and its record is put back immediately. _write_state also fsyncs before the rename, since a rename made visible with its content unwritten strands the notebooks the same way. strip_notebook is split into _stage_clean and _publish to make that possible and keeps its own behaviour, including the JupyterLab race guard, unchanged; the 15 existing tests pass untouched. Seven new ones: the happy path records what it published, a failing state write publishes nothing and leaves no staged copy, a write that fails partway strands nothing and the rest is cleaned on the next start, a failed publish gives its record back, the fsync precedes the rename, and malformed or unmanaged state lines survive verbatim. Restoring the old ordering turns three of them red. Not fixed here, and worth saying plainly: the same swallow-and-carry-on shape is in unsloth_sync_notebooks.sh at the four state writes. Its blast radius is only the notebooks one refresh actually changed rather than the whole set on first boot, and the fix there is a larger job than this one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reconcile the four windows the notebook sync round found Recover a publish interrupted between its record and its rename. Recording the cleaned hash before publishing closed the window that stranded the whole notebook set at once, but it left one of its own: a stop between os.replace on the state file and os.replace on the notebook leaves the record describing the cleaned bytes while the file still holds the pristine ones, and every later run reads that as a user edit and carries the stale record forward for good. _resume tells the two apart by cleaning what is on disk, because only the interrupted case reproduces the record byte for byte. Keep the record of an upstream removal that could not be unlinked. A writable single-file bind mount returns EBUSY, the case the publish path already works around for rename. The failure was ignored, so the file stayed while its record was dropped and the sync marker was still stamped: the copy became unmanaged and no later refresh retried it. The record is now carried forward and the operation counted as failed, which is what holds the marker back for the retry. Require an install marker in a shell cell to open a command. %%bash qualified every line that merely contained one, so msg="pip install foo # bar" counted and the normalization then stripped the quoted comment and collapsed the spacing, both of which are data in shell. Editing that string upstream produced SAME and the container kept running the old commands. Quoted stretches are blanked and the marker has to start a command; all 1064 marker lines in the 152 shell cells of the 561 shipped notebooks already do, so no digest changes today. Match the transformers pin the way pip and the shim match a requirement name. The scanner was case-sensitive and extras-unaware while unsloth_pip_shim canonicalises before it decides what to drop, so Transformers==5.5.0 and transformers[torch]==5.5.0 left the pin unseen with the install still suppressed, and the import in that same cell froze the base version for the life of the kernel. The same pattern also stops sentence-transformers==5.5.0 from being read as the transformers pin. unsloth_run now calls the shared helper rather than the regex, so the two halves of the scan cannot disagree about the name either. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-fetch a notebook the baked restore walked backwards Phase 1b copies a deleted notebook back from the baked template and records the template's hash for it. When the refresh had already carried that notebook past the image, the restore silently returns it to the older version, and phase 2 exits on remote == last, so it stayed downgraded until upstream happened to commit again. The marker now comes off, but only when a restore actually moved a notebook backwards: the recorded hash is compared with what the template just wrote, so an ordinary delete of a notebook that never moved past the image does not cost a full clone on the next start. The marker is re-stamped as soon as the refresh succeeds. This mirrors the populate arm, which already drops the marker when it could not finish, for the same reason. Also name an encoding on the two marker writes in the pre-run-cell tests. The repo gate for that reads the whole test tree, and a checked-in file read with the platform default breaks on Windows the moment it gains a non-ASCII byte. * Filter requirements by logical line and forward extras of baked packages Two defects in the notebook pip shim, both reachable from an ordinary cell. Filtering walked physical lines, but pip and uv join a trailing backslash into one logical requirement, which is exactly the shape pip-compile and uv pip compile emit with --generate-hashes: "torch==2.11.0 \" followed by indented --hash rows. Dropping a protected requirement removed only its first row and published the orphaned --hash rows. pip warns and ignores them, silently disabling hash checking when the protected package was the only hashed entry, and uv rejects the file outright with "Unexpected '-', expected '-c', '-e', '-r' or the start of a requirement", killing the whole install. Group the physical lines into the logical lines pip parses, so a dropped requirement takes its continuation rows with it and a kept one keeps every hash row. Multi-line includes stay verbatim. The console line now reports the requirement rather than the joined hash blob. Extras of an already-baked package were dropped whole, so pip install "unsloth[studio]" on the core image was a successful no-op and ddgs, boto3 and fastmcp never arrived. pip and uv treat pkg[extra] against a satisfied pkg as "add the optional dependencies", not "replace pkg", and _protected_constraints_file already pins the baked version, so forwarding name[extras] without its version specifier can only add. A direct reference or a path is a replacement request and keeps being dropped. Regression tests cover the hashed continuation rows, the transformers marker under hashes, the extras forms including the env-marker and pinned ones, and a guard that the git+ direct reference stays dropped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Authenticate the Docker Hub publish with DOCKER_API_KEY and publish daily The four docker/login-action steps read secrets.DOCKERHUB_USERNAME and secrets.DOCKERHUB_TOKEN, and neither secret exists on the repository, so every login would have failed and nothing could ever publish. The account token is DOCKER_API_KEY. The username is not a secret, so it goes in the top-level env block next to REGISTRY and IMAGE_NAME and is referenced by all four steps, which keeps them from drifting apart. Only the secret reference appears in the workflow; the value is never echoed or written anywhere. Also move the schedule from weekly to daily at 18:17 UTC. Actions cron is UTC and does not follow daylight saving, so that is 11:17 in San Francisco under PDT and becomes 10:17 under PST; no single expression holds 11:00 local all year, and the comment says so rather than implying the time is fixed. The minute stays off the hour because runs scheduled on the hour queue behind the platform-wide peak and can be dropped. * Create the notebook root through the ownership-preserving helper mkdir -p "$DEST" ran before any helper was defined, so on first boot the notebook root was created root:root. That root is the anchor everything else inherits from: mkdir_keep_owner walks up to the nearest existing ancestor, and own_like_dir copies the owner of the directory a file lands in. Under a host-owned bind mount, with UNSLOTH_NOTEBOOKS_DIR pointed at the mount, every category folder and every notebook underneath then took root ownership and the host user could not edit or delete their own files. The Python twin, unsloth_run.py _makedirs_as_host, has always chowned the leaf it creates. Create the root with mkdir_keep_owner instead, at the first point after the helper is defined. Only function definitions sit between the old and the new call site, so nothing that runs in between depended on the directory existing, and the helper calls mkdir -p internally, so parents are still created. The old "cannot create, give up quietly" behaviour is kept as an explicit directory test. The guard test that was supposed to catch a bare mkdir scanned only the text after the helper, which is exactly why this site slipped through, so it now scans the whole file. It also counted a bare substring, so a comment naming the helper was counted as a call site; it now counts real invocations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a quoted hash as data and fail closed without protected constraints Two defects in the notebook install path, both found by review of the previous round's changes. The install-line digest stripped comments with a regex that cut at any `#` preceded by whitespace, including one inside a string literal. A compound line such as !pip install foo && python -c 'print("a # b")' && pip install bar==2.0 normalized to everything before the quoted `#`, so the trailing `bar==2.0` vanished from the digest and an upstream bump to `bar==3.0` hashed the same. The sync then kept the stale notebook and stamped the new upstream commit, which is exactly what the "a changed package spec is NOT cosmetic" branch exists to prevent. POSIX starts a comment only on a `#` that opens an unquoted token, so the regex is replaced by a quote-aware scan that keeps the existing rule that a glued `#` is data, as in a git+https fragment. An unterminated quote keeps the whole line, which can only report a difference. Digests over all 561 shipped notebooks are unchanged. _protected_constraints_file returned None both when there was nothing to pin and when it had failed, so the caller could not tell the two apart and ran the install with protection entirely off. The same mkstemp error is already fatal in _filter_requirements_file, which refuses to forward a requirements file it cannot filter. This matters more since extras forwarding landed: that change forwards name[extras] with the version specifier stripped precisely because the pin holds the baked version, so an unconstrained forward of, say, pip install -U unsloth[colab] can replace the baked package rather than only add to it. Enumeration and write failures now abort; a package whose metadata cannot be read is skipped rather than dropping every other pin, because a half-removed dist-info left by an interrupted install is ordinary. The legitimate "nothing installed to protect" case still returns None. Regression tests cover the quoted hash, a pin hidden behind one, a genuine trailing comment staying cosmetic, the two abort paths, and the unreadable distribution. * Warn instead of silently dropping an unwritable transformers marker The transformers requirement is removed from the real pip arguments before the marker is written, so an unwritable marker path left a transformers-only install printing the "nothing to install after keeping the baked stack; ok." no-op while the model cells went on importing the baked version. The request was lost with no indication anywhere that the pin had not been honoured. The install is still not aborted. Nothing has been installed or mis-forwarded at that point and the baked transformers keeps working, so failing a notebook cell over a read-only path or a full /tmp would be the worse trade for the user. What changes is that the failure is now reported on stderr and the success message is no longer printed when the write failed. Also fixes a latent bug on the same path: os.path.dirname(MARKER) is "" when UNSLOTH_NB_TF_MARKER is a bare relative filename, and os.makedirs("") raises FileNotFoundError, an OSError that the previous blanket handler swallowed. That configuration silently lost the marker on every run; it now records normally. Both paths get a regression test. Verified they fail against the previous swallow and pass with the fix. * Keep the baked IPython profile writable for non-root kernels The profile was built as root and left read-only, so under the documented docker run --user <uid> mode IPython did not merely lose history.sqlite. IPython.paths.get_ipython_dir() checks the directory with os.access W_OK and, when it cannot write, warns once on stderr and substitutes tempfile.mkdtemp(). A fresh temp directory has no profile_default/startup, so the startup hook never ran: no transformers sidecar activation, no %pip or %uv magic, no colab-compat and no shim marker. The notebook still executed, on the baked transformers, with nothing in its output to say the pin had been ignored. Both claims in the old comment were wrong: the hook did not load under --user, and writable state did not land per-user. Making only the top directory writable is worse. ProfileDir.check_dirs then creates security, log and pid inside profile_default and raises PermissionError, which kills the kernel before it replies to kernel_info. So both directories get the mode, and the sticky bit keeps the shared hook safe: 00-unsloth-nb.py and its startup directory stay read-only and no other uid can unlink or rename them. The regression test replays the Dockerfile's own mkdir, cp and chmod steps into a temp tree rather than asserting a hardcoded mode, remaps each path's other bits onto its owner bits so os.access answers the question a foreign uid would ask, and then runs the installed IPython to see which directory it really picks and whether the hook is in it. * Publish the refresh state beside the notebooks it describes The refresh staged its new state file with mktemp, so publishing it moved across filesystems from /tmp into $DEST. That move can fail on its own, after the notebooks have already been written, and the failure was discarded: failed stayed at zero and the commit marker was stamped with the new upstream revision. The next start then exits on remote == last, and once upstream finally moves, the stale hashes no longer match the notebooks that run wrote, so every one of them reads as user-edited and is kept forever. The tree stops updating for good. Holding the marker back is necessary but not sufficient, which the test for it showed: the notebooks are published before the state is, so once a publish has failed on its own the disk is already ahead of the record and no marker check undoes that. So the temp file moves beside $STATE, matching the two state writers above it, which do the same and whose temp name the notebook scan already skips. A sibling makes the publish an atomic same-directory rename that cannot half-succeed, and it makes the state fail together with the notebooks rather than independently: a $DEST that is full or read-only fails the notebook writes too, and those already count into failed and hold the marker. A publish that fails anyway now also counts, so the marker is held and the next start retries instead of exiting early. Two regression tests: the marker is not stamped when the publish is refused, and the state is not staged outside $DEST. * Fix four defects the tenth review round found Refresh state staging had a /tmp fallback that fired in exactly the case the sibling temp file was introduced to prevent. With $DEST unwritable but a notebook subdirectory still writable, which is the container shape since the image bakes the root and only the subdirs are chowned to the host user, staging fell back to /tmp, the new notebook bytes were published, and the cross-device move of the state then failed. The marker is withheld on that run, so it is not the hole; the next run is. It compares the new bytes against the old hashes and classifies every managed notebook as a user edit, and a user edit counts as kept rather than failed, so the marker IS stamped and the whole set is stranded for good. Nothing is published at that point, so the refresh now bails and leaves the tree untouched, matching record_state's abort above it and the mid-clone bail below it. unsloth-run staged a local notebook inside the --out directory before executing it. nbconvert makes the input notebook's directory the kernel working directory, so the notebook's relative opens, local imports and saves resolved against the output tree instead of the source tree. Local notebooks now execute where they live and only the result is staged beside --out. A URL has no source tree, so that branch keeps staging the download next to the output. Note the consequence: the source tree is now written to, so a run whose notebook writes relative paths will drop them beside the notebook, and a read-only source tree that previously worked will now fail. build.sh passed its default mutable "main" refs straight to docker. Docker matches a RUN layer on the command string alone, so the pip-install layer was a cache hit on every rebuild and the image kept the commits of the first build while reporting success. Both refs are frozen with git ls-remote first, the way the publish workflow and the llama.cpp tag resolution already do. An unreachable remote warns and passes the name through rather than failing before docker starts. The EXDEV copy fallback in activate_staged_dir dereferenced symlinks, because shutil.copytree defaults to symlinks=False, so each soname alias was written out as a second full copy of its library and the loader's link topology was replaced by independent files. The two other copies in that file already pass symlinks=True. Measured on the linux-x64-cuda12 bundle this branch exists for: 5 symlinks, 13.1 MiB duplicated. The 358 MB libggml-cuda.so is a regular file and is copied once either way. Each fix has a regression test that fails against the previous behaviour. * Clear a stale state temp file before bailing on an unstageable one The abort added a commit ago is correct but was too absolute. A run killed between the truncate and the mv leaves $STATE.tmp behind, and if that run was a different uid, root once and then --user, the leftover cannot be truncated even though $DEST is perfectly writable. The refresh then bailed on every subsequent start and the tree never updated again, which is the same permanent staleness the bail exists to prevent. Unlink it first. That needs write on $DEST, which the case the bail is really for does not have, so it cannot weaken the guard: an unwritable $DEST still fails the truncate and still bails before publishing anything. Verified both ways: a stale unwritable temp file now refreshes normally, and the unwritable-$DEST case still publishes nothing and holds the marker. * Docker: fix stale state staging on first boot, forward transformers extras, and let :latest start without an NVIDIA GPU unsloth_sync_notebooks.sh: the initial-population path had the same stale $STATE.tmp hole that the refresh child already guards against. A first boot killed between the truncate and the mv leaves the temp file behind; if that run was a different uid the truncate cannot empty it, the state appends fail one by one while the loop keeps going, and the mv publishes the foreign file as our state. Every notebook that run copied is then missing from it, so the next start reads them as user edits and never refreshes them, and the commit marker is stamped anyway because no copy failed. Unlink before truncating, and if staging still fails, bail before copying rather than publish notebooks we cannot record. unsloth_pip_shim.py: a transformers target dropped its extras. transformers [deepspeed]==5.5.0 recorded the version and discarded the whole token, so the install reported a successful no-op and deepspeed was never installed; transformers[torch] was silent altogether. The transformers branch ran before the extras forwarding every other protected distribution gets. Both the CLI and the requirements-file path now forward name[extras] while still suppressing the version replacement. Dockerfile.studio: default UNSLOTH_ALLOW_CPU=1. This image publishes as :latest, so a plain docker run on a CPU-only, AMD or Docker-Desktop host would exit 1 from the entrypoint before Studio started. Studio chat over GGUF, Data Recipes, JupyterLab and the GGUF tooling need no GPU. The entrypoint only takes the CPU path when no GPU is visible, so a GPU host is unaffected, and -e UNSLOTH_ALLOW_CPU=0 restores the strict check. The base training image keeps it strict: FastLanguageModel really does need a GPU. run.sh: stop failing at the daemon. Docker resolves --gpus before the container exists, so on a host with no NVIDIA GPU it dies with "failed to discover GPU vendor from CDI" and exit 125, and the entrypoint never runs to explain what is wrong. Drop the flag when no NVIDIA GPU is present and let the container start. On an AMD host also pass /dev/kfd and /dev/dri through for llama.cpp, with numeric gids because --group-add resolves names inside the container. This is not AMD acceleration: torch in the image is cu128 and torch.cuda.is_available() stays False there. * docker/run.sh: do not abort the AMD fallback when a group record is missing getent exits nonzero for a group that is not in NSS. Under set -o pipefail that status propagates out of the command substitution, and set -e then takes the whole script down before docker run is reached, so the AMD device fallback added in the previous commit could never start on a host without a render group. Minimal images really do ship without one. Trailing || _gid= puts the assignment in an OR-list, which leaves the gid empty and keeps whichever gids do resolve. An empty but successful lookup was already safe, so only the nonzero case was affected. The existing command -v guard still covers a host with no getent. The AMD test passed only because the machine it ran on has both video and render, which is the same host dependence that made the nvidia-smi probe pass vacuously earlier. getent is now stubbed there too, so the missing-group cases are reachable: both new cases fail without this change. * docker/unsloth_pip_shim.py: scope the protected pins to the base venv The protected constraints file and the installed-name scan both called importlib.metadata.distributions() with no path, so they walked sys.path in order. When a notebook activates a transformers sidecar, unsloth_nb_compat and unsloth_run prepend that sidecar to PYTHONPATH, and the sidecars are built with uv pip install --target, so each one carries a real transformers-X.dist-info. The scan therefore found the sidecar's transformers first and the first-wins loop pinned that version instead of the baked one. The pin is what stops the resolver moving the baked stack, and _extras_only_target leans on it directly: it forwards a protected target with the version specifier stripped precisely because the pin holds the baked version. Pinning the sidecar version instead means an install run from a sidecar-activated kernel could move the shared base transformers to it, for every later kernel, not just the one that asked. Both scans now take the site-packages of the venv that owns the real pip. The fallback is sysconfig purelib, which is safe for the same reason the fix is needed: it derives from sys.prefix, which PYTHONPATH cannot influence, so it can never name a sidecar. The distributions stubs in the tests took no arguments. One of them made test_protected_metadata_enumeration_failure_refuses_install pass for the wrong reason once the call had a keyword: the TypeError was caught by the same except that handles a real enumeration failure, so the test asserted SystemExit without ever reaching the code under test. * docker: count failed state appends, and give every unsloth-run its own marker unsloth_sync_notebooks.sh: staging the temp state proved DEST was writable once, not that it stays so. A quota or ENOSPC landing on a post-copy append left the notebook on disk with no record while populate_failed stayed 0, so the partial state was published and the commit marker stamped. An unrecorded file is read as a user edit, so those notebooks were never refreshed again. Only cp failures were counted; the appends had no failure path at all. Every append is now checked. In the populate loop a failure counts exactly like a failed copy, which keeps the partial flag and leaves the marker off so the next start retries. The refresh loop had the same shape at two more sites, where the previous state is still valid, so it abandons the rewrite and keeps the old state rather than publishing a truncated one, and drops the marker to force a retry. unsloth_run.py: always allocate a fresh per-run marker instead of honouring an inherited one. The IPython startup hook gives every kernel its own UNSLOTH_NB_TF_MARKER, so unsloth-run launched from a notebook cell inherited the caller's. That broke both ways: a target with a pin overwrote the calling kernel's pin, and a target with no pin ran against the caller's stale pin. Either way a kernel that has not imported transformers yet can be handed the wrong sidecar. The makedirs is gone with it, since mkstemp's directory always exists. record_state() has the same unchecked append but no caller anywhere in the tree, so it is left alone here. * CI: log in to Docker Hub as the unsloth org, not a personal account DOCKER_API_KEY is now an organization access token for the unsloth org rather than a personal access token. Docker Hub authenticates an OAT as the organisation, not as whoever created it, so REGISTRY_USERNAME has to be the org name. Left as a person's account it fails authentication on every login. The comment said only that this was the account the token belongs to, which reads like an arbitrary owner field and invites someone to put a person back there. It now states the rule. All four login steps already resolve the single top-level env var, so this is the only place the username appears; docker/ does not reference it and no test asserts on the old value. * docker/unsloth_pip_shim.py: intercept only the uv pip install command path The shim selected the first argv token equal to install. pip's command path is pip [opts] install, but uv's is uv [opts] pip [opts] install, and uv has other subcommands ending in install, so those were claimed too. The damage was not limited to a wasted rewrite. uv python install 3.13 and uv tool install ruff had the protected --constraint appended, which neither accepts, so both failed. Worse, uv tool install transformers matched the protected-package filter, was reduced to nothing, and reported ok, so no tool was installed at all and nothing said so. For uv the nearest preceding non-option token must now be pip. Scanning back over options keeps uv pip --quiet install and uv --directory X pip install working, and pip itself is unchanged. * docker: locate uv's install subcommand past option values, and check the refresh state appends unsloth_pip_shim.py: the previous commit narrowed interception to uv's pip command path by stepping back from install over anything starting with a dash. uv accepts its global options between pip and install, so uv pip --directory /tmp install torch==9.9 stopped at /tmp and the command was not recognised at all. That is the worse direction of the same bug: the install then ran with no filtering and no protected constraints, free to replace the baked torch and CUDA stack. The command path is now matched positionally, install for pip and pip install for uv, skipping option values using _VALUE_FLAGS. That is the same set the tail scanner uses and _selfcheck_value_flags holds it to the real CLIs at build time, so it stays correct as uv adds options. The --flag=value form carries its own value and does not consume the next token. unsloth_sync_notebooks.sh: the refresh publish loop had the same unchecked appends already fixed in the populate and restore loops. A lost record there left failed at 0, so the truncated state was published and the commit marker advanced, which is exactly the wedge the comment at the publish step describes: the next boot exits on remote equals last, and once upstream moves, every notebook whose hash was dropped reads as user-edited and is kept forever. The five sites that did not already count a failure now go through a helper that does. This is the fourth place in this script where a staged-state append was unchecked, so the tests now assert the property rather than the instances: every append to a staged state file must handle its own failure or have one counted immediately after. record_state is exempt because it has no caller, and a companion test fails if it ever gains one, since its loop is a pipeline and a counter set inside would not survive the subshell. The race test that pinned the literal printf for the staged hash now accepts the helper too; what it guards, that the recorded value is the staged hash rather than a re-read of the destination, is unchanged. * docker/unsloth_sync_notebooks.sh: roll back a notebook that cannot be recorded Counting a failed state append held the commit marker back but still published the truncated state, so the one unrecoverable combination survived: a notebook written to disk with no record. The next refresh reads it as a user edit, keeps it, sees nothing fail, and stamps the marker over it, and it is never updated again. Reproduced end to end with a local upstream and RLIMIT_FSIZE filling the disk part-way through a refresh of 3000 notebooks: run 1 3000 published, 2926 recorded, marker held back run 2 75 kept (your edits), no failures, marker advanced those 75 are unmanaged from then on Discarding the staged state instead would be worse, since every notebook the run did record would go stale in the same way. Instead the notebook is removed when its record cannot be written. That branch is only reached when the destination was absent or hashed exactly to our own previous record, so it never holds a user edit, and the next refresh re-copies and re-records it. The unlink also returns the space that just ran out. After the fix the same scenario ends with run 2 reporting 3000 updated, 0 kept, and a complete state. The test asserts the property rather than the numbers: no notebook may be on disk without a state record, and the retry must restore everything rolled back. Sizing the failure window needs care, which is why the previous round could not demonstrate this path: git's index costs ceil((62 + len(path)) / 8) * 8 per entry while the state costs 67 + len(path), so only a path of 17 characters or more lets the state outgrow the index and leaves a cap that admits the clone but not the whole state. * docker: handle every failed state record, and freeze the notebooks ref in build.sh unsloth_sync_notebooks.sh: three state appends were left raw last round on the grounds that they already counted a failure. That reasoning was wrong. Counting withholds the commit marker for one run only; the truncated state is published either way, so the next refresh finds no entry, reads the notebook as a user edit, keeps it, sees nothing fail and stamps the marker over it. The file is unmanaged from then on, which is the same permanent outcome the publish branch was fixed for. Rather than patch two more sites, record_tmpstate now only reports the status and every caller has to handle it. A failed record rolls the notebook back through drop_unrecordable, which removes it only while it still hashes to the last record, so it can never touch a user edit; the next refresh re-derives it, and the unlink returns the space that ran out. The publish branch keeps its own removal because there the file holds the new bytes rather than the recorded ones. The guard is generalised with it. The class that keeps recurring is not the raw append but the unhandled record failure, so the tests now require every record_tmpstate call to handle a failed write, and a behavioural test covers the part that cannot be asserted structurally: a user-edited notebook is never rolled back. build.sh: freeze the notebooks ref too. It already resolves unsloth, unsloth-zoo and llama.cpp for exactly this reason, and the publish workflow resolves this one as well, but the local script never passed it, so the Dockerfile default of main was byte-identical on every rebuild and docker served the cached clone layer. A rebuild after unslothai/notebooks moved silently kept the old set and stamped the old commit into .unsloth_template_commit, so the image also misreported which set it carried. The runtime refresh usually corrects this, but not when it is skipped or the container is offline, which is the case the baked copy exists for. * docker: fix the section 1b state writer, short-option clusters, uv pip sync and BOMs unsloth_sync_notebooks.sh section 1b never got the fix bee9e5a52 applied to the other state writer. It staged through mktemp rather than beside $STATE, which meant /tmp: a different filesystem from $DEST under the documented bind mount, so publishing was a cross-device copy that can leave a half-written state rather than an atomic same-directory rename. mktemp also creates 0600 owned by whoever booted, so this was the one writer that re-owned and downgraded $STATE, undoing what own_like_dir and mkdir_keep_owner exist to preserve. That downgrade produces the input that breaks it. There is no set -e, so an unreadable $STATE makes the loop redirect fail, the body never runs, rs_ok stays 1 and an EMPTY state is published over a valid one. Reproduced end to end: 20 of 20 recorded, then 0 records, then "1 updated, 20 kept (your edits)" with the marker advanced, so every notebook is unmanaged from then on, silently, exit 0. Section 1b rewrites the whole state on every boot, so this is not a rare path. It now stages beside $STATE, refuses to replace a state it cannot read, and its publish failure is loud and drops the marker like the sibling branch three lines below, which was already doing all three. unsloth_pip_shim.py: short-option clusters reached no handler. Only tok[:2] was tested against the attached-value flags and the remaining comparisons are exact, so `pip install -qr reqs.txt` forwarded the ORIGINAL requirements file, unfiltered and with nothing recorded, and -I hid inside a cluster and escaped the reinstall handling. Clusters are now expanded the way both real tools read them, with a value-taking flag consuming the rest of the cluster or the next token. uv pip sync is refused rather than forwarded. It uninstalls every package absent from the requirements file, so unlike install there is nothing to strip: keeping a protected package out of the file is exactly what deletes it, and --constraint bounds versions rather than preventing removals. Requirements files are read as utf-8-sig. A BOM left "transformers==X" on line 1 matching no handler, so a file whose only protected pin was first forwarded unchanged. Both real tools strip it and honour the line. The user-edit rollback test was passing without executing the code it named. Its edited set was every twenty-fifth notebook while the cap only bites near the end of the walk, so no edited file was ever a rollback candidate: instrumenting the suite recorded zero calls to drop_unrecordable and zero to unrecorded, and a mutant making the rollback an unconditional rm -rf still passed. It now edits every second notebook, and asserts that some notebook really was rolled back, so it cannot go quiet again. The neighbouring "nothing was rolled back" guard was also vacuous, since the run yields the same count either way. * docker: guard the refresh child's state reader, and five more of the same class The unreadable-state guard added last round covered the section 1b reader and not the refresh child's, which is the one that runs in the default configuration. The child's redirect fails, LAST stays empty, every notebook is counted kept rather than recorded, and TMPSTATE publishes empty with failed still 0, so the marker is stamped. Section 1b's own recovery drops the marker, which guarantees the child does not exit early and walks straight into it. Reproduced: 3 recorded, then 0 updated and 3 kept, state size 0, exit 0. The test that was meant to prove this passed only because its helper sets UNSLOTH_SKIP_NOTEBOOK_REFRESH=1; the new one does not. The populate merge reader had the same -f-only gate and now requires -r. The rollback test measured the wrong rollback. Across the durability suite drop_unrecordable was called seven times and removed nothing, because every call site handed it either a user-edited file, which its hash gate declines, or a missing one. Mutating its removal to a no-op left the suite green. The unchanged branch is the reachable one that must remove, and reaching it needs notebooks that are valid JSON, since the body-aware comparison can only report SAME for those. Populate no longer publishes a staged state that lost a merge record. Those are the only records for notebooks that exist upstream but not in the baked template and nothing can re-derive them, unlike a failed copy, which the next boot re-walks from the template and which therefore still publishes. Section 1b re-tests its guard inside the loop instead of once, so a doomed rewrite stops restoring, and undoes what it did restore. Those files were absent before it ran, so removing them restores the prior tree exactly, rather than leaving baked content behind a state that describes something else. UNSLOTH_KEEP_REMOVED_NOTEBOOKS keeps the record with the file it keeps, as its KEEP_DELETED sibling already did. Dropping the record while keeping the file made the next refresh read it as a user edit, and turning the option back off never recovered it because by then it is no longer in the state. unsloth_pip_shim.py: seven pip aliases and one uv global were missing from _VALUE_FLAGS, so their separated value was read as the install target and the install forwarded with the protected package kept and no constraint at all. The selfcheck could not have caught them: it scraped --help, which prints one spelling per option, and --python-preference appears in no uv help output. It now introspects pip's own option table, checks uv at all three help levels, probes the undocumented global directly, and fails rather than reporting OK when it cannot inspect something. _is_protected's installed check gains coverage. Deleting it left the suite green, and not by omission: the shared fixture pins _installed_names to an image where every bake succeeded, which is exactly the case where the check cannot fire. The Dockerfile lets the torchcodec and non-amd64 vLLM bakes fail on purpose, so that is the wrong image to test it on. * tests: read the checked-in docker files as utf-8 Repo tests (CPU) went red on test_source_read_encoding: the five reads that test_docker_cpu_fallback.py does of run.sh, the two Dockerfiles and the entrypoint used the platform default encoding, so they break on Windows the moment any of those files gains a non-ASCII byte. The sixth open() in the file writes a temp fixture rather than reading a checked-in file, so it is correctly not flagged and is left alone. * docker: read a relative requirements file from uv's working directory uv changes directory before it resolves a relative -r or -c path, so the shim was opening the file from its own cwd, missing, and forwarding the untouched relative path. uv then chdir'd and installed a file the shim had never inspected, recording no transformers pin, so the sidecar never activated and the request failed on the injected constraints instead of working. --directory is a uv global, so it is accepted before pip, between pip and install, and after install, in both the separated and --directory=VALUE spellings; uv rejects it more than once, so there is always exactly one. It is also documented as [env: UV_WORKING_DIR=], and that variable alone relocates the file, with the flag winning when both are set. pip has no equivalent and never changes directory, so nothing changes for it. Nested includes follow, because the source directory is now derived from the resolved path. A read failure still forwards the caller's original path so uv resolves it itself, exactly as before. * docker: stop destroying notebooks the sync cannot record Two ways a failed state append could cost a notebook permanently. UNSLOTH_KEEP_REMOVED_NOTEBOOKS dropped the file when its record would not fit. Every other caller may do that, because the clone still holds a copy to republish on the next start. This one may not: upstream had deleted the file, so the copy in the destination was the last one in existence, and no retry recovered it with the option on or off. It was also the only one of the nine record_tmpstate failure handlers with no failed counter, so the truncated state and the sync marker could both be published as a success. Keep the file and count the failure. The publish rollback cannot unlink a single-FILE bind mount, which returns EBUSY there just as it does for the rename the fallback above already works around. The published bytes then stay on disk with no record, and a later refresh reads them as a user edit and stamps the marker over them, so the notebook stops tracking upstream for good. Nothing further can be appended in the run that loses the record, and rolling the bytes back would leave the file just as unrecorded, while discarding the staged state would strand every notebook the run did record. So repair it from the other side: an unrecorded file that is byte-identical to the clone was never a user edit, and is taken back into the state. The bytes on disk already are the bytes we would publish, so nothing of the user's can be lost, and a file that genuinely differs is still left alone. * docker: resolve a local project target from uv's working directory too The requirements-file fix stored uv's effective directory but only used it for `-r` and `-c` reads. A local project target is a filesystem lookup as well: _classify_flag_target goes through _canon into _local_project_name, which stats the directory and reads its pyproject.toml to learn the real project name. That lookup still resolved against our own cwd, so `uv pip --directory X install -e ./unsloth` run from elsewhere reported the protected project as unknown and forwarded it. This one is silent where the requirements-file case was loud. The injected constraints file survives the chdir and rejects a protected package, but only on a version MISMATCH: a local checkout declaring the same version as the baked one resolves cleanly and installs over it. The stat also has to happen on the path with any fragment stripped. `#egg=` is recognised earlier in _canon and never reaches the directory lookup, but every other fragment arrives intact, and leaving it attached makes the stat miss and leaks the project. That behaviour predates this change and had no test. * ci: verify the published manifest belongs to the run that pushed it The sha tags are immutable across commits, which is what the tag selection in the digest export was defending against, but they are not unique across refs. The concurrency group is keyed on the ref and, only on main, the sha, never on the inputs, so a branch push and a lightweight v* tag push at one commit sit in different groups and never wait for each other. Four of the last five v* tags in this repo are lightweight, so the tag ref is the commit and both runs emit byte-identical core-sha-<short> and sha-<short> names, about a minute apart, against builds that run for 90 and 150 minutes. The smoke probes are gated behind HAS_GPU_RUNNER and cannot currently run. The path that does run every push is the base digest export, whose result is baked into the Studio image as BASE_IMAGE and published as :latest and :studio. If the other ref retags between this run's manifest creation and its inspection, this run ships another run's base image with nothing to show for it. So both merge jobs now confirm the manifest they resolved contains the per-arch digests this run pushed, and fail if it does not. No tag choice can be made collision-proof while two refs legitimately share a short sha, so the check is the part that matters. merge-studio also exports its digest, which it never did, and both probes pull by digest instead of re-deriving a name. That made the two metadata-action steps in smoke-test dead, so they are gone, along with the requirement that their enable= expressions stay byte-identical to the merge jobs' gates. That coupling only existed because the probes re-derived tags. This workflow has no runs yet and its probes are gated off, so there is no end-to-end evidence, only the workflow-parsing tests: the digest export is driven against a stubbed docker and must refuse a manifest that does not hold this run's arches. * docker: do not filter installs aimed at another environment --target, --prefix, --root and --python send the install somewhere other than the venv this shim guards, so the reason for filtering does not apply. The shim filtered anyway: it recognised the install, dropped the protected target, found nothing left to run, printed "nothing to install after keeping the baked stack; ok." and returned 0 having installed nothing into the environment the caller named. Verified against the real tools rather than their help text: `pip --python <venv>/bin/python install idna==3.6` puts 3.6 in that venv and leaves the base at 3.18, and --target writes into the target directory. The destination also moves with nothing on the command line. pip honours PIP_<OPTION> for every option and uv documents --python as [env: UV_PYTHON=]; PIP_TARGET redirected a plain `pip install` and UV_PYTHON redirected a plain `uv pip install` in the same way, so PIP_TARGET, PIP_PREFIX, PIP_ROOT, UV_PYTHON and PIP_PYTHON are all recognised. The bypass is one-sided on purpose. It fires only for a destination that positively resolves outside the guarded venv, because getting this wrong in the permissive direction hands over the exact thing the shim exists to prevent. A bare `--python python3` or `--python 3.12` may name the baked interpreter, and a --target inside the venv is still the venv, so both keep filtering. It sits ahead of the `uv pip sync` refusal too: pointed at another environment there is no baked stack for sync to strip. One existing case moved with the behaviour. The subcommand-search test used `uv pip --python /usr/bin/python3 install snac` and asserted the protected constraints were injected; that install is now deliberately bypassed. The test is about not stopping at a flag's value, and the interpreter was incidental, so it now points at /opt/unsloth-venv/bin/python: same parser path, same assertion, and a destination that really is the guarded venv. * docker: scope the destination variables to the tool that reads them The destination bypass collected PIP_ and UV_ variables unconditionally, so `pip install torch==...` with UV_PYTHON set was granted a bypass on a variable pip never reads, and the install then landed in the baked venv with no filtering and no constraints. That is worse than the bug the bypass was written to fix: that one left an unrelated environment empty, this one replaces the CUDA stack the image is built on. Measured against both CLIs, using a package present in the base environment and absent from the target venv so "already satisfied" means the base was targeted: pip with UV_PYTHON reported already satisfied from the base, pip with PIP_PYTHON reported "Would install", uv with PIP_PYTHON reported "environment at: <base>", and uv with UV_PYTHON reported "environment at: v1". uv also ignores PIP_TARGET, PIP_PREFIX and PIP_ROOT, and has no environment form for its own --target or --prefix, so the directory variables are pip-only. The three pip directory variables were confirmed with real installs rather than dry runs, because --dry-run reports "already satisfied" for PIP_PREFIX from the current environment and would have made it look ignored. Command-line flags stay unscoped: a tool that does not accept one fails loudly rather than silently doing the wrong thing, so there is no silent bypass to prevent there. * docker: filter install targets that arrive through pip's environment pip reads PIP_<OPTION> for every option, so a requirements file or an editable target can arrive with nothing on the command line to show for it. The scan only walked argv, so `PIP_REQUIREMENT=/tmp/r.txt pip install packaging` filtered packaging and handed the real pip an uninspected file. Measured against the bundled pip, using its own no-target error as the control: with nothing set it reports "You must give at least one requirement to install", with PIP_REQUIREMENT set it reports "Would install idna-3.6", and with PIP_EDITABLE set it reports "Would install dummyproj26-1.2.3". Both variables are whitespace separated when they carry more than one value. The injected constraints do not cover this, and not because they are overridden: a CLI --constraint still binds when PIP_CONSTRAINT is also set, and the two conflict rather than one winning. What defeats them is a same-version editable or direct reference, which resolves cleanly against a matching pin. So the variables are folded in as ordinary -r and -e flags and go through the same filtering as the command line, and are cleared from the child so the real tool cannot consume them a second time beside our filtered copy. uv has no environment form for --requirements, so folding one into a uv command would hand it a file it was never asked for; this is pip-only. Not covered, and stated rather than implied: the same input can arrive through pip's config file. With PIP_CONFIG_FILE pointing at a pip.conf carrying [install] requirement, pip consumed it with nothing on the command line, and setting the variable empty does not neutralise it. Covering that means parsing pip's config search path or shelling out to `pip config list` on every install. * docker: say plainly that a failed Studio update already replaced the venv The failure path printed "NOT restarting Studio: the running process keeps serving", which reads as though nothing is wrong until the operator chooses to restart. Three things were left unsaid, and all three were reproduced. The venv on disk has already been replaced by the time the import check runs, and the previous distribution is gone, so recovery needs the network. "The running process keeps serving" holds only for code it has already imported. The backend defers thousands of imports, so the first lazily imported module the replaced tree no longer satisfies fails in the still-running process. Any restart parks Studio in FATAL. Under a real supervisord with this repo's own autorestart, startretries and startsecs from docker/supervisord.conf, the next start gives "Exited too quickly" and then "gave up: studio entered FATAL state", and nothing in studio_launch.sh or entrypoint.sh repairs it. Because the venv lives under $UNSLOTH_STUDIO_HOME, which the header recommends putting on a named volume, docker rm plus docker run does not restore it either. Message only. Backing the venv up first is a separate design decision: it is a 2.5 to 3.5 GB tree, the hardlink shortcut that would make it cheap forces an overlayfs copy-up inside a container, reinstalling previous versions needs the network at the moment the network may have just failed, and the pre-update state is an editable install rather than a wheel. A broken restore would be worse than a broken update, and this script has no image-level CI. * docker: close four holes in notebook pip filtering Four ways an install could reach the baked venv without being filtered, or could pin the wrong transformers version. --root is a relocation prefix, not a destination directory, so --root / is the identity and the install lands in the venv it was supposed to be diverted away from. It is now resolved against the venv root instead of being treated as an external target on sight, and it is classified apart from --target and --prefix, which really are directories. Direct-reference detection was case sensitive, so GIT+FILE:/checkout or HTTPS://... slipped past the scheme test and the local-project lookup. Both the scheme prefix test and the direct-reference match are now case insensitive, which is what PEP 508 and pip both do. --group and --requirements-from-script name install targets that live in a pyproject table or a PEP 723 header rather than on the command line, so the shim cannot see or filter them. Parsing either would mean following include-group chains and rewriting the result, and getting that subtly wrong reintroduces the bypass. An install carrying one of them is refused instead, pointing at -r, which is filtered. The refusal runs after the install-index check and after the destination bypass, so a non-install subcommand and a group aimed at another environment are both untouched. A PEP 508 marker on a pin was ignored, so transformers==5.5.0 with a marker that evaluates false still produced a 5.5.0 pin even though pip and uv skip the requirement outright. Both sites that read a pin now evaluate the marker: the pre-run hook in unsloth_nb_compat and the sidecar marker written by the shim. Lines are split the way a shell splits them, so an unquoted marker, which the shell cuts before pip sees it, still pins unconditionally. packaging stays optional, as it already is for sidecar version ordering: if it is missing or the requirement does not parse, nothing is vetoed. test_dependency_group_flags_are_install_targets asserted these flags reach the real tool, which the refusal contradicts. Its purpose was that they must never quietly succeed, so it is rewritten to assert the refusal, keeping all five of its spellings. * Rename the sync marker into place so a --user boot can advance it A root boot leaves .unsloth_sync_commit root-owned 0644 inside a notebook directory the host user owns. A later boot as that user renames the state file into place without trouble, because that only needs the directory, but the marker was written with a plain redirect and the truncate fails. The refresh then printed its success line while the marker stayed at the old commit, and every start after that re-synced all 1241 notebooks. Reproduced in the shipped base image: root boot, rewind the marker as root, boot as 1000:1000. The state came back owned 1000:1000 and the marker did not. Write the marker to a sibling temp file and rename it, the same path the state already takes, and say so when even that fails instead of swallowing it. The group redirection also keeps the shell's own open() error from leaking, which the old trailing 2>/dev/null could not do because redirections apply left to right. The test models the root-owned marker with a read-only one, which fails the same open(O_TRUNC), and is skipped under root. It fails on the previous script. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <info@unsloth.ai>	2026-09-04 16:26:55 -07:00