24 commits

Author	SHA1	Message	Date
Daniel Han	888ccc54df	Studio: install from a package index that lags PyPI (#14133) ... Fixes https://github.com/unslothai/unsloth/issues/14031 ## Problem Every release raises the installer floors to itself (`unsloth>=2026.10.6`, `unsloth-zoo>=2026.10.6`), and Desktop asks for its own shipped backend (`UNSLOTH_DESKTOP_BACKEND_VERSION`). A package index that lags PyPI (a delayed corporate feed, a mirror, an upload cutoff) never has that release yet, so with releases every two or three days and a 7 day feed delay the install is unsatisfiable on every run: ``` × No solution found when resolving dependencies: ╰─▶ Because only unsloth<=2026.10.3 is available and you require unsloth>=2026.10.6, we can conclude that your requirements are unsatisfiable. ``` Desktop then cannot launch either: even when the backend is installed by hand, preflight sees it below the shipped version (`desktop_backend_version_outdated`) and starts a repair, which hits the same wall. ## Change - `install.sh` / `install.ps1`: one dependency-free `uv pip compile` (about 0.05 s warm) checks the floors against whatever index configuration uv reads (env, `uv.toml`, cutoffs). On PyPI it passes and nothing else changes. When the index cannot satisfy them, the installer takes that index's newest `unsloth-zoo`, then the newest `unsloth` not above it, pins that pair (`==`) and prints a warning naming both versions. The probes force `--resolution highest` and write to their own scratch file, so neither `UV_RESOLUTION=lowest` nor a configured `output-file` changes the result or touches user files. - `studio/install_python_stack.py`: the same fallback on the `studio update` path Desktop runs, for Desktop's backend floor. - When Desktop asked for the backend, the installer writes `unsloth_feed_ceiling.json` (`{"requested": ..., "available": ...}`) next to `pyvenv.cfg`, and removes it on every run where it is not needed. - Desktop preflight: an `outdated` backend counts as ready only when that marker names this build's expected version and the installed backend is at or above the marker's version. A newer Desktop, or any other stale reason, is graded as before. - The zoo floor literal moves into one variable per installer; `tests/test_installer_zoo_floor_parity.py` reads it from there. ## Tests - `tests/sh/test_relax_unsloth_floors_to_index.sh`, `tests/studio/test_relax_unsloth_floors_to_index.ps1` and `TestDesktopBackendVersionConstraint` (uv stubbed): floors kept when the index has the release or is unreachable, lowered and pinned when it lags (env index, uv.toml index, cutoff), zoo lagging unsloth, hashed resolver output, the marker written only for Desktop and cleared once the feed catches up, the probe scratch file removed. - Rust: `a_feed_ceiling_marker_covers_only_the_release_it_names`; the full `cargo test` suite passes. - Base vs head installer on a local mirror that serves only unsloth / unsloth-zoo 2026.10.3 ahead of PyPI, `--no-torch`, Desktop floor set: on ubuntu-latest, windows-latest and macos-15, main fails with the error above, this branch installs 2026.10.3, warns, writes the marker and the backend reports itself ready. - Same comparison with the full CUDA install on a Colab A100 and T4: main fails, this branch installs 2026.10.3 with a working CUDA torch; a plain PyPI install on this branch still gets 2026.10.6 and writes no marker.	2026-10-18 09:50:33 -07:00
Daniel Han	50b9deacb0	Trim comments in tests (#13844) ... Comment and docstring changes only, in test files, split out of #12968 so the test-only part can merge first.	2026-10-15 05:30:27 -07:00
Long Yixing	8c8dd864cd	feat(install): fall back to CERNET and npmmirror when package hosts are blocked or slow (#11786) ... * feat(studio): let UNSLOTH_NODE_MIRROR rebase the Node.js dist downloads install_node_prebuilt.py always fetched index.json, SHASUMS256.txt and the archive from nodejs.org. UNSLOTH_NODE_MIRROR now replaces that base for every Node dist URL; archives are still verified against the sha256 pins committed in node_prebuilt_pins.json, so a mirror cannot change what gets installed. * feat(install): fall back to CERNET and npmmirror when package hosts are blocked or slow install.sh and studio/setup.sh now probe pypi.org, download.pytorch.org, nodejs.org and registry.npmjs.org in parallel (5 s connect, 10 s total) before the first package install. A host that fails the probe while its mirror passes is redirected through the env vars the package tools already read: PyPI to https://mirrors.cernet.edu.cn/pypi/web/simple, PyTorch wheels to UNSLOTH_PYTORCH_MIRROR=https://mirrors.cernet.edu.cn/pytorch/whl, Node.js to UNSLOTH_NODE_MIRROR=https://mirrors.cernet.edu.cn/nodejs-release, and npm to UNSLOTH_NPM_REGISTRY=https://registry.npmmirror.com. When pypi.org still answers but misses the budget, uv gets the mirror first and pypi.org second with UV_INDEX_STRATEGY=unsafe-first-match, and pip gets the mirror as PIP_INDEX_URL with pypi.org as PIP_EXTRA_INDEX_URL, so packages the mirror has not synced yet still resolve. When pypi.org does not answer at all, only the mirror is used: unsafe-first-match fetches every index and fails outright when one is unreachable. A host the user already redirected (UV_* / PIP_* index vars, an index in uv.toml or pip.conf, UNSLOTH_PYTORCH_MIRROR, UNSLOTH_TORCH_INDEX_URL, UNSLOTH_NODE_MIRROR, UNSLOTH_NPM_REGISTRY) is left alone, setup.sh skips the probe when install.sh already ran it, and UNSLOTH_MIRROR_FALLBACK=0 turns the fallback off. * feat(install): mirror fallback for blocked or slow package hosts on Windows install.ps1 and studio/setup.ps1 get the same fallback as install.sh: pypi.org, download.pytorch.org, nodejs.org and registry.npmjs.org are probed in parallel, and a host that fails while its CERNET or npmmirror counterpart passes is redirected through the same env vars (UV_*/PIP_* index vars, UNSLOTH_PYTORCH_MIRROR, UNSLOTH_NODE_MIRROR, UNSLOTH_NPM_REGISTRY), with the same slow-versus-blocked PyPI split and the same respect for indexes the user already configured. The helpers live in the block shared with setup.ps1. The probe uses WebRequest.GetResponseAsync rather than HttpClient, which would need Add-Type, and sends a User-Agent because the CERNET mirrors answer 403 without one. On a session whose SecurityProtocol excludes TLS 1.2 it pins TLS 1.2 for the probe only. install.ps1 restores every variable the fallback may set when it exits, since under irm | iex they belong to the caller's session. setup.ps1 now computes its npm --registry args after the probe, so a fallback registry reaches every npm and bun install. * feat(install): fall back to npmmirror for uv-managed Python builds when releases.astral.sh is blocked or slow * feat(install): install the pinned uv from a PyPI mirror wheel when releases.astral.sh is blocked or slow * feat(install): install the pinned uv from a PyPI mirror wheel on Windows when releases.astral.sh is blocked or slow * fix(install): define the mirror fallback outside the Unsloth Studio launcher heredoc The mirror fallback functions sat inside the quoted heredoc that writes the launcher script, so install.sh never defined them and aborted with "_mirror_fallback: not found" right before installing uv. Move the block next to its call and add a check that the block is outside every heredoc in install.sh and studio/setup.sh. * feat(install): pick mirrors by measured speed on real artifacts Each default package host is timed alone on a 1 MiB slice of an artifact it serves, and one below 1 MiB/s, or whose index does not answer, is raced against its mirror and swapped for it only when the mirror is faster and its own index answers. CERNET redirects each tree to a different university mirror, so each tree is timed on its own, and releases.astral.sh is timed once for both uv and its Python builds. * feat(install): pick Windows mirrors by measured speed on real artifacts Ports the speed probe from install.sh to install.ps1 and studio/setup.ps1: each default host is timed alone on a 1 MiB slice of an artifact it serves, and one below 1 MiB/s, or whose index does not answer, is raced against its mirror and swapped for it only when the mirror is faster and its own index answers. Each probe uses a fresh connection and its own deadline, as a separate curl process would, and the probes raise Windows PowerShell 5.1's two-connections-per-host limit so the race's CERNET requests do not queue. The uv Python-builds mirror is dropped on Windows, where uv is always given a resolved interpreter. * feat(install): retry a step whose default host's transport failed through its mirror The mirror probe leaves healthy defaults alone. When a later step fails because the default host stopped answering (a timeout, reset, DNS or TLS error, a 5xx/429, or a download that stalled), the installers now rerun that step once through the host's mirror, after the step's own same-host attempts. The host is read from the failure output, or from the command when a stalled download names no URL; resolution, not-found and other hosts' failures are never retried, so try-then-fallback steps keep their own fallback. A host stays on its mirror for later steps only when the rerun succeeded. Covers install.sh's install commands, setup.sh's npm installs, install_python_stack.py's pip_install and pip_install_try, and the node and pinned uv downloads (only when no source served the file). * feat(install): retry a Windows step whose default host's transport failed through its mirror The Windows mirror fallback now exports the same spare list as the POSIX installers: one mirror per probed host still on its default. install.ps1 reruns a failed package install once through that host's mirror after its own attempts, only when the output shows a transport failure on that host (or a stalled download in a command that uses the default index). A torch mirror rewrites the default torch URLs the command reads, and a mirror is kept for later steps only when the rerun succeeds. studio/setup.ps1 does the same for the frontend and OXC npm installs, and retries the Node download and the pinned uv download (the latter only when no source answered). Every install_python_stack.py run on Windows gets the same retry through the exported list. The shared-helper sync script now also copies a new shared function that has no comment above it. * fix(install): use TUNA's CERNET mirror and npmmirror for Node mirrors.cernet.edu.cn sends the PyPI simple index to a mirror that rate-limits uv's parallel metadata requests (429), failing resolves; tuna.mirrors.cernet.edu.cn serves the index and files from Tsinghua. The Node dist mirror moves to registry.npmmirror.com/-/binary/node, which keeps nodejs.org's layout (index.json, SHASUMS256.txt, archives), and is timed on its own tarball. * feat(install): fall back to pypi.org only for releases the PyPI mirror lacks A switched PyPI host now uses the mirror alone, also when pypi.org was only slow: uv's unsafe-first-match fetched every package from pypi.org too, so a pypi.org that dropped mid-install failed every step. When pypi.org still answered, a failed resolve that reports a version or package as not available (a fresh release the mirror has not synced yet) reruns once with pypi.org behind the mirror, and later steps keep that setup if the rerun works. Commands that pin their own index are left alone. install_python_stack.py classifies its installs the same way. * feat(install.ps1): fall back to pypi.org only for releases the PyPI mirror lacks The Windows installers now match install.sh: a switched PyPI host uses the mirror alone, also when pypi.org was only slow, and when pypi.org still answered, an install whose resolve reports a version or package as not available reruns once with pypi.org behind the mirror, keeping that setup for later steps if it works. Commands that pin their own index or pass --no-index are left alone. * fix(install): keep uv on a project-level index when falling back to the PyPI mirror uv reads uv.toml, or pyproject.toml with a [tool.uv] table, from the current directory or the nearest parent, and an exported UV_DEFAULT_INDEX overrides it. The mirror fallback only checked UV_CONFIG_FILE and the user and system uv.toml, so launching the installer from a project that configures its own index would redirect uv to the mirror. It now also checks the nearest project config, as install_python_stack.py's _uv_config_files does, and recognizes [[tool.uv.index]]. * fix(studio): probe package hosts only before a setup step downloads packages setup.sh and setup.ps1 ran the mirror speed probe at startup, so every unsloth studio update, even one with nothing to install, fetched a 1 MiB sample from each default package host. The probe now runs right before the Python dependency pass and before a Transformers sidecar is rebuilt, the update steps that download from PyPI and PyTorch. install.sh and install.ps1 still probe up front and pass the result on, so fresh installs are unchanged; a standalone update's Node, npm and uv steps use the default hosts, as before the mirror fallback. * test: expect _pip_install_once among the requirements-filter callers pip_install now wraps _pip_install_once with the mirror retry, and the install itself, including its _effective_requirements call, lives in _pip_install_once. * fix(install): leave pip on an index set in the Unsloth Studio venv's pip config pip also reads site config from the venv it runs in (<venv>/pip.conf, pip.ini on Windows). The mirror fallback only checked PIP_CONFIG_FILE and the user and global files, so a private index set in the Unsloth Studio venv was overridden by the exported PIP_INDEX_URL whenever the install fell back from uv to pip. The check now includes the venv's site config. * fix(install): keep a verbose install working on a host without tee The mirror retry reads a failed command's output, so the verbose install arm copied it to a log with tee. install.sh otherwise needs no tee, and on a minimal image without it the pipe closed and the command died of SIGPIPE. The arm now falls back to cat there, which only costs that host the retry. The tests that lift run_install_cmd out of install.sh also lift the functions it now calls, and stub the mirror retry. * fix(studio): arm the mirror retries before setup's first download and skip the probe on a --local re-overlay With the probe moved to the Python dependency pass, a standalone setup reached its Node and npm downloads with no mirror retry armed. setup.sh and setup.ps1 now arm every host's one-shot mirror retry at startup without probing (_mirror_fallback spare / Invoke-MirrorFallback -SpareOnly), and still probe lazily before the first package download. setup.sh arms it from the directory its installs run in, once VENV_DIR is set, so the retry respects the same uv and pip config they do. A --local update runs the dependency pass every time to re-overlay the checkout, almost entirely from uv's cache, so on an existing venv that pass relies on the armed retries instead of probing every host first. * Trim installer mirror fallback comments * Respect NPM_CONFIG_REGISTRY and recheck default indexes in the mirror race A registry set through NPM_CONFIG_REGISTRY / npm_config_registry outranks the frontend .npmrc today; the fallback exported UNSLOTH_NPM_REGISTRY, which becomes --registry and overrode it. Treat it as user configuration. The race re-timed the default artifact but reused the first-round default index result, so one index answer lost in the first round forced a healthy host into blocked mode. Probe the default index again beside the mirror's. * Only use the mirror fallback in mainland China Outside mainland China the installers now behave as before the fallback: no probe, no mirror retries armed, nothing exported. Location is read without a network call: a mainland time zone (Asia/Shanghai and its aliases, Windows "China Standard Time"), or a resolver from a mainland public DNS or cloud (AliDNS, DNSPod, 114DNS, Baidu, CNNIC, Alibaba Cloud and Tencent Cloud internal resolvers), which covers UTC cloud VMs there. UNSLOTH_MIRROR_FALLBACK=1 opts in anywhere; =0 still opts out. * Keep the installers byte-identical to before outside mainland China With no mirror retry armed (outside mainland China, or opted out): - run_install_cmd runs the plain runner, so set -e stops the installer where a command fails as before, and verbose output is not copied to a temp file through tee. - The PowerShell runners stream verbose output without buffering it, and the npm / install retries return before reading any output. - _mirror_retry in install_python_stack.py returns before decoding and scanning the failed output (183 ms for 1 MB of output before). - A retry state inherited from a parent process is dropped, so neither a stale _UNSLOTH_MIRROR_SPARE nor UNSLOTH_MIRROR_FALLBACK=0 can be bypassed. - setup.sh no longer changes directory (and OLDPWD) to arm the retries. The location is read once per process, resolvers come from .NET instead of Get-DnsClientServerAddress (no module or CIM load on Windows), and the 114DNS filtering resolvers plus DNSPod 182.254.116.116 / 119.28.28.28 count as mainland China. An interrupted install also removes a kept output copy. * Mirror fallback: honour XDG_CONFIG_DIRS, skip permission failures, retry sidecars - pip and uv read an index from every directory in XDG_CONFIG_DIRS; an index configured there now counts as user configuration, so the fallback no longer overrides it. - The Node install no longer retries through the mirror on exit 4 (permission denied), matching setup.ps1. - Transformers sidecar installs rerun once with pypi.org behind the mirror when the probe switched a slow pypi.org, as the dependency pass already does for a pin the mirror has not synced. All three only act when the fallback is on (mainland China or UNSLOTH_MIRROR_FALLBACK=1). * Trim mirror fallback comments * Mirror fallback: probe nothing when UV_OFFLINE is set uv's own no-network switch (its spellings: 1, t, true, y, yes, on) now skips the speed probes and the mirror race; the one-shot retries stay armed, as they only follow a failed network command. * Keep vendor names out of the installer comments tests/studio/test_installer_av_shapes.py bans vendor names in shipped scripts, which are handed whole to scanners; the location check comment now describes the resolvers without naming the clouds. --------- Co-authored-by: danielhanchen <danielhanchen@gmail.com> Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>	2026-09-26 02:57:24 -07:00
Daniel Han	4a955d2bee	Let the stdout guard read writes, not flushes (#11576) ... * Let the stdout guard read writes, not flushes test_no_direct_stdout_writes flags sys.stdout.write and sys.stdout.flush outside the progress helpers, and has failed on main since #11566, which flushes stdout and stderr before rerunning the installer it just installed. That flush is needed: the child writes to the same descriptors, so anything still buffered would land after its output. A flush emits nothing and cannot glue text onto the progress bar, which is what the guard exists for. It now matches write only; a direct write on the same line still fails it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-09-22 12:40:18 -07:00
Daniel Han	6a4a041380	Studio: keep the operator's package-manager policy in the installer (#10902) ... * Studio: keep the operator's package-manager policy in the installer #8579 made the installer's own pip and uv commands survive a hardened uv.toml / pip.conf, but it did that with a blanket amnesty: every pinned install dropped no-build, only-binary and exclude-newer from the environment and ran with PIP_CONFIG_FILE at devnull and UV_NO_CONFIG=1, so the operator's build-time code execution controls went with it. Narrow that to the one thing the installer actually cannot satisfy. * Hash enforcement (PIP_REQUIRE_HASHES / UV_REQUIRE_HASHES) is still cleared: every requirements file we ship is pinned but unhashed, so require-hashes has no artifact to check and can only abort the install. * no-binary is still cleared for pinned commands, because it would FORCE a source build of a wheel the pin exists to fetch. That is less build-time execution, not more. * UV_NO_BUILD, UV_NO_BUILD_PACKAGE, PIP_ONLY_BINARY and UV_EXCLUDE_NEWER are now left in force. Every pinned index we install from serves wheels, so honouring them costs the pin nothing. * PIP_CONFIG_FILE points at a rewrite of pip's own config minus the four source keys (the existing _pip_config_without_sources) instead of devnull, so cert, proxy, trusted-host and only-binary survive while the pin still gets the index determinism it needed. Computed once per run. * UV_NO_CONFIG=1 has to stay (uv has no per-key override and a discovered uv.toml outranks the CLI pin), so a no-build in that file is read back and re-asserted as UV_NO_BUILD=1 rather than silently lost. * The hash relaxation is keyed on the pip SUBCOMMAND now, not on the word "install" appearing somewhere in argv. Measured against a real pip.conf carrying require-hashes, only-binary, no-index and cert: raw pip refuses the install, the installer path installs the wheel, the installer path REFUSES the same package when only an sdist is available, and a pinned --index-url install succeeds with cert and only-binary intact. The package-scoped --no-binary exemptions for the four audited wheel-less requirements and the diffusers source archive are unchanged; they are what keeps #8530 fixed. * Keep the pin deterministic: devnull, then put the policy back key by key Simulation caught a real regression in the first pass of this branch. Pointing PIP_CONFIG_FILE at a rewritten config does NOT suppress a SITE or GLOBAL pip.conf: pip documents, and pip 26.2 confirms, that naming an existing file skips the per-user file only, while os.devnull is what "disables the loading of all configuration files". So a venv-level or /etc-level `no-index` came back to life and killed the pinned install: pip install --index-url <pin> packaging==24.0 -> No matching distribution That would have broken every torch repair, the XPU triton fetch and the torchao pin on any host with a site or global pip.conf. Reverted to os.devnull and put the operator's settings back one key at a time instead: * _pinned_pip_config_overrides() reads `pip config list` once per run and re-asserts an ALLOWLIST as PIP_ environment variables: cert, client-cert, proxy, trusted-host, timeout, retries, keyring-provider and only-binary. The transport half also fixes a pre-existing bug, since devnull used to drop the CA and proxy a private index needs. * index-url, extra-index-url, find-links and no-index are never re-asserted (the pin replaces them), nor is no-binary (it would force a source build of a pinned wheel), nor require-hashes (unsatisfiable against unhashed requirements). * Only global/install/download/wheel sections are read, so a `list.format` does not become a global PIP_ variable, and `:env:` rows are skipped so the child cannot re-inherit the variables the scrub just cleared. Also dropped the uv side of the previous commit. Measured against uv 0.10.7, UV_NO_BUILD, UV_NO_BINARY and UV_ONLY_BINARY are not uv environment variables at all: uv reads no-build from its config file, and re-asserting the variable was dead code promising a guarantee uv does not honour. UV_EXCLUDE_NEWER and UV_REQUIRE_HASHES are real and are handled as before. A uv.toml no-build still applies to every non-pinned command, which is where source builds actually happen. _executable_stem() splits on both separators, so a Windows argv0 is recognised off-Windows too, and a degenerate command can no longer raise out of the subcommand test. Verified: 78 of 90 base-vs-head environment comparisons are byte-identical and the other 12 are the intended hardened-host cases; real installs across pip 21.3.1 to 26.2.1 and Python 3.9 to 3.13; a site-config no-index no longer breaks a pinned install while only-binary still refuses an sdist. * Make the config read unable to fail the install, and the tests hermetic Three defects found reviewing the previous commit, all in code this branch introduced. The `pip config list` read sits on the path to every pinned install, including the final torch repair, so it must degrade to "no overrides" for ANY unexpected answer and not just for a missing pip. It caught OSError and TimeoutExpired but would have propagated anything else: a listing that is not bytes, a stdout of None, a decode that raises. Parsing moved into _parse_pinned_pip_config() and the whole read is now wrapped, with cases for each shape. A command section now beats `global` for the same option, which is pip's own precedence. It previously depended on the order `pip config list` happened to print the two lines in, so `global.only-binary=:all:` plus `install.only-binary=numpy` resolved differently run to run. The tests were not hermetic. A CI image carrying its own /etc/pip.conf would have leaked into the pinned-env assertions, and a test that mocks subprocess could populate the memoised read with a Mock and poison whatever ran next under random ordering. An autouse fixture clears the cache around every test and stubs the read, with an opt-out marker for the few tests that drive it themselves. Verified by running the three files repeatedly under random ordering. * Fix eight defects found reviewing this branch All eight are in code this branch introduced. The exec-extraction in tests/studio/test_xpu_triton_swap.py stopped pulling the parser once it was split out, so the "real scrub" that file deliberately executes raised NameError into a broad except and returned {} for every case: the file still reported 62 passed while asserting nothing. The loader now executes the extracted parser once against a known listing, so an inert extraction fails loudly instead of agreeing with everything. That mask was the second defect. The read caught OSError and TimeoutExpired around the subprocess but then wrapped the parse in `except Exception`, turning any programming error into "this operator has no pip config", which on a corporate host is a silent loss of cert, proxy and trusted-host on every pinned install. Narrowed to (OSError, subprocess.SubprocessError) around the call, with the parser skipping unparseable lines as before. Section precedence let [wheel] and [download] overwrite [install], which is not "more specific", it is a different subcommand: an operator with `[install] only-binary = :all:` and `[wheel] only-binary = numpy` had their install-wide policy downgraded to one package. Only [global] and [install] are read now, install winning, since the PIP_ variable is command-wide anyway. The memoisation cached failures as well as successes, so one transient miss (the 60s timeout, or a read racing the pip bootstrap) cost the operator their transport settings for the rest of the run. Only a successful read is cached. Value joining applied one rule to keys with two. `cert` is a single value, and collapsing whitespace turned `C:\Program Files\ca.pem` into a path that does not exist; `only-binary` is comma separated, so a two-package value became one bogus name and the policy silently stopped applying. Measured against pip 26.2: PIP_ONLY_BINARY="a,b" refuses both as sdists, PIP_TRUSTED_HOST="a b" is accepted, and `pip config list` renders a multi-value setting on one line with an escaped \n. The subcommand scan skipped leading-dash tokens but not the VALUE of a global option that takes one, so `pip --cache-dir /tmp/c install x` read as not an install and would have lost the hash relaxation. It now looks for the first token that names a pip subcommand. UV_EXCLUDE_NEWER goes back to being cleared for pinned commands, which is the pre-existing behaviour. Only uv reads it, _build_pip_cmd never adds pip's --uploaded-prior-to, and pip_install falls back to pip whenever uv fails, so honouring it on the uv leg alone means a pinned install that fell back quietly installs an artifact the cutoff forbids. And a dead REAL_PINNED_PIP_CONFIG_OVERRIDES assignment that read as if it were the fixture's restore mechanism is gone. Suites: 10768 passed. The 2 torchcodec failures reproduce on unmodified main. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments this branch added Comment and docstring lines in the diff: 178 to 105. Every measured fact is kept, in one line where it took four: the devnull suppression rule, the pip 26.2 and uv 0.10.7 measurements, the separator each config key is read with, and the NameError trap in the exec-extraction. AST-gated comment-only. * Read the pip config section that belongs to the command being run pip config is per subcommand, and the re-assertion was reading [install] for every pinned command. Measured on pip 26.2: a pip.conf carrying [download] no-index stops a pip download and leaves pip install untouched, so translating [install] onto _ensure_xpu_triton's pinned pip download both drops that command's own cert, proxy and trusted-host and imposes an install-only policy on it. _pinned_pip_config_overrides() now takes the subcommand and reads [global] plus that command's section, with the command section winning. _pip_subcommand_of() resolves it from the command; a uv command defaults to install, since the PIP_ variables only ever reach its pip fallback. The listing is memoised instead of the parsed result, so this still costs one subprocess per run. Suites pass, and the real-pip, audit, differential and old-pip simulations are unchanged: 27/27, 15/15, 78 identical environments with 12 intended, 30/30. * Accumulate repeatable pip options, and do not read an empty env value as set Two ways the re-assertion could still lose the operator's policy, and one test that could pass for the wrong reason. pip ACCUMULATES a repeatable option across sections rather than letting the command section replace the global one. Measured on pip 26.2: a pip.conf with [global] only-binary = :all: plus [install] only-binary = numpy still refuses an unrelated sdist, so emitting PIP_ONLY_BINARY=numpy dropped the :all: policy on every pinned install. List keys now accumulate global plus the command section, deduplicated, with the separator each key is read with; scalars still take the command section alone, since two certs cannot be concatenated. Verified end to end: that config now yields PIP_ONLY_BINARY=':all:,numpy' and a pinned sdist install is refused. An empty inherited value is no longer treated as an override. pip ignores an empty environment value and falls through to the config file, which the pinned branch has just removed with devnull, so PIP_CERT= in the parent environment left the child with neither and a private index unreachable. Confirmed with env_var: PIP_CACHE_DIR='/mnt/disks/unslothai/daniel1/.cache/pip' env: global: /etc/xdg/pip/pip.conf, exists: False /etc/pip.conf, exists: False site: /mnt/disks/unslothai/daniel1/workspace_1/pip.conf, exists: False user: /home/daniel1/.pip/pip.conf, exists: False /home/daniel1/.config/pip/pip.conf, exists: False: PIP_TIMEOUT='' shows the variable and then uses global.timeout from the file. A whitespace value is left alone; that is a value pip would use, not ours to second-guess. The download-section test now clears PIP_CERT: the caller's environment legitimately wins over the re-assertion, so on a host that exports one the test asserted on the ambient value. * Keep the order of a repeatable option instead of deduplicating it pip applies a repeatable option IN ORDER, and :none: empties the set, so a re-add after a reset has to survive. Deduplicating dropped it: with [global] only-binary = a,b and [install] only-binary = :none:,a the parser emitted a,b,:none:, which ends on the reset and allows exactly the sdist build the operator forbade. Measured on pip 26.2 against a venv pip.conf carrying that config: pip itself refuses the sdist, the deduplicated string allows it, and the order-preserving concatenation refuses it again. End to end, the installer and pip with its own config now return the same exit code for the same install. * Give trusted-host section precedence instead of accumulating it pip's own parser, asked with [global] and [install] both set, resolves trusted_hosts to the install value alone (an append option, assigned per section) while format_control holds both (a callback that mutates in place). Accumulating trusted-host re-trusted a host the install section had dropped, which is a TLS decision, so the two keys are now separated: _PINNED_PIP_CONFIG_SEPARATORS says how a value is spelled in the environment, _PINNED_PIP_CONFIG_ACCUMULATING says which key carries across sections, and only only-binary is in it. * Make both files a fixed point of the formatting hook Backend CI's formatter fixed point check was failing on this branch. One of the two files is mine: ruff 0.15 formats the new parser differently from the 0.6.9 the hook pins, so what I committed was not what the hook produces. The other, tests/python/test_docker_nvidia_toolkit_install.py, is pre-existing drift that reached main in #10623 and is exactly the case that test was written to catch; it is untouched by this branch otherwise, and the change here is the hook's own output. * Decode pip config output with the encoding the child wrote A piped child encodes stdout with its own locale encoding, which on Windows is the ANSI code page rather than UTF-8, so the unconditional UTF-8 decode corrupted a cert or client-cert path spelled with non-ASCII characters. That is worse than losing the setting: the corrupted path exists nowhere, so pip fails the pinned install outright on exactly the corporate host the allowlist exists to serve. UTF-8 is still tried first and strictly, so the POSIX case and Windows UTF-8 mode are unchanged; the locale codec is the fallback. * Enforce the restored only-binary on the uv leg, and dictate the read encoding Two gaps the review found. uv reads neither pip.conf nor PIP_ONLY_BINARY, and a pinned command runs with UV_NO_CONFIG=1 because a discovered uv.toml outranks the CLI pin (#6898), so restoring the policy in the environment alone left the leg that actually runs free to build a source distribution the operator had forbidden. Measured against uv 0.10.7: a pinned uv install builds the sdist with PIP_ONLY_BINARY=:all: set, and refuses it given --only-binary. _build_uv_cmd now translates the restored value into uv flags for pinned commands only, since a non-pinned one keeps its config file and uv applies the policy itself. Second, UTF-8 first was not a reliable encoding detector: cp1252 bytes can form valid UTF-8, so a mis-encoded cert path could decode cleanly into the wrong path. The read now dictates PYTHONIOENCODING=utf-8 rather than sniffing what the child wrote, at both call sites that read pip config, and the locale decode stays only as a fallback. * Tighten the comments added by this branch Comment and docstring text only, with no code change: the AST gate confirms it. Ten lines out, mostly from the two longest measurement notes, with each measured fact kept. * Single-source the uv policy flag, and bound what a wedged pip can cost Three findings from an independent audit of this branch. _build_uv_cmd asked for the pip config separately from _install_env_for_cmd, so if the first read failed and the second succeeded the uv command went out without --only-binary while its own environment carried PIP_ONLY_BINARY. The flag now comes off the env that very command will run with, so the two cannot disagree. The 60s timeout was per attempt and unbounded across the run, because failures are deliberately not memoised. A wedged pip therefore paid it once per pinned command. A HANG is now budgeted, twice at 30s for the whole run, which is what the comment always claimed. Budgeting every failure would have been wrong in the other direction: a fresh venv has no pip for the first part of the run, that answer is instant, and spending the budget on it would lose the operator's cert the moment pip appeared. Cheap failures stay freely retryable, with a test for each direction. * Tighten the comments added by the audit fixes Comment text only, no code change; the AST gate confirms it. * Derive uv's only-binary flags from the same config read as its environment * Exempt the sdist-only rocm package from the only-binary a pinned install re-asserts * Add an exported only-binary to the configured one instead of replacing it * Limit the rocm exemption to AMD per-arch index pins --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-09-14 19:15:00 -07:00
Daniel Han	38f78af7c3	Fold duplicated backend, hub and package tests into parametrized runs (#10499) ... * Collapse duplicated tool call parser tests into parametrized cases * Parametrize transformers tier detection tests and share a config writer * Parametrize sandbox tool policy allow and block cases * Fold duplicated test cases into parametrized runs and shared setup helpers * Hoist verbatim repeated test setup runs into shared helpers * Drop the scratch area file list from the branch * Name the generated parametrize arguments after what each value is * Share the hardware verdict guard and fold the emission form loop tests * Share the Responses stream drivers across the adapter tests * Share the page fetch stub across the content type tests * Restore the fuzz input labels and the finding tag lost when folding parametrized cases * Name the folded parametrized tests and their arguments for what they check * Restore the repo quote and keyword spacing style on the rewritten test lines * Drop the dead return and its assignment from the tool output stream helper * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-record the patch targets the parametrized folds collapsed * Preserve the full former test name in every folded parameter ID The parametrize folds shortened or reworded some case IDs, so the pre-fold test name could no longer be recovered from the node ID by grep. Restore it: every folded case now carries the original test name minus its test_ prefix, and originals that were themselves parametrized carry that name plus their original case label. IDs only. No test input, expected value or assertion is changed. Collected node id count is unchanged at 57741. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Return only the routes module from the shared KB setup helper _shared_setup_4 built its knowledge base inside `with _connection() as conn` and then returned that connection. _connection is a `closing(...)`, so the handle is already closed by the time the caller sees it. All four call sites happen to shadow it with their own `with _connection() as conn` before any use, which is why nothing fails today, but the helper is new here and handing a closed connection to whoever writes the next test against it is a trap. Return the routes module alone and unpack a single value at the call sites. * Return only the paths from the shared archive and record setup helpers _shared_setup_7 built its zip inside `with zipfile.ZipFile(archive, "w") as zf` and returned that handle; _shared_setup_10 did the same with the file object from `with open(...) as f`. Both are closed by the time the caller sees them. Nothing fails today because the four zip call sites immediately shadow `zf` with their own `with zipfile.ZipFile(archive) as zf`, and the three record call sites never touch `f` at all. Both helpers are new here, so this is a trap we would be creating rather than inheriting. Return the archive path and the root directory alone, and unpack a single value at the seven call sites. * Trim comments in the folded test files Drop the generated per-helper setup narration and collapse multi-line case notes to one line where a single line carries the same fact. Comments only; the parsed AST of every touched file is unchanged. --------- Co-authored-by: danielhanchen <elliegouldingstuff@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-09-09 17:11:13 -07:00
Daniel Han	971eb36f48	Trim comments in tests, scripts, CI helpers and unsloth_cli (#10113) ... * Trim comments in tests, scripts and CI helpers Comment-only pass over 453 changed files under tests/ (excluding the studiobench harness), scripts/ and .github/scripts/. 17,749 -> 10,635 comment lines (-40.1%) 1,183,079 -> 757,190 comment characters (-36.0%) Distinctive-fact retention against the base commit: 93.2%. A sample of 100 restoration and deletion decisions was judged by hand: 98 correct. No comment line exceeds 120 physical characters. Nothing outside comments changes: no code, docstrings, string literals, licence headers or pragmas. Verified with an AST code-only signature of every file against the base. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Put the moved comments back on the code they describe The signal-handler rationale in launch.py had been reattached to the handler registration loop, which has no try/finally, and its except-branch note had been split into a fragment starting mid-sentence with the noqa marker duplicated into the prose. The PORT_FILE launcher comments had been reattached to the _studio_root_id caching test, which does not touch PORT_FILE. * Put the section banners back, and use plain dashes in comments The trim removed the rule-line comments that index these files. Tests locate a region by finding its banner and end it on the next one, so a missing banner is a broken anchor rather than lost decoration. Each is restored above the code line it preceded at 1d2679f; where that anchor is not unique the banner is left out rather than guessed at. Em and en dashes are rewritten as plain dashes inside comment tokens only. String literals are code and are untouched, so the diff stays comment-only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Put reattached comments back on the code they describe The comment pass moved a number of explanatory comments off the code they were written for and onto unrelated statements nearby, so they now read as false or as truncated fragments. This restores each one at its correct site: - lint_duplicate_definitions: the import-key rationale goes back on the emitted Finding key instead of being half-pasted onto `implicit =`. - lockfile_supply_chain_audit: "Empty by default" belongs on the empty CARGO_IOC_STRINGS tuple, not on the non-empty source allowlist; the missing-lockfile and traceback notes go back on their own branches. - scan_packages: "archives landed in dest" describes the successful pip download, not `if not downloaded:`; the backslash-continuation note goes back on the continuation check in _scan_line_end. - interrupted_install_probe: the start_new_session/getpgid note belongs on the posix branch that calls getpgid, not on the win32 branch. - virustotal_scan: the missing-secret rationale goes back on the skip path and the :⚠️: note back on the detection print. - scan_npm_packages: the rmtree cleanup note goes back on the rmtree. - check_frontend_dep_removal: one per-key rationale per key again, instead of five concatenated onto browserslist. - profile_startup: failed-launch and nothing-measured notes swapped back. - enforce_kwargs_spacing: fold-rejection reasons back on the guards that reject for them, and the f-string fold comment back in order. - tests/conftest: the bitsandbytes broken-wheel note back on the bitsandbytes handler. Comments only; no code, string literal, docstring or pragma changes. * Repair the banner restore: drop duplicate headings, keep the blank line The scripted restore re-added each section banner without removing the de-bannered plain heading the trim had left behind, so a heading appeared twice, once ruled and once bare. It also inserted the banner flush against the next line of code where the base had a blank line after it, which turns a section separator into a doc comment for whichever def follows. Both are the same fault the trim itself was being screened for, introduced by the remediation rather than by the trim. * Reattach 12 review comments to the code they describe Each of these was a rationale that ended up on a neighbouring test or assertion, or a fragment cut mid-sentence, so it described something other than the line under it. - test_gguf_disk_preflight: the 3GB-embeddings arithmetic belongs to the untied case, not the tied one (2GB there). - lockfile_supply_chain_audit: reassemble the workflow-command escaping rationale on the skip branch. - studio_smoke/multi_turn_chat: restore the retry-policy explanation in main(). - test_virustotal_scan: restore the advisory-posture rationale on the scan step. - test_amd_fastpath_probe: return each ROCm rationale to its parameter row. - test_cuda_repair: the Blackwell span note goes back on the Blackwell test; complete the Kepler sentence. - test_windows_rocm_bnb_version: realign the sitecustomize / opt-out rationales with their cases. - test_dataset_num_proc: the Pool(1) note belongs to the low-memory auto path. - test_install_whisper_prebuilt_logic: decoy-DLL, libomp, build-bump and same-ggml notes back on their own tests; the link copy fallback needs none. - playwright_loaded_models_indicator: stale-eject notes back on their sections and the preference banner restored. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the review comments that were cut mid-sentence or reattached to the wrong code * Restore the comment tails the 120-column pass cut off The trim enforced the column limit by chopping the physical line rather than rewrapping it, so a block's last sentence stopped mid-thought and the clause carrying the reason was gone: -# ...fails in a second rather than +# ...fails in a second rather than after the quantise and the multi-gigabyte save. -# fp8 / mxfp8 need bf16 weights, so skip non-bf16 Linears; +# fp8 / mxfp8 need bf16 weights, so skip non-bf16 Linears; nvfp4 handles fp32. Mirrors the runtime gate. 883 blocks across 266 files, restored from the merge base and rewrapped at word boundaries within 120 columns. Blocks that end on proper punctuation followed by a new sentence are left alone: those were shortened on purpose, and only a base that continues the same sentence where the head stops counts as a cut. No word is invented; every restored word comes from the base. Pragmas, shebangs, coding lines, licence text and rule-line banners are skipped. Dangling truncations drop from 336 to 77. Full suite is 65 failed / 18683 passed, matching main exactly, so no test changes behaviour. * Restore comment tails cut inside a parenthetical or abbreviation The first repair pass keyed on terminal punctuation, so a block cut after "(e.g." or mid-code-span looked finished and was left alone. Balance the brackets and backticks instead, and treat a trailing abbreviation as mid-sentence: -# ...skip the whole module up front on runners without it (e.g. +# ...skip the whole module up front on runners without it (e.g. CPU-only) before importing unsloth, otherwise +# collection errors instead of producing a skip. 22 further blocks across 20 files. _ts_deps.py is fixed by hand: it had also lost a sentence from the MIDDLE of the block, so it was not a prefix of the base text and no prefix rule could reach it. * Reassemble the review comments that were left scrambled or cut short Restore the full rationale for the comments the trim pass left as reversed or truncated fragments, and put the openenv, reasoning-effort and recovery window notes back on the code they describe. * Put the scrambled comment blocks back over the code they describe * Put the inline labels back on the entries they label * Restore an inline label lost in the merge * Restore the facts the reworded comments dropped * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Move the displaced notes back onto the tests they describe * Re-pair the fixture rationales with their fixtures * Put the relocated notes back and finish the sentences the trim cut * Restore the dropped facts and correct the ones that were never true * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Unfuse the merged notes and drop the reformatting the trim smuggled in * Restore the clauses the trim cut off and unfuse the notes it merged Thirty sites where a comment was shortened past the point it still said anything. Most ended on a colon or a dangling word, so the reason the branch exists was gone while the sentence introducing it stayed. The rest had absorbed a neighbouring test's note, which left one row documented as another row's behaviour. Each restored clause was checked against the code under it rather than copied back on trust. Three were not restored: a pointer to a measurement that is not in the function it names, a claim that a missing import fails as NameError when it fails at collection, and a redundant restatement in the autoscroll check. tests/kaggle/t4_smoke/training_evidence.py ends up identical to main, which is the right outcome for a file whose only trimmed sentence was load-bearing. * Give back the halves of the comments the trim cut off Two hundred and twenty comments across a hundred and twenty files had been shortened to the sentence that introduces a fact and none of the fact. A note ending on a colon or on "so" or "carried over" still reads as a comment and still parses, which is why every structural check has been passing over them: a truncated comment is a well-formed comment. Most of these were only findable by asking what a block ENDS on rather than what it starts with. The seven passes before this one all keyed on the first line, so they were blind to the commonest damage in the whole trim. The supply-chain audit had seven "Mini Shai-Hulud May-12 wave:" headers with the package names deleted, which was the entire content of each one. A handful were worse than truncated. A "can't fit on one line, so leave it" note had moved from the test asserting changed is False onto the one asserting changed is True; the sm_61 and sm_70 rationales in the CUDA repair tests were stacked on a test that exercises neither, leaving their own tests bare; an unpinned-latest note sat on a test that stubs a working install on disk. Restored clauses were checked against the code under them, not copied back on trust. Around a dozen were left out or rewritten because the base was wrong about the code as it stands: a pip stub credited with an append it does not do, five exports said to be filled by a loop that fills one, a __dict__ rationale that argues for the opposite of what the loop needs, and UI strings quoted as current that the sources no longer use. * Put the fused row notes back on the rows they are true of Ten sites where a comment described a sibling. The torchao skew table had three rows' notes concatenated onto the 0.18.0+cu130 row; the revision-forwarding table had the adapter-tokenizer exception one row below the only row it holds for; the vision LoRA pair had the tied and untied cases on a single anchor. Four of these were mine, not the trim's. The 120-column rewrap left an orphaned continuation line sitting on a class, one repair copied an untagged-build note onto a fixture that pins a URL and has no untagged build in it, another duplicated a namespace-package note and pushed the interrupted-install label onto the wrong row, and a fourth moved a collision note onto a fixture with a single host, where it is false. A pass that rewrites 220 comments will get some of them wrong, and these are the ones it got wrong. Two reported sites were left alone. The masking note in the VirusTotal client already carries its full clause, reflowed; the IOC scan comment reads complete, and its missing "4." went with the numbering on every other step in that function. * Move the row and branch notes back onto what they describe Nineteen sites. The token-count parity table had every row comment one param too low, so each seeded reasoning style was documented as the next one; the font-scale contract had two rationales slid down a continue each; the validator fixture banner for R-INST-001 sat under R-INST-005, leaving three tests unlabelled and two double-labelled. Three of these I caused. The parity table was misplaced by the previous commit, which moved fused notes onto rows without checking the seeds underneath them; a timing bound landed in a test that has no timing assertion; a restore note was reinserted between the two commands it claims to cover instead of above them. The pacing benchmark had lost its measurement table entirely, including the #7892 result that is the reason there are two budgets rather than one headline number. Restored with the columns intact. * Undo the damage the last four repair passes did Twenty-five sites, and almost none of them were the trim's. Round nine was called "move the row and branch notes back onto what they describe", and it moved a good many onto things they do not describe: the xpu swap tests had the pinned-mirror rationale on the test that asserts the index is dropped, the cuda repair file had a whole run of rationales slid one test upward, and several notes were duplicated onto a neighbour rather than moved. Where a clause was already correct on its real anchor, the copy is deleted rather than parked somewhere new. Separately, this PR had been changing code, and the gate could not see it. Five files: three where a repair split a tuple across lines to make room for a restored trailing comment, two where the trim deleted a trailing comment and then joined up the wrapping that existed only to carry it. Parentheses and line breaks leave the parse tree untouched, so an AST comparison calls all of it comments. Every one is back to base byte for byte, with the comments that justified the wrapping restored. scripts/code_reflow.py is the check for it: same tokens in a different arrangement is a code change, whatever the AST says. * Let ruff rejoin the two lines the shorter comments freed Both were split only to make room for a trailing comment. Round ten cut those comments back to what is true of the row, the expressions now fit, and the formatter joins them. Running it is not optional: pre-commit.ci was red on exactly these two. * Delete the stray copies rather than move them again Five sites. Three were duplicates: the correct sentence still sits on the test it describes, and a repair pass left a second copy on a neighbour. Moving those would have been the fourth round of shuffling one clause between two tests, so the copies are simply gone. The mislabelling was the point in each case. An empty handover tag and an absent one are different precedence rules, and the PowerShell 7.5 parsing note had been copied from the empty-string test onto the one whose variable is unset. A non-zero-exit rationale sat on a parser test that asserts a successful exit. The other two are the trim's own: "#5854)," with its subject cut away, and a regex description that stopped before naming the self. prefix it matches, which is what explains the narrower filter below it. * Put the nine slid notes back on what they assert Seven had moved one branch or one call down. The autocast one had landed on the case that forces float32 and asserts dtype float16, directly under a note saying there is no dtype at all; the JAX rationale sat on the USE_FLAX env-var case, whose module map has no jax in it; the kill-and-sentinel note sat on a test that only checks a GPU row regex. Each is back on the assert it describes, with a tenth found in the same file as the probe timeouts. Two are the trim's own. A mid-block deletion in the settings-tabs driver spliced two half sentences into "blocking a / error-handling regression and was a stale constant"; the macOS slot table lost the only statement of why macos-14 is absent from a set named for images GitHub still schedules. The retirement dates are restored, verified against all three workflows that carry them. The settings-tabs note is shorter than base on purpose. It now says the invariant, that blocking a panel must not collapse the dialog, and drops the tab count, which nothing asserts. * Move twelve more notes back onto the branch they are about All twelve had slid onto an adjacent test or branch, and in each the displaced note contradicted what it now sat on. The CPU-backend note about a wheel being left alone had landed on the test asserting a repair runs; the XPU one about a dispatched shard sat on the single-device case; the redirect-body note sat in a test that never binds served; the draft-state refusal sat on the test where the target is present and the run returns 0. One lost text on the way as well: the missing-target rationale had been cut to its first two sentences, dropping the reason the draft and prerelease refusal above it would otherwise degrade into no check. Also a rewrap that duplicated its own tail instead of replacing it, so the merge-budget note read "else a sharded own device, not GPU0, else a sharded model OOMs". * Return six notes to the test whose behaviour they describe Eight reports, six defects: two pairs described the same misplacement from either side. One was worse than misplaced. "A comment naming a probe is not a probe" had moved onto the metadata scan, whose predicate matches the raw line with no comment handling at all, so it claimed an exemption that scan does not implement. It belongs on the Test-Path scan, the only one that splits on # before matching. Moved, not reworded. The rest argued against the test they sat on. The lazy-trl note was on a stub with no __getattr__ to raise; the bogus-variant note was above a request sending a valid variant; the imported-TensorFlow note was on the case that supplies no tensorflow and asserts the flag is cleared; the 62-token rationale was on the test that asserts 12. * Move ten more notes onto the statement they explain Seven had slid off their anchor, and in most the stray was also cut short, so the fix was to restore the whole sentence where it is true and delete the fragment. The torchao dependency reason sat on a test about re-verification rather than on the --no-deps assertion it justifies; the bitsandbytes heading sat on a test that fetches quantizers_utils.py; the no-selection latch note sat on the delete block, which asserts nothing reaches the action. Two are the trim's own truncations. The stand-aside rationale in the shortcut driver stopped at a colon, losing why Enter on a focused Deny button would otherwise be defaulted away into approve. The kwargs-union case stopped at a semicolon, and the trim had eaten the leading ** from **kwargs as well. One was about the driver rather than a test: the JSON note sat beside the final print, where it described the document, instead of on the stderr redirection that is what actually keeps stdout clean. * Restore what the reflow cut, including three markers it mangled Nine sites. Most are the trim's own truncations: the dill fixture had lost the property that makes lookup by reference fail, so an otherwise unexplained __module__ = "builtins" sat there with no reason; the rope drift test had two clauses spliced into "every buffer keeps scaling in a buffer"; the playwright guard had lost the three command shapes it matches and why the regex is not anchored on an engine list. Three are a shape I had not seen. The reflow rewrote the `#:` variable documentation markers in the fast-copy constructs to `# :`, and dropped stray colons into the wrapped prose, leaving "text : node". Those markers are syntax, not punctuation to be tidied. One was misplacement rather than truncation. The dotted-import rationale had moved onto the second copy of its case and lost its tail, leaving a dangling "and" above a capitalised "Still caught" and the first row bare. Restored where base has it, stray deleted. lint_duplicate_definitions self-test still passes, 63 cases. * Put five drifted comments back on the code they describe The trim slid several notes onto neighbouring tests, where they describe something they no longer sit on: - test_qat.py: the TODO moved off _test_full_model_fake_quantize, whose leading underscore is what keeps pytest from collecting it, onto a test that is collected and was never disabled. - playwright_chat_ime_i18n.py: 6f carried a stray copy of 6e's compositionstart note above an Enter KeyboardEvent, and had lost its own rationale for why Enter reports isComposing=false while composingRef is pinned. - test_install_node_prebuilt_logic.py: the short-circuit note landed on the transient-download-failure test, leaving the SHA-enforcement test bare and dropping the carve-out that a different usable version is still kept for offline resilience. - test_chat_autoload_failure_gate.py: the default/namespace note moved onto the named-import loop, and the mixed-form note explaining that regex's optional prefix was dropped. - test_frontend_dep_removal.py: the unquoted CSS url() note moved onto the quoted fixture, and the HTML package-name-boundary note moved off the script-src case onto the link-href case. Comments only; the gate reports code unchanged on all five files. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <michaelhan2050@gmail.com>	2026-09-04 15:31:50 -07:00
Daniel Han	c303050c67	Studio: report the GPUs the OS sees, not just the ones PyTorch opened (#9858) ... * Studio: keep a Windows update from leaving a CPU-only PyTorch The in-app updater runs `unsloth studio update`, which runs studio/setup.ps1, which runs studio/install_python_stack.py. install.ps1 is never on that path, and install.ps1 holds the only PyTorch flavor repair there has ever been. Meanwhile the dependency steps in install_python_stack.py install with deps and without an --index-url, so the resolver's default source is PyPI, whose Windows torch is 2.11.0+cpu, and constraints.txt names no torch to stop it. Both Windows repair passes are disabled there: _ensure_cuda_torch returns early on Windows, and the step 13 repair block is gated on `not IS_WINDOWS`, even though its own comment says the steps above can pull torch from PyPI. The result is a cu124 venv that comes out of an update as 2.11.0+cpu with the update reporting success. The app then runs everything on CPU, Settings shows no GPU, and the bundled CUDA llama-server prints "Available devices: (none)" because a +cpu venv ships no cudart64_*.dll for it to load. Add a Windows torch flavor invariant as step 13w. It resolves the flavor the install actually selected (env var from setup.ps1, then the install manifest, then a live probe), compares it against what is in the venv, and reinstalls the bounded trio from the correct index on a mismatch. It prints install.ps1's line verbatim so a support log from either path reads the same. If torch is still CPU-only afterwards on a host that expected a GPU build, the install now fails rather than reporting "dependencies up to date". It is conservative by construction: no-torch mode, a deliberate UNSLOTH_TORCH_BACKEND, a non-CUDA or absent expectation, a hidden GPU via CUDA_VISIBLE_DEVICES, and a torch that is missing or unimportable all no-op. Also stop the update path deleting a healthy venv. setup.ps1 collapses the expected tag to "cpu" whenever its bounded nvidia-smi probe comes back empty. On a direct update none of the existing escapes apply, so a working cu124 venv was removed and setup then aborted telling the user to run install.ps1, with no rollback copy because only install.ps1 keeps one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: unset the published torch flavor portably The publication block relied on `$env:X = ""` deleting the entry. That held on Windows PowerShell 5.1 and on 7.0-7.4, but 7.5 picked up .NET 9's change: an empty string is now a real value and only $null removes the name. The comment asserted the old behaviour as fact. Both spellings still read as unset on the Python side, which tests truthiness rather than presence, so nothing was broken today. The point is that a setup script should not publish a value whose meaning depends on which PowerShell the user happens to have. Remove-Item behaves the same on every version. Deleting rather than blanking matters on its own too: a value inherited from the caller's shell must not survive a run that decided it cannot name this host's flavor, or the stack enforces a stale expectation against a fresh venv. Found by running the block under a real PowerShell 7.6 over the host matrix. * Studio: close three gaps in the Windows torch flavor invariant Three separate holes in the invariant added earlier in this PR, all reachable. Do not route around an explicit index pin. When a pin's leaf names no flavor (a corporate /simple mirror), setup.ps1 publishes nothing, so the expectation fell back to the manifest, which records whatever was installed BEFORE the pin was set. Repairing off that stale tag rejects the pinned URL, because its leaf does not match, and reinstalls from the public pytorch index: it overrides an administrator's deliberate package source and fails outright on a network-restricted host. Decline instead, on the same test _ensure_cuda_torch already uses. Enforce the published XPU flavor, not just CUDA. setup.ps1 publishes "xpu" for an Arc host and installs the XPU trio before handing over, so treating that expectation as a no-op left the invariant carrying an answer it refused to act on. The exposure is identical: the dependency steps re-resolve torch from PyPI, and _ensure_xpu_torch cannot clean up afterwards because step 13's whole repair set is gated off Windows. Repairs with the XPU trio, whose floor is 2.6 rather than 2.4, since unsloth raises at import for an XPU device below that and a 2.5 wheel would satisfy the CUDA range and be kept. ROCm stays with the path that owns it: its wheels come from a per-architecture repo.amd.com index that cannot be reconstructed from a generic "rocm" tag. Classify a CUDA venv from disk when the torch probe hangs. A wedged display driver faults or hangs `import torch` on CUDA exactly as a faulted HIP runtime does on AMD. XPU and ROCm both rescue that venv by reading torch/version.py, but CUDA had no such arm, so the probe-failure chain fell through with a null tag, the no-wipe escape could not see a cu* wheel to preserve, and a direct update deleted a healthy CUDA environment before aborting. Keeps the family rather than a flat "cuda", because the stale comparison it feeds is cu126-vs-cu128, and matches with [regex]::Match so $Matches in the caller's scope is untouched. One existing test asserted the XPU no-op and has been updated: it was pinning the behaviour above, not guarding it. * Studio: enforce the published ROCm flavor too, by delegation The invariant enforced cu* and xpu but treated a published "rocm" expectation as a successful no-op. That left the AMD Windows path with the same hole the whole change exists to close: _ensure_rocm_torch runs at step 2b, which is BEFORE the dependency steps that can resolve torch back to PyPI's CPU wheel, and nothing ran it afterwards, so the update could still write a successful CPU-only manifest. The reason given for excluding it was that AMD's Windows wheels live on a per-architecture repo.amd.com index that a generic "rocm" tag cannot name. That much is true, and it is why this pass still does not build a ROCm URL of its own: setup.ps1 hands over $TorchInstallIndexUrl, which on the AMD path still points at /cpu. But it is not a reason to leave the flavor unchecked, because _ensure_rocm_torch already owns the whole problem -- it detects the arch, maps it to an index, honours an explicit ROCm pin, and re-probes the stale UNSLOTH_ROCM_TORCH_INSTALLED marker rather than trusting it. So the ROCm arm delegates to it and then applies the same verdict as the other two: still CPU-only afterwards fails the update rather than reporting success. The warning block is now a helper shared by both exits, so the three flavors cannot drift apart in what they tell the user. * Studio: do not skip the flavor invariant on an explicitly pinned GPU family The two gates at the top of _ensure_expected_torch_flavor both fired on a host that had pinned XPU or ROCm on purpose, which is the opposite of what the XPU and ROCm enforcement added just before this was for. _TORCH_BACKEND is not only install.sh's export: when it is empty, this module derives it at import from UNSLOTH_TORCH_INDEX_URL / _FAMILY, so a pinned xpu or rocm index sets it to "xpu" or "rocm" and the `not in ("", "cuda")` gate then skipped the only post-dependency Windows invariant there is. Those two values now fall through and are re-tested once the expectation is known: a GPU backend that DISAGREES with what setup.ps1 published is still a no-op, because the pin is the newer and more specific instruction, while one that agrees is enforced. "cpu" and unrecognised values still exit before the expectation is resolved, so that path costs no GPU probe. The second gate was a flat veto on _explicit_unknown_family_torch_index_url(). That helper's known set is rocm*/gfx*/cpu/cuXXX -- it predates XPU and does not count "xpu" as a family -- so an explicit XPU pin read as "unknown" and skipped the very host it was set for. It is now compared against the expectation instead: a pin whose leaf IS the family being enforced is not an unknown pin, and it is also the correct index to repair from. Widening the helper itself would have changed _ensure_cuda_torch and _ensure_rocm_torch too and drifted from install.sh / setup.ps1 / install.ps1, which it is documented to match. The parametrized deliberate-backend test still passed throughout, because it pairs a rocm/xpu backend with a cu124 handover, which is the disagreement case. It is split so the two reasons are stated separately rather than one standing in for the other. * Studio: let an explicit torch index pin outrank the recorded flavor The manifest records what a previous install left behind; an index pin is the instruction for the run happening now. Resolving the manifest first meant a freshly set UNSLOTH_TORCH_INDEX_FAMILY=cu128 over a cu124 manifest resolved to cu124, and _expected_torch_index_url then rejected the cu128 pin as a family mismatch and repaired from the public cu124 index, undoing both the family and the source the user had just selected. The pin now resolves second, after the setup script handover and ahead of the manifest. AMD leaves collapse to "rocm" so the tag matches the installed flavor vocabulary, and a leaf naming no family still falls through to the manifest, so the unknown-pin escape added earlier is unchanged. * Studio: keep the Windows ARM64 no-torchaudio exception in the flavor repair No win_arm64 torchaudio wheel is published on any index. setup.ps1 drops it from all four of its install trios, but the new final repair asked for it unconditionally, so a cu* expectation reaching that path on Windows on ARM turned a repairable venv into a failed install. The repair now drops torchaudio on the same condition, for the CUDA and XPU trios alike, using the _is_windows_arm64 helper the script already carries. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: verify the delegated ROCm repair, and enforce a pinned CPU index Two ways the invariant could report success over an environment that never got what it asked for. The ROCm arm accepted any GPU build. _torch_build_is_gpu is deliberately family-blind, so a transient repo.amd.com failure inside _ensure_rocm_torch, which is non-fatal there, left the old cu124 or xpu wheel in place, passed the check, and the manifest recorded rocm over it. The arm now re-probes the flavor and requires the family it asked for, with a distinct warning for a wrong family and the existing CPU-only warning kept for a venv that came back CPU. A venv nothing could read still passes, because ambiguity must not fail an update by itself. An expectation of cpu was never enforceable, but the unpinned dependency steps honour UV_TORCH_BACKEND, so a venv deliberately built against /cpu can come out of them holding a GPU wheel. A pinned CPU index is now enforced, and only a pin: setup.ps1 also publishes cpu when its nvidia-smi probe returns nothing, which is the wedged-driver host whose healthy cu124 venv the no-wipe escape in that file exists to protect, and acting on that handover would force it down to CPU. The verification for that case compares the family rather than asking whether a GPU build is present, which is the opposite question. * Studio: verify the repaired family, and re-settle what the torch release pins Four ways the invariant could still finish over the wrong environment. An untagged wheel under a CPU pin. _torch_flavor_tag reads every untagged version as cpu, which is right for PyPI and wrong for a private index serving an untagged CUDA or ROCm build: that wheel compared equal to a cpu expectation, skipped the repair, and was recorded as cpu. The runtime probe already carries the markers _ensure_cpu_torch uses to tell them apart, so the comparison consults them. The non-ROCm tail accepted any GPU build. A misconfigured mirror can answer a /cu128 request with a cached cu124, rocm or xpu wheel, and _torch_build_is_gpu is family-blind, so the update exited 0 and recorded the requested tag over a build that never arrived. The tail now re-probes the flavor and requires the family it asked for, as the ROCm arm already does. A CPU-only outcome keeps its own warning, since the remedy differs, and an unreadable venv still passes. torchao and xFormers are re-settled, but only when the repair actually moved the torch release, since the common case restores the wheel the venv already had. torchao's cpp extensions are release-specific and step 4 chose its pin from the torch this repair replaced, typically 0.17.0 for a 2.11 that the <2.11.0 spec takes down to 2.10, whose matched build is 0.16.0. xFormers is stricter: its _C.pyd is linked against one exact torch and CUDA pair, and beside any other pair the load is downgraded to a log line, so the import succeeds with memory-efficient attention, SwiGLU and the sparse ops silently gone. This script never installs xFormers and cannot reconstruct install.ps1's wheel selection, so a mismatched one is removed, which is what the backend's own resolver concludes as well: torch SDPA beats an extension that cannot load. Neither half can fail the update, both being secondary to the defect the repair has just fixed. * Studio: stop the post-repair resync from undoing the repair torchao depends on torch, so the --force-reinstall I added last round resolved dependencies and could re-resolve torch from the unpinned default index, which on Windows is the 2.11.0+cpu wheel the repair had just removed, reinstalled moments after removing it and after the verification had already passed. The update would then exit 0 and record a GPU tag over a CPU-only venv, which is the exact defect this whole change exists to close, reintroduced by its own follow-up step. The reinstall now passes --no-deps, which is not an optimisation here but the safety of the call: torch and the rest are already present and correct at that point. The helper additionally reports whether it ran anything that could touch torch, and the caller re-verifies the flavor when it did, so a regression cannot be silent even if the flag is ever wrong. The xFormers half was gated on the release moving, but its _C.pyd is linked against the exact torch and CUDA pair, so 2.10.0+cu124 to 2.10.0+cu128 after an index family change keeps the release and still leaves an extension that cannot load. The two are now gated separately: torchao on the release, xFormers on the full build label. * Studio: re-pin torchao when the CUDA major moves, not just the release _select_torchao_spec branches on cuda>=13 as well as on the release, so a repair that takes a venv from cu124 to cu130 at the same torch release changes the matched build while the release comparison alone reports nothing happened. The 0.16.0 extension cannot load beside a cu130 wheel, which is the same silent drop to the slow fallback the release gate was added for. The gate now fires on either. A flavor move within one CUDA major, cu124 to cu128, still leaves torchao alone, since nothing about its matched build changed there. * Studio: report the GPUs the OS sees, not just the ones PyTorch opened Every GPU field in the System tab starts from get_device(), which is torch-only. The moment torch.cuda.is_available() returns False the whole chain goes quiet: get_backend_visible_gpu_info() and get_visible_gpu_utilization() both short circuit on their DeviceType gate, so the nvidia-smi probe that would have listed the cards correctly is never reached, and the UI says "No visible GPU" on a host with two working A4000s. CHAT_ONLY_REASON is set to "no_gpu", which the sidebar renders as "needs an NVIDIA or AMD GPU" - the opposite of the truth. This is the Windows shape of #8473 (reported there on AMD/Linux, also #7485 and #7307): a managed venv whose torch was replaced by PyPI's 2.11.0+cpu, with nvidia-smi listing every card throughout. Ask the OS as well as PyTorch: - get_physical_gpu_inventory() probes nvidia-smi, and on Windows the DirectX adapter registry that _rocm_windows_per_device_vram already trusts. It runs regardless of DEVICE, is bounded, never raises, and is cached with a TTL because a driver restart or an eGPU can change the answer. - classify_torch_build() separates a CPU-only wheel from an accelerator wheel whose runtime will not initialise. The advice differs, and telling someone with a healthy cu124 wheel to reinstall torch sends them the wrong way. - The CPU fallback in _detect_hardware_locked now reports "torch_cpu_build" or "torch_cuda_unavailable" instead of "no_gpu" when the host does have GPUs. An emptied CUDA_VISIBLE_DEVICES / HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES is a deliberate hide, not a mismatch, and is excluded. The inventory is display-only. physical_devices and mismatch sit BESIDE devices in the payload and are never merged into it: devices is the runtime-usable list that the model-fit estimate budgets against and that the training device picker pins from, so a card torch cannot open must not become selectable. Also adds a diagnostic for the second half of the same report. The CUDA llama.cpp prebuilt takes cudart64_*.dll / cublas64_*.dll from the managed venv, so a CPU-only torch leaves it with nothing to load and llama-server prints "Available devices: (none)" while the prebuilt marker still says backend cuda. That was entirely silent; it now logs once per binary and changes nothing else. Finally, the repair the banner points at. Both affected users were fixed by rerunning the installer, and one asked for a button. start_managed_repair gains force_installer, which skips the update attempt: an update reuses the environment it finds, so a venv on a CPU-only wheel survives any number of successful updates. Settings > General grows a "Repair installation" row next to the existing reset row, wired through a small context to the same startRepair the startup screen uses, so it renders the same repairing UI and restarts the backend afterwards. * Studio: four corrections to the GPU mismatch reporting Extended CPU wheel tags. PyTorch publishes CPU builds whose local version carries a suffix, such as 2.8.0+cpu.cxx11.abi. An exact match on "cpu" classified those as accelerator wheels whose runtime failed to start, so the UI pointed the user at a driver instead of at the reinstall that actually fixes it. nvidia-smi off PATH. A driver install can leave nvidia-smi.exe in the NVSMI directory or the driver store without putting either on PATH, and the bare name then raises FileNotFoundError, leaving the physical inventory empty on exactly the host this inventory exists for. Resolution now falls back to the same two locations setup.ps1 already tries. Export and Video capability. Both render their message verbatim, and neither knew the new reasons, so a host with two working A4000s was told no supported accelerator was found, contradicting the inventory the same server had just published. Both now carry a reason-specific message that names the installed build and points at the repair, with the driver case worded differently from the CPU-build case. The llama.cpp CUDA runtime warning. It scanned only the directories prepended to the search path, but the inherited PATH is appended immediately afterwards and Windows resolves from it, so a hand installed CUDA toolkit was told to repair a working installation. The check now runs against the full search path. Also hides the Settings repair row for an externally started backend: start_managed_repair refuses that mutation, but startRepair swaps the shell to the repairing screen before the refusal arrives, leaving a connected user on the repair-error screen. * Studio: inventory Linux AMD cards, and stop three false mismatch verdicts Linux AMD had no physical inventory at all. nvidia-smi contributes nothing on that host and the AMD branch was Windows-only, so a ROCm wheel replaced by a CPU one left the inventory empty, the verdict at no_gpu, and the System tab still claiming there is no GPU, which is the AMD/Linux shape of the report this feature exists for. Added a sysfs walk of /sys/class/drm: vendor 0x1002 and a byte-valued mem_info_vram_total are both documented amdgpu interfaces, and the kernel driver publishes them whenever a card is bound. sysfs rather than amd-smi deliberately, because amd-smi is separate ROCm userspace and this is the host whose ROCm install is in question. No name is reported, since the kernel publishes none and borrowing an amd-smi row whose ordering is not guaranteed to match this walk would attach the wrong name to a card. Visibility masks are now applied only where they can hide something. Windows HIP has no ROCr layer, and this module's own resolver already ignores ROCR_VISIBLE_DEVICES there, so a stray empty one restored the no_gpu verdict on a Windows NVIDIA host with a CPU wheel. The HIP-layer variables are consulted only when an AMD card is actually in the inventory. An inventory that answered nothing keeps every variable, because unknown has to stay conservative. A deliberate CPU install is no longer reported as broken. Pinning /cpu on a machine that still has a GPU leaves no mask empty, so the wheel the user asked for was classified as a fault and the UI offered a repair whose only effect would be to replace it. An explicit CPU pin, or a cpu flavor recorded in the venv manifest, now suppresses the CPU-build reason. Only cpu counts: an absent or unknown record is not a choice. The dead-accelerator reason is untouched, since a wheel whose runtime will not start is a real fault whatever was recorded. * Studio: record the flavor off Windows, and keep the GPU verdict current Three ways the deliberate-CPU and mismatch reporting could still be wrong. The manifest carried no flavor on any non-Windows install, because the tag was only ever resolved inside the Windows branch. A Linux GPU host installed with a transient explicit CPU pin therefore recorded nothing, and on the next launch, with that variable gone, the backend saw a CPU wheel beside a physical GPU and offered a repair that would reverse the choice. The tag is now resolved on the other platforms too, for the record only: nothing outside Windows enforces it, since step 13's repair set owns those hosts. A token-authenticated CPU pin was not recognised. Credentials in the query are supported and common, and a raw final-segment split saw cpu?token=... rather than cpu, so exactly the deliberate CPU install this was meant to protect was still reported as broken. The leaf is now read the way the installers read it, query and fragment stripped, trailing slashes off the path only. The chat-only verdict never followed the inventory. detect_hardware runs once at startup while the inventory refreshes on a 60 second TTL, so an eGPU attached after launch, or a driver that finished restarting, left the sidebar and the Export and Video pages insisting no accelerator exists while /api/system listed the card and published a mismatch. A card that goes away produced the mirror image. current_chat_only_verdict re-derives the three inventory-sensitive verdicts and leaves every other one exactly as detection set it, since a 60 second probe cannot change those and re-deriving them would fight detect_hardware rather than follow it. * Studio: inventory Intel GPUs, keep unknown distinct, and stay off the request path Intel was missing from both inventories. An Arc host whose XPU wheel was replaced by a CPU one is exactly the shape this reporting exists for, and nvidia-smi contributes nothing there either, so the inventory came back empty and the mismatch was discarded. The Windows DirectX registry scan now takes a vendor id, defaulting to AMD so every pre-existing caller is unchanged, and the Linux sysfs walk accepts 0x8086 beside 0x1002. Intel publishes no vram total on the discrete path, so an Arc card is reported with an unknown capacity rather than left out: knowing the GPU exists is the point. An unknown inventory is no longer collapsed into an empty one. nvidia-smi timing out or exiting non-zero comes back as a structured empty result, not an exception, so a transient failure read as the GPU having disappeared and retired a settled mismatch to no_gpu for a whole cache interval, handing the user the opposite advice. The probe now reports unknown when nothing was found AND some probe declined to answer, and the refreshed verdict keeps what it had in that case. The refreshed verdict no longer runs a subprocess on the request path. /api/health and /api/liveness both read it, the NVIDIA half shells out with a 10 second timeout, and the lock makes concurrent callers queue behind it, so a hung or restarting driver, which is the host this feature is for, could stall the event loop for ten seconds against a 2 second desktop timeout every time the TTL expired. Non-blocking callers now take the last known answer and hand the refresh to a daemon thread, single-flighted so a second caller does not queue a duplicate probe. Startup detection and /api/system still block, which is what keeps the cache warm for everyone else. * Studio: corroborate Windows adapters, exclude plain Intel iGPUs, re-detect on recovery Registry records are no longer trusted on their own. The DirectX registry outlives the hardware, which setup.ps1 states outright and works around by using these records only to relabel an adapter its live WMI scan also returned, never to add one. A CPU-only machine with a driver record left behind was being told it had an unusable GPU and offered a repair that cannot restore absent hardware. Each record is now joined against Win32_VideoController, the same live source setup.ps1 scans, by substring in either direction because the two spell the same card differently. A live scan that cannot answer contributes nothing and marks the inventory unknown, which keeps a settled verdict rather than inventing one. An ordinary Intel iGPU no longer establishes a mismatch. setup.sh does not autodetect Linux XPU at all and setup.ps1 limits automatic XPU detection to Arc and Data Center GPU by name, so an Intel UHD beside a CPU wheel is the correct state of that machine, and counting it reported a broken install and offered a repair that would reinstall the same CPU build. Intel now counts only when the name matches setup.ps1's rule, or an XPU flavor was pinned or recorded for this venv, or the installed torch carries an XPU runtime. The last two matter because the Linux sysfs walk publishes no name. The cards stay in the inventory either way, since the OS really does see them; what changed is what may be called a fault. A recovered accelerator now retires the cached verdict. The refresh added earlier updates only the reason and detail, so a driver that finished restarting, or a usable eGPU attached after startup, left DEVICE and CHAT_ONLY frozen at CPU while the reason flipped to no_gpu: Train and Export disabled and the UI insisting there is no GPU, until a restart. The refresh now invalidates the detection epoch instead and keeps the current answer until the next pass republishes, at most once per recovery so a poll cannot starve detection of a chance to settle. * Studio: XPU recovery and masks, one-to-one adapter corroboration, quieter probes classify_torch_build asked only about CUDA, so an Intel host that started while its runtime was down and later recovered kept a healthy +xpu wheel classified as torch_cuda_unavailable forever. The recovery branch added last round was therefore never reached and the process stayed chat-only until restart. It now consults torch.xpu.is_available as well, with a probe that raises still counting as no answer rather than as a verdict. ZE_AFFINITY_MASK joins the deliberate-hide set. The rest of this module already reads an emptied one as hiding every Intel device, but it was missing here, so a deliberately masked XPU install was reported as broken and offered a repair for a configuration the user had chosen. Applied only where an Intel card is present or the inventory could not answer, matching the rule the HIP masks already follow. Registry records are claimed one to one. Two identical cards leave two identical records, and if one is removed WMI reports a single live adapter that the previous predicate matched to both, reporting a GPU that is gone and counting its VRAM twice. Each live entry is now consumed by at most one record. The live scan no longer swallows its own failure. -ErrorAction SilentlyContinue exits 0 with empty stdout, which is indistinguishable from no adapters, so the corroboration would have dropped every real card on a host with damaged WMI, recreating the false no-GPU result this change exists to remove. It now stops on error so the caller sees the documented unknown. A missing nvidia-smi is no longer a warning. That is the normal state of every CPU-only, AMD and Intel host, and the inventory runs on a 60 second refresh reached from the health and system polls, so it was a log line a minute on machines that are working correctly. A driver that IS installed and then hangs still warns. * Studio: the XPU triton swap now has a third repair point on Windows Step 13w migrates torch on Windows for the same reason the two Linux repair passes do, and a direct `python install_python_stack.py` there has no setup.ps1 postlude to swap triton back afterwards. The platform guard test still asserted the old contract, where Windows was skipped unconditionally, so it covered the case the invariant exists for and called the fix a regression. Windows is now covered on both sides: it defers when setup.ps1 published the handover variable, and does the swap itself when nothing else will. The wiring tests count the third call site and check that it follows _ensure_expected_torch_flavor, which is that platform's torch migration. * Studio: recovery really re-detects, the health path never blocks, and the sidebar keeps looking Five gaps from the last review, four of them in code I added earlier in this same review: - The recovery path called invalidate_detection() and stopped there, which retires the cached answer but schedules nothing to replace it, so DEVICE and CHAT_ONLY stayed frozen at CPU while the reason flipped. It now discards the detection under the lock and starts a fresh pass. - The mismatch report probed the inventory with block=True. /api/health and /api/liveness both reach it, and the NVIDIA half shells out with a 10 second timeout, so a hung driver -- the exact host this feature is for -- would have stalled liveness behind it. Non-blocking now. - classify_torch_build() gave up entirely when torch could not be imported, so a venv broken badly enough to matter reported nothing at all. It now reads the wheel's own version label off disk, with no interpreter started. - ZE_AFFINITY_MASK was missing from the visibility masks, so an Intel host that had masked its GPU away was told the install was broken. - The sidebar stopped polling at the first settled verdict, so a host that gained an accelerator after launch could not learn about it without a restart. It now keeps polling at the backend's own 60s inventory cadence, and only for the three verdicts a probe can actually move. A deliberate CPU, ROCm or XPU backend is also recorded as the expected flavor now, but only when the wheel on disk agrees with it: a backend is what the run asked for, and recording one that did not install would assert something untrue about the venv. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: the health path measures torch off-thread, and an unimportable or untagged XPU build is classified correctly Three more from the review, all on the same function: - classify_torch_build() returned early on _has_torch(), which is False both for a torch that is absent and for one whose native runtime will not load. The second is the case this classifier exists for, so the on-disk fallback added last round was unreachable on precisely that host. The test forced _has_torch to True and hid it; it now asserts the real value instead. - An untagged build that sets torch.version.xpu is an XPU wheel whose runtime is down, not a CPU wheel. It read as torch_cpu_build and the UI offered a reinstall, which is the one remedy that cannot help, while _torch_reports_an_xpu_runtime() already knew better and let the card establish the mismatch. - The classification itself still ran inline on /api/health and /api/liveness. It imports torch and asks CUDA and XPU whether they are available, and both block while a driver is wedged, so making only the inventory lookup non-blocking left the stall in place. It is now cached with the same 60s TTL and single-flight background refresh the inventory uses: detection takes the one blocking pass, the request paths read the measurement, and a cold cache reports unknown rather than guessing that torch is fine. A recovery drops the measurement along with the detection, or the fresh pass would settle on the answer it just retired. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: per-vendor masks, untagged GPU builds off disk, and a broken runtime is a mismatch not a failure Four more, all genuine: - An emptied ZE_AFFINITY_MASK on a hybrid NVIDIA plus Arc host was cancelling the whole classification, so a CPU-only wheel went unreported for the NVIDIA card the user never masked. Masks are per vendor now: they suppress the classification only when they account for every accelerator the host has, and the cards they do hide drop out of the mismatch inventory instead. An inventory that cannot answer still stays conservative. - The on-disk classifier read only __version__, so an untagged conda or source CUDA build that failed to import was called a CPU wheel. It now reads the cuda/hip/xpu markers out of torch/version.py as well, which is what the importable path already looks at, parsed rather than executed. - UNSLOTH_TORCH_BACKEND was consulted after the manifest, so a reinstall that deliberately changes flavor re-recorded the previous install's tag: the next launch called the new CPU wheel a mismatch and a later unpinned update would have restored CUDA over the user's choice. The current run's backend now outranks the manifest, still only when the installed wheel agrees with it. - A wheel whose native runtime will not load took the detection_failed arm and never reached the classification, so the host this change exists for got a pointer to the server log rather than the repair. It is now classified from the wheel on disk, and detection_failed remains the answer when nothing on disk says otherwise or the OS found no card. The disk path deliberately avoids importing torch anywhere: that import is what failed, it takes seconds to fail on a real broken wheel, and a retry re-runs torch/__init__ against the partial module tree the first attempt left behind. Both hardware caches are also cleared between tests under their locks, or one suite's host answers for the next one within the TTL. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: CUDA_VISIBLE_DEVICES masks AMD too, and no request path retries a failed torch import - HIP honours CUDA_VISIBLE_DEVICES alongside its own variables, which is why the visibility resolver in this module already reads all three together on an AMD host. Mapping it to NVIDIA alone meant an AMD-only box launched with CUDA_VISIBLE_DEVICES="" did not count as masked, so its cards still established a mismatch and Studio offered a repair for a mask its owner set on purpose. - The version label was still read with _torch_version_label(), which imports torch. On the broken-runtime host that import is exactly what fails, it can take seconds, and _has_torch() purges the partial module afterwards so every call genuinely re-runs the native load. /api/health, /api/liveness and GET /api/system all reach it, so the one place that seeded the classification from disk was being undone by the line that reported it. All three now read what detection published, or the label the wheel carries on disk, and only a host whose torch imports fine asks torch itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: an AMD card the installers decline is not a broken install setup.sh ships a ROCm wheel only for RDNA 2 and newer, so a Polaris or RDNA 1 card is left on CPU torch on purpose. The DRM sysfs walk reports it as vendor amd like any other, and every non-Intel device was counted as evidence, so that host was told its installation was broken and offered a repair that cannot make the GPU usable. The arch is now consulted the same way the installer picks one: rocminfo, then amd-smi, read once inside the cached inventory probe rather than on any request path. The targets are attached to every AMD record rather than matched one to one, because setup.sh says in as many words that these orderings are not guaranteed to agree and a guessed pairing would label the wrong card. On Windows the driver already writes the arch into AdapterFamily, so that record answers directly. A host with no ROCm userspace names no arch, and a card with no arch is still counted: that is the conservative direction and the shape of the original report, an RX 7900 XT, which this stack does support. A ROCm expectation or a torch carrying a HIP runtime also outranks the table outright, since a wheel this install asked for is a fault whenever it is missing. The supported set is pinned against setup.sh's own table by a test, so the two cannot drift. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: an absent nvidia-smi is an answer, and the masks are compared against a measured inventory - The NVIDIA probe reported "could not answer" when nvidia-smi simply is not installed, which is the normal state of every AMD, Intel and CPU-only host. On an AMD-only machine that made every empty inventory unknown, so once the card that established a mismatch was detached the verdict refresh preserved the old reason for good while /api/system had already dropped the device rows. The probe now distinguishes an absent CLI from one that was found and failed, and only the second is unknown. A driver that is installed and hangs is still loud. - On a cold start the mask check read the inventory non-blocking, got the explicit unknown, and therefore counted every mask as relevant: an irrelevant HIP_VISIBLE_DEVICES="" on an NVIDIA-only host suppressed the classification and cached "torch is fine" for a whole TTL, withholding the repair from a host nvidia-smi could describe moments later. The pass that caches the measurement now blocks on the inventory, which it can, since it runs on detection or on the background refresh thread and never on a request path. An inventory that was measured and still could not answer stays conservative, as before. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: release the re-detection guard, read the verdict once, and stop guessing at unnamed AMD cards - The guard that limits recovery to one detection request was cleared only by detect_hardware(), while the recovery starts its pass through start_background_detection(), which runs ensure_hardware_detected(). So the first recovery raised it for the life of the process: a driver that flapped, or any later CPU verdict the inventory could move, would never get a second pass. It now comes down wherever a pass publishes, and only after it has published, so a retired pass cannot release a guard it did not satisfy. - export_capability() and video_capability() read the chat-only verdict three times while building one response. It refreshes on a TTL, so an eGPU attached or removed mid-response could give the reason from one host and the message from another. One reading now feeds the whole response. - An AMD card that nothing can name is no longer treated as evidence. setup.sh detects an AMD host through rocminfo and amd-smi and nothing else, so a machine with neither was never going to be given a ROCm wheel, and its CPU torch is the intended state rather than a fault. A recorded ROCm expectation still outranks the arch table, which is what covers a venv whose ROCm wheel really was replaced. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep the hardware module loadable without the rest of the package The Windows console-hiding helper was imported at module scope, and tests/python/test_e2e_no_torch_sandbox.py executes hardware.py against a minimal stub tree (loggers, structlog, utils/hardware and nothing else) to prove the module still loads on a host with no torch. That import made it unloadable there, so ten sandbox cases failed on a property that has nothing to do with Windows. It is now imported inside the one function that uses it, which also happens to be the only Windows-only path in the file, and empty kwargs are the right answer on every other platform anyway. A test pins the rule so the next module-scope import under utils is caught here rather than in the sandbox suite. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: five more gaps in the Windows flavor invariant A visibility mask no longer vetoes an explicit expectation. An update inheriting CUDA_VISIBLE_DEVICES="" or -1 disabled the invariant outright, so the dependency steps could still put PyPI's CPU wheel over a cu124 venv and the update recorded the CUDA expectation over it, which is the whole defect this change exists to close. An emptied mask is a reason not to CONCLUDE a flavor from an nvidia-smi probe; it is not a reason to ignore one setup.ps1 published, one the user pinned, or one the last install recorded. The gate now applies only to the inferred case. The post-repair classification uses the same rule as the pre-repair one. Both compare through _torch_flavor_tag, which reads every untagged version as cpu, and only the pre-repair side consulted the runtime markers, so a private index answering a /cpu repair with an untagged CUDA wheel was accepted on a rule the check that triggered the repair had just rejected it on. The resync notices its own failures. pip_install_try and _uninstall_distribution report by return value rather than by raising, and both results were ignored, so an unreachable PyPI behind a reachable torch index, or an xFormers whose files were locked, left the wrong build in place while the update reported success. Both now warn. Neither fails the update, which stays deliberate. A direct Windows run now performs the XPU Triton swap. setup.ps1 does it after this script exits, so the helper skipped Windows entirely, but a direct invocation has no such postlude and the core package install pulls triton-windows over torch's XPU triton, leaving torch.compile unable to use the XPU. The handover variable gates it, since setup.ps1 sets it immediately before invoking this file and clears it otherwise. * Studio: ARM64 exception on the delegated ROCm repair, and a broken torch is not a hung one _ensure_rocm_torch's Windows branch asked for the full torch, torchvision and torchaudio trio unconditionally, so on win_arm64, where no torchaudio wheel exists on any index, the whole install is unresolvable. That is worse on the delegated path than on the plain one: the failure there is nonfatal, so the CPU build the repair existed to replace stays in place and the family verification then fails the update. The same _is_windows_arm64 exception the flavor repair and setup.ps1 apply now covers it. setup.ps1's disk-label rescue treated a definitive import failure exactly like a driver timeout. Both leave a +cu* version.py behind, and keeping the venv is right in both cases, since deleting one does not fix a driver and a faulted display or HIP runtime raises at DLL load rather than timing out, which is the bug the rescue was written for. But only a timeout says nothing about whether the installation is sound: a truncated torch took the same path, the family-matched install below then ran with bare requirements and no reinstall flag, and the run could write a completion manifest over a torch that still could not import. Invoke-BoundedPythonProbe now reports TimedOut, so the two are distinguishable, and a definitive failure keeps the venv but forces a same-family reinstall in place. Restricting the rescue itself to timeouts would have resurrected the original wipe, so it is the reinstall that is gated, not the rescue. One existing assertion in test_pr5940_followups.py looked for the three companion names inside the call itself; the trio is now built just above it so the ARM64 exception can drop torchaudio, and the assertion follows it there. * Studio: probe fallbacks the installer already has, and an automatic CPU choice is not a deliberate one - export_capability() and video_capability() asked _has_torch() before the mismatch branch, so on a host whose wheel will not import they reported pytorch_not_installed for a PyTorch that is installed, and retried the failing import on every request that renders them. The already-derived mismatch is consulted first now. - The NVIDIA inventory treated a missing nvidia-smi as no cards. _has_usable_nvidia_gpu() falls back to /proc/driver/nvidia/gpus, so the installer could select or repair a CUDA wheel on a host where the backend insisted there was none. Same fallback here, publishing no name and no capacity because procfs gives neither. - An AMD card nothing can name is eligible again when the KFD topology confirms it, which is the fallback _has_rocm_gpu() uses so `studio update` can repair a minimal host with no ROCm userspace. Positive evidence of an unsupported arch still excludes it; the NVIDIA open driver's own KFD nodes are rejected by vendor id, as in the installer. - A recorded CPU flavor is only treated as a deliberate choice when the manifest says someone named it. setup.ps1 selects /cpu automatically on a GPU-less host and records it identically, so the tag alone left a machine that later gained a card reporting no_gpu with no repair on offer. The installer now records whether the flavor was pinned, and a manifest predating the marker reads as not pinned: an offered repair can be declined, a silently CPU-only GPU host cannot. - The test fixture took the two hardware cache locks in the opposite order to the background refresh, which could deadlock the suite against a refresh a previous test left running. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: per-vendor probe accounting, exact adapter matches, and a derived backend is not a pin - A vendor whose probe did not answer no longer has its unknown cancelled by some other vendor's card. On a hybrid Intel plus NVIDIA host an nvidia-smi timeout was cancelled by the Intel sysfs row, and since an ordinary iGPU cannot establish a mismatch the verdict then saw neither an eligible card nor an unanswered probe and downgraded a settled mismatch to no_gpu, hiding the repair while the NVIDIA probe was merely unavailable. - The registry-to-live adapter match now prefers an exact name. The registry outlives the hardware, and a stale "RX 7900 XT" record is a prefix of a live "RX 7900 XTX", so it could claim the card that is really installed and publish the removed one's name and VRAM. The prefix rule stays: the driver description and the WMI display name spell the same card differently. - install.sh derives UNSLOTH_TORCH_BACKEND from the index it resolved, which is "cpu" on any machine without a GPU whether or not anyone asked for it, so treating it as a pin recorded every ordinary Linux CPU install as a deliberate choice and left a later eGPU with no repair on offer. install.sh now marks the value as derived, and only an unmarked one -- the form setup.sh documents for a deliberate CPU install -- counts, alongside the index pin and family. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: every supported AMD arch, HIP precedence over the CUDA alias, and procfs for a broken CLI - The supported-arch set was built from setup.sh's marketing-name table, which is a subset of what the installers actually ship a wheel for: gfx1031, gfx1033, gfx1035, gfx1036, gfx908, gfx90a and the gfx906 ROCm 6.3 path were all missing, so those cards were reported as no_gpu with no repair. It is now pinned against install.sh's own arch-to-index-family map, which is the map that decides, and the drift test fails in both directions rather than only on an over-broad set. - An AMD host that names its devices in HIP_VISIBLE_DEVICES while inheriting an empty CUDA_VISIBLE_DEVICES was read as fully masked. HIP consults its own variables first and falls back to the alias only when neither is set, which is the precedence _get_parent_visible_gpu_spec already applies in this module. - The procfs fallback covered only an absent nvidia-smi, not one that hung or exited non-zero, though the kernel driver enumerates its cards either way. On a cold start there is no settled verdict for the resulting unknown to protect, so the host was reported as having no GPU for as long as the CLI stayed broken. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep a stated backend's provenance, count an untagged XPU runtime, and drop a stale flavor under a mirror pin - install.sh marked every backend it exported as derived, including one the caller had already stated. setup.sh documents UNSLOTH_TORCH_BACKEND=cpu as the way to keep a deliberate CPU install, and on a GPU-less host the resolved value is cpu too, so the two were indistinguishable. It now looks before it overwrites, and only a value it derived itself is marked. - The shared torch probe reported hip and cuda but not xpu, so an untagged source or conda XPU build read as having no GPU support at all -- the one verdict that fails an update outright -- while the backend classifier added here reads the same torch.version.xpu as a GPU build. The probe carries it now, kept beside the tuple rather than in it because thirteen call sites unpack five values. - With an explicit pin whose leaf names no family (a corporate /simple mirror), the manifest wrote the PREVIOUS install's flavor back and marked it pinned, since any explicit index counts as one. The wheel in the venv came from the mirror and the old record describes a venv that no longer exists, so a later unpinned run could have "repaired" the mirror's build to a public one on the strength of a flavor nobody verified. Nothing is recorded for that case now. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reduce comment volume across the branch Comment-only pass over the eight files this branch touches: delete restatements of the code, section banners and narration, and collapse multi-line blocks that did not carry a distinct fact per line. Added comment lines 409 -> 140. No code or behaviour change. Installer suites still pass (3123 passed). (cherry picked from commit e25b0caa90201ec72cca0f22971496cd90d09d47) * Reduce comment volume across the branch Comment-only pass over the 26 files this branch touches: delete restatements of the code, section banners and narration, and collapse multi-line blocks that did not carry a distinct fact per line. Added comment lines 1164 -> 447. No code or behaviour change. Installer and hardware suites pass (3238 passed), gpu-torch-mismatch.test.ts passes (11), cargo check clean. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: evaluate the sidebar poll decision, do not just match its text The existing test pins the shape of the polling guard by regex. It would still pass if the sense of the condition were inverted, and that regression is expensive in exactly the wrong direction: every healthy install would start forcing a /api/system read a minute for the life of the session. Lift the guard and the reason set and run them, over both the hosts the inventory can still reclassify (torch_cpu_build, torch_cuda_unavailable, no_gpu, since an eGPU can arrive on a CPU-only box) and the hosts that must not poll at all (a healthy GPU host, an Intel Mac, a failed detection). The two pre-existing polls, the MLX self-heal and the unknown-verdict one, are asserted alongside them so neither can be dropped. * Studio: the unimportable-torch test passed for the wrong reason below 3.13 The test put an object in sys.modules whose every attribute access raises, then asserted _has_torch() reports False. Whether that holds depends on import machinery internals: reading __spec__ off an existing sys.modules entry changed in 3.13, so on 3.10 through 3.12 `import torch` handed the stub straight back untouched, _has_torch() returned True, and the premise silently inverted. The body then ran against a host that looked like it had a working torch, so the on-disk fallback the test exists to cover was never reached, and the assertion that would have caught it is the one that stopped holding. Install a meta-path finder whose loader raises OSError from exec_module instead. That is a real import that really fails, which is also closer to the case being described -- a wheel whose native library will not load -- and it records TORCH_IMPORT_ERROR the way the real failure does, which the old stub did not. Verified on 3.10, 3.11, 3.12, 3.13 and 3.14: _has_torch() is False on all five and the recorded error is the OSError. Before this commit it was True on the first three. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fixes from review round 2 setup.ps1: a torch that will not import now clears $SkipPythonDeps. Its only consumer, the CUDA --force-reinstall, lives inside `if (-not $SkipPythonDeps)`, so on a current core package with a valid manifest the fast path skipped the repair and the update reported dependencies up to date over a broken wheel. install_python_stack.py: the two "untagged wheel under a /cpu pin" checks now read torch.version.xpu alongside .hip and .cuda through _gpu_family_from_runtime_markers(), so an XPU build no longer compares equal to cpu. _expected_torch_flavor_was_pinned() takes the flavor being recorded and counts a pin only when its own family matches it, so a ROCm or XPU pin that setup.ps1 fell back to the CPU index for is no longer recorded as a deliberate CPU choice. hardware.py: a Windows AMD adapter the DirectX registry gave no AdapterFamily is now read through the installers' marketing-name arch table instead of falling to a Linux-only KFD probe; a registry read that failed is distinguishable from a vendor with no adapters and marks that vendor unanswered; an inventory refresh that could not answer carries the previous devices forward per unanswered vendor rather than caching the empty failure; the ROCm pin check uses the installer's exact family predicate; and a detection_failed that was classified from the wheel on disk can transition once the inventory recovers, without ever falling through to no_gpu. use-tauri-backend.ts: retry on a failed forced repair re-runs the forced repair. The installer is transactional, so the preflight would find the restored install and restart the same CPU-only backend the user pressed Repair about. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: clear the retry state before resuming a forced repair startRepair resets most of it, but not startTimedOutRef or the elevation package list, so the forced branch skipped two resets the generic path does. * Studio: the XPU and CPU force gates must read the broken-import flag Same defect the CUDA arm was already given: an unimportable +xpu or +cpu wheel still reports its on-disk tag unchanged, so no flavour change is seen, the bounded range is satisfied, and the resolver keeps the broken wheel. * Studio: an unreadable /sys/class/drm walk must not read as no cards The helper swallowed every OSError into an empty list, so the inventory could not tell a driver restart from a host with no AMD or Intel card and the carry-forward had nothing to act on. A MISSING /sys/class/drm still answers: a host with no DRM subsystem has no such cards. A cardN whose vendor file cannot be read makes the walk partial and is treated the same way, while one with no vendor file at all is a virtual device and never was a GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fixes from review round 3 hardware.py reads UNSLOTH_TORCH_INDEX_URL before UNSLOTH_TORCH_INDEX_FAMILY, as install.sh's get_torch_index_url() does: it returns on the URL and never reads the family, so when the two name different families the family is dead, not a second opinion. Reading either as authoritative let a stale ..._FAMILY=cpu beside a new ..._URL=.../cu128 suppress a real CPU-wheel mismatch. All three expectation helpers now share one resolver. detect_hardware's disk-only branch applies the two suppressions classify_torch_build() applies before its own disk fallback. A deliberately CPU-only install whose torch will not import was published as torch_cpu_build and offered a repair that would reinstall the very wheel that was asked for. app-sidebar.tsx counts detection_failed as inventory-sensitive. The backend can replace it with torch_cpu_build or torch_cuda_unavailable once the OS probe recovers, so treating the first answer as settled stopped the only forced health re-read and froze the sidebar on the failure for the rest of the session. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: the flavor provenance check must not re-read the index family _explicit_torch_index_url() already reads both variables with install.sh's precedence: the URL wins outright and the family is consulted only when no URL was supplied. Checking the family again after it reintroduced the precedence it exists to enforce, so an authoritative corporate /simple URL, whose leaf names no family, fell through to a stale UNSLOTH_TORCH_INDEX_FAMILY=cpu and recorded the CPU wheel a GPU-less host legitimately got as a deliberate choice. An eGPU attached to that host later would then get no mismatch and no repair offered. The second read was redundant as well as wrong: a family with no URL beside it already reaches the same check through the shared resolver. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fold the duplicated cache and manifest plumbing _schedule_physical_gpu_inventory_refresh and _schedule_torch_build_snapshot_refresh were identical single-flight refresh implementations differing only in which globals they named. One _schedule_single_flight_refresh now serves both, so the concurrency logic has one definition rather than two that a later fix could change apart. The in-flight flags stay module globals, which is what the tests reset. The three _expected_*_flavor_was_chosen helpers each opened and parsed the manifest with their own copy of the swallow-everything guard. That read is now _recorded_install_flavor(), returning (tag, pinned). In the tests: twenty resets of _physical_gpu_inventory_cache that the autouse fixture already performs, and five hand-rolled "nvidia-smi is absent" subprocess stubs that _smi now expresses with raises=FileNotFoundError. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fixes from review round 5 _ensure_cpu_torch's own GPU-build predicate now reads torch.version.xpu. An untagged source, conda or private-index XPU wheel carries its runtime only there, so the predicate declined to reinstall while _installed_flavor_tag_now, which does read the marker, went on to report "still xpu" -- the detection had improved and the repair had not followed it, and the update failed instead of honouring the CPU pin. The parity test claims equivalence with the marker empty, which is everything the merge-base predicate could see, and pins the divergence separately. The GPU-present-but-unusable message names both recovery routes. Settings only carries the repair row inside the desktop app and only for a backend it manages: DesktopRepairControl returns null without a Tauri repair context and null for an externally started server. A browser-hosted Studio, or a desktop attached to a server someone started from a terminal, was being sent to a control that is not on its page. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fixes from review round 6 install.sh marks a resolved backend as derived unless the family the caller STATED survived the resolution. The case statement overwrites UNSLOTH_TORCH_BACKEND before the marker is decided, so a caller who said "cuda" on a host with no visible GPU carried the resolved "cpu" and had it recorded as a deliberate choice, which denies that host the repair for good if the GPU ever becomes visible. _expected_torch_flavor_tag stops at an unknown-family pin rather than falling through to the manifest. The wheel came from that mirror, so the recorded tag describes the install the mirror replaced and the CUDA probe would hand back the mirror's own leaf as a flavor. This also made _recordable_torch_flavor_tag's guard unreachable: it only fires on an empty resolved, and the stale tag arrived non-empty. The setup handover still outranks it. Both manifest readers require an actual boolean for expected_torch_tag_pinned. bool("false") is True, so a migrated or hand-edited manifest read as a deliberate pin and suppressed the repair on a host that never chose one. The sidebar collects the refresh its poll triggers. The backend's caches carry their own 60 second TTL and the health path reads them non-blocking, so the read that finds them expired only schedules the refresh; polling on the TTL alone put the collecting read a whole interval later and left an attached eGPU invisible for close to two minutes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: type the injected window's timer in the lifted-poll harness The block that test evaluates now reaches for window.setTimeout, and CI runs tsc against tsconfig.test.json, which my local --noEmit on the app tsconfig never covered. * Studio: bind the XPU runtime marker in the lifted CPU-repair predicate The predicate is read out of install_python_stack.py and evaluated here, so the marker it now consults has to be in the namespace it is evaluated in. Empty for the existing cases, plus one that states it, which is the only way an untagged XPU wheel is known to be one. * Studio: let only the highest-priority HIP mask that is set decide HIP stops at the first of HIP_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES that is set, the precedence _first_visible_amd_gpu_id already applies. Reading any of them as un-hiding the cards meant HIP_VISIBLE_DEVICES=-1 beside ROCR_VISIBLE_DEVICES=0 reported a mismatch, and offered a repair, for a host masked exactly as it asked to be. * Studio: scope the frozen mismatch to its own vendor, and let a resolved CUDA install re-record the venv Two things a stale answer outlived. The frozen chat-only verdict held on any unknown inventory, so a broken probe for a vendor that never had a card kept the mismatch: detach an AMD eGPU that sysfs then conclusively reports as gone, and a broken nvidia-smi keeps saying the GPU is unusable for as long as it stays broken. _carry_unanswered_vendors_forward already re-adds the devices an unanswered vendor had reported, so nothing left to point at means no vendor that ever named a card is still unread. The verdict now records which vendors established it and holds only for uncertainty about those. An unknown that names nobody, which is what a cold cache looks like, stays conservative. _expected_torch_flavor_tag excluded a resolved cuda backend, so a CPU pin that had been REMOVED outlived the install that replaced it: install.sh resolves cuda and installs a CUDA wheel, the old manifest still says cpu, and the healthy venv is re-recorded as deliberately CPU-only, with the pinned provenance carried forward. A later update that swaps in a CPU wheel then reads as the install working as asked, and the backend suppresses the mismatch and its repair. The wheel's own cu tag answers instead, under the same rule the cpu/rocm/xpu arm already applies: only when the family agrees with what is actually installed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: three stale answers, one per layer The vendor set behind the frozen mismatch was recorded at detection and never after, so a mismatch that MOVES vendors inside one process -- swap an AMD eGPU for an NVIDIA one -- left the scope pointing at a card a later refresh had already watched leave. The refresh that re-establishes a mismatch now re-records who says so. nvidia-smi answering [N/A] for memory.total dropped the whole row, which took the card out of the inventory: the Linux procfs fallback does not cover it either, since that answers for a query that FAILED rather than one that came back short. A host lost its mismatch and its repair guidance over an unknown capacity. _parse_smi_value already reads [N/A] as a missing metric, so the row is kept with memory_total_gb None, which every consumer already handles. _expected_torch_flavor_was_pinned revived an old CPU record underneath the very arms that had just refused to call this run's CPU result deliberate. A ROCm pin that settled on CPU was therefore re-recorded as a pinned CPU install, and the mismatch is suppressed for good once the requested GPU works. A record now only speaks for a run that named no conflicting family; a backend install.sh derived is not a preference and still does not contradict anything. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the repair row findable from the settings search The capability message for a host whose PyTorch cannot use its GPUs tells the user to use Repair installation in Settings, and searching Settings for "repair" answered "No settings found": the row is in the General tab but was never added to the search index, which carries its neighbour in the same Danger zone section. In the desktop-only arm, like the row itself. DesktopRepairControl renders nothing without a Tauri repair controller, so an entry on the web build would scroll to a row that is not on the page, which is what the startup and close-to-tray entries already avoid. Found by driving the real desktop app: searching for the row returned nothing. * Reduce comment volume across the branch --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-31 22:28:51 -07:00
oobabooga	1400031e2f	CI: fix llama.cpp seed caching, Diffusers wheel installs, and isolated test coverage (#9979) ... * CI: fix llama.cpp seed caching, Diffusers wheel installs, and isolated test coverage * Keep a llama.cpp install only when its binaries actually run * Fix the prompt-cache, Diffusers, and MCP menu CI legs * Check the kept install's runtime payload, so Windows is not exempt * Give the CLI launch tests a writable studio home * Give each isolated unsloth_zoo file its own pytest process * Require the paired cudart trio and keep an install that already runs the stored backend * Tolerate an empty collection in the zoo rerun and re-anchor the node dir test * Skip the inline code scan when no backtick escapes the block regions * Answer the AppImage E2E preflight with the headers it asked for * Ask an unreadable marker for the payload every kind of its platform needs * Keep the mlx family out of the parallel zoo pass and run it serially * Shorten verbose PR comments * Reject a kept install whose binaries the OS will not execute * Probe the launched entrypoint, read Windows loader statuses, and honour this run's selection * Give the probe the recorded runtime line and stop reading GPU detection as a choice * Shorten the comments added since the last pass * Backfill the paired runtime asset when an install is reused * Compare the seed in llama.cpp's uint32 domain and require the published payload * allow cpu security import probe * Treat every uint32 sentinel seed as random in the per-choice offset _apply_seeded_llama_request compares the seed in llama.cpp's uint32 domain, so -1, 4294967295 and 2**64-1 are all LLAMA_DEFAULT_SEED and all keep prompt caching. _choice_seed still exempted the literal -1, so for n > 1 the two helpers disagreed inside one request: choice 0 kept the caller's random seed and its cache, while choice 1 was offset to ((seed & 0xFFFFFFFF) + 1) % 0xFFFFFFFF, a fixed seed with the cache turned off. seed=4294967295 with n=3 served [4294967295, 1, 2]. Confirmed against a real llama-server: -1, 4294967295 and 2**64-1 each varied across repeated requests, while 4294967294 and 0 did not. llama.cpp applies the same test at its own multi-choice offset in tools/server/server-task.h, where the guard is seed != LLAMA_DEFAULT_SEED rather than a literal. * Stop reporting a kept install as one that was just installed The keep path answers a transient update failure by exiting 0 with the tree already on disk. Both setup scripts only distinguish exit 0 by looking for "already matches", so a kept install took the other branch and printed "prebuilt installed and validated", naming a release that was never fetched. setup.ps1 also went on to print the old release as the installed one. The in-app updater had the same problem on macOS. start_update passes no pin_release_tag there, so the "pinned release X but installer produced Y" check that catches this on Linux and Windows does not run, and the phase reported "Updated llama.cpp to <the tag the user already had>". It now compares the marker before and after and says the install is already up to date, which is what happened. * Fail the zoo step when its mlx group glob matches nothing mlx_group is passed to pytest unquoted so the list word-splits into file arguments, which means an empty list passes no files at all and pytest collects the whole rootdir serially instead: green, much slower, and no longer the group the step describes. The glob only has to stop matching once, upstream renaming the family for instance, for that to be what runs. Also pass each rerun's own filename to _keep. Exit 5 stays tolerated, but a module-level skip and a file that stopped collecting for a new reason both produce it, and only the log says which file it was. * Cover the kept-install path against every marker shape that has shipped _existing_install_runs and _kept_install_payload_is_healthy decide whether an install the user already has survives a transient update failure, and they had no direct tests: the existing keep-path cases drive install_prebuilt with a hand-built two-key marker. UNSLOTH_PREBUILT_INFO.json has grown twelve shapes since #4562 with no version field and no migration, so an absent key is the normal case for anything installed before the release that added it. Covers each shipped shape on Linux, Windows and macOS hosts, including the S1-S8 markers that carry no backend key and resolve through the asset name, and the S1-S11 Windows CUDA installs that predate runtime_asset and so must not be asked for a cudart trio they cannot record. Also pins two boundaries that were unasserted: a source build has no marker, so confirm_install_tree rejects it and it still falls through to the source-build fallback, while a corrupt marker keeps the tree eligible and holds it to every kind's shared payload. Adds the missing SIGSEGV and hang cases for the --version probe. Platforms are simulated through HostInfo, as the rest of this suite does. That covers the path decisions and the payload tables; it does not cover the Windows loader or macOS dyld, which no Linux runner can answer. * Cover the keep path across every OS and accelerator, and its exit contract Extends the marker-shape cases with the product the installer actually has to serve. Fifteen cells across Linux, WSL, Windows and macOS by NVIDIA, AMD, CPU and Vulkan, each asserted three ways: a complete tree is kept, a gutted one is refused, and an accelerator tree missing only its own backend library is refused. That last case is the one that matters after a partial extraction, where the binaries still start and only fail once a model is loaded. Vulkan, windows-hip, windows-rocm, windows-arm64 and linux-arm64-cuda had no payload coverage at all before this. WSL is asserted to behave exactly like Linux, which is the claim worth pinning: HostInfo carries no WSL field and the WSL2 ROCDXG handling lives in host detection and setup.sh, upstream of the keep path. Adds the exit contract the setup scripts branch on, driven through install_prebuilt rather than the deciders: a transient failure returns 0 for every shipped marker shape, a tree that is not runnable still reaches EXIT_FALLBACK so a source build can proceed, an explicit version request is never answered with the old install, and a marker recording a backend this build cannot install exits EXIT_ERROR before the keep branch is reached. That last ordering was untested and is what stops a newer marker being answered with the tree already on disk. Also covers a marker whose keys were grafted on by sync_marker_selection rather than written in one go, which is the shape a long-lived install really has, and a marker naming another platform's backend, which falls open rather than being judged by a table that does not apply. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments added for PR #9979 Same behaviour, fewer lines. AST comparison confirms every changed .py file is identical once docstrings are stripped, so this is comments only. setup.sh passes bash -n, setup.ps1 parses, the workflow YAML parses and its zoo step body still passes bash -n. * Report a kept llama.cpp install as an unavailable update for PR #9979 --------- Co-authored-by: mahiatlinux <110882203+mahiatlinux@users.noreply.github.com> Co-authored-by: Daniel Han-Chen <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-31 04:08:01 -07:00
Daniel Han	fa57a7986f	Studio: notice a quarantined payload before trusting the update fast path (#10053) ... * Studio: notice a quarantined payload before trusting the update fast path An update can report success while leaving the venv unbootable. Every check the fast path makes reads metadata: installed_version_probe reads dist-info, verify_install compares the manifest and walks missing_requirements through importlib.metadata. None of them looks at whether the recorded files are still on disk. So when an antivirus quarantines site-packages/unsloth after a good install, the version still answers, verify_install still says ok, and setup takes the fast path and skips the dependency pass that would have repaired it. Measured on a real 2026.8.22 install: deleting a recorded file, truncating one, removing a nested module, or removing the whole tree all leave `studio update` exiting 0 and reporting the update successful. damaged_payload_files walks the managed distribution's own RECORD and reports rows that are gone, are not regular files, or are shorter than recorded. verify_install runs it last, once every metadata check has already passed, and fails with a new reason, studio_install_damaged. That is the whole change on the shell side: setup.sh and setup.ps1 already call verify_install and already force the dependency pass when it says no, so neither script is touched. Deliberately narrow, in three ways. It answers for the managed package alone rather than for every installed distribution, because its only job here is deciding whether to repair ours. It runs last, after manifest_ok and deps_ok are both established, so none of the six existing reason strings can be diverted into the new one. The reason is additive and nothing matches it against a closed set: the Tauri preflight carries it as an opaque Option<String>, and is_context_reason is a negated two-item allowlist, so an unknown reason keeps can_auto_repair true, which is what we want here. It is off on the boot path. `deep` defaults on so the two shell callers, which pass no arguments, get the scan; install_state passes deep=False because it feeds `desktop-capabilities`, which the preflight spawns under a 10 second timeout. Spending that budget on a stat of every recorded file risks a timeout reporting a healthy install stale and repairing it, which is worse than the quarantine it would have caught. `unsloth studio update --verify` already scans the whole tree after the pass. The kwarg is guarded by the same _verify_install_supports introspection as installed_conflicts, so a newer CLI against an older install_manifest.py still works. Two carve-outs, both for trees our own setup rewrites. package-lock.json loses only its size claim, since npm dedupes hoisted entries and shrinks it, matching _INSTALLER_REWRITTEN_NAMES. studio/frontend/dist is skipped as a subtree: setup runs `npm run build` in the installed tree, vite empties dist and rewrites every asset under a fresh content hash, and the wheel ships all 704 of those files in RECORD, so a rebuild leaves rows naming files that no longer exist. A size waiver cannot reach that, and without the carve-out one rebuild would fail every later update permanently, since a no-op pip pass restores nothing. RECORD is parsed with csv over io.StringIO, not splitlines, because a quoted field may contain a newline and splitlines also breaks on \v, \f and the Unicode separators. Unreadable is not missing: anything but FileNotFoundError is skipped, because a reinstall cannot fix EACCES. An environment the scan cannot read reports undamaged, so a check that cannot answer leaves the fast path as it found it. Verified: 294 distributions in a real environment, zero flagged, slowest scan 0.168s; unsloth 2024.10.0 through 2026.8.22 all clean; all four damage states caught; a frontend rebuilt in place is clean while real damage in the same tree is still caught. Existing suites unchanged at 2887 passed, including the 83 that pin verify_install's reasons, and the shell fast-path suites still pass with no shell changes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the payload scan opt-in and bounded Three problems with the first cut, all found by auditing it rather than by using it. The scan defaulted on, which is the wrong direction under version skew. An external CLI driving a managed venv resolves install_manifest out of THAT venv: _manifest_candidates yields the CLI's own sibling copy first, then falls through to sys.prefix, so an older published CLI can load a newer copy of this file. It calls verify_install(root=root) with no `deep`, so defaulting the scan on put a stat of every recorded file inside the 10 second timeout that desktop-capabilities runs under on the Tauri boot path, with no way for the old caller to decline. interrupted-install-ci.yml builds exactly that pairing on purpose. `deep` now defaults False and the two shell callers opt in; they import the module from their own directory, so they can never be skewed against it, and they fall back to the old call on a TypeError so an older tree still works. The scan was unbounded. Neither installer wraps that call in a timeout, and setup.sh only reaches it inside the "is up to date" fast path, so the cost lands on the no-op run. 3644 of unsloth's 4358 recorded rows get stat'ed, ~65ms warm. Measured with latency injected into a real scan of a real install: at 5ms per stat a full walk is 18s, at 50ms it is 182s of wedged setup. A 5 second budget, checked every 64 rows so the clock read costs less than the stat it guards, holds those to 5.3s and 6.3s. Giving up reports undamaged, so a scan that cannot finish leaves the fast path exactly as it found it, which is what main does today. `unsloth studio verify-install` could not see the failure mode this was written to catch. It shares _install_state with desktop-capabilities, so opting the boot path out opted the diagnostic out too. install_state takes `deep` now and verify-install passes it: nothing times that command out, and a quarantined payload is precisely what a user running a verify step wants told. _fail_if_install_damaged does not cover this, because it returns silently when the CLI is outside the managed venv, which is the desktop configuration. Also: reuse the manifest already in scope instead of re-reading it from disk, which removes a second TOCTOU window. The boot path staying cheap was load-bearing and unpinned, so a refactor could have reinstated the scan there with a green suite. Two tests now hold it: one that both installers ask for the scan and keep their older-tree fallback, one that install_state and _install_state default it off while verify-install turns it on. Verified: the fast-path decision is unchanged from main on a healthy install and now forces the repair pass on a quarantined one, across [Linux, WSL, macOS, Windows] x [NVIDIA, AMD, CPU, XPU], 32 cells run twice each against both module versions, driving the real setup.sh block against a real venv. Run differentially because absolute expectations are useless here: unrelated escapes clear the flag too, and the anyio #6483 one masked the effect entirely until the venv was pinned to the anyio the requirements actually ask for. 15 compatibility checks over all four old/new pairings of the two files. 294 distributions scanned, zero flagged. unsloth 2024.10.0 through 2026.8.22 clean. unsloth_cli/tests fails the same 41 tests as origin/main, name for name. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair the damage the scan finds, and widen what counts as damage The scan told setup an install was damaged and setup then ran an upgrade, which for a distribution already at the wanted version installs nothing: uv audits it, pip calls it satisfied, and both are reading metadata that a quarantine of the payload leaves intact. Detection without repair only moved the failure later, so the core phase now names a damaged distribution for reinstall, with --no-deps so the pre-installed torch build is not re-resolved. --reinstall-package is uv-only; the pip fallback gets --force-reinstall, which is safe there only because that call also passes --no-deps and one name. Also five cases the first pass read as healthy: - a package directory replaced by a regular file. NotADirectoryError is an OSError but not a FileNotFoundError, so every row beneath it fell to the generic arm and reported merely unreadable. - unsloth-zoo. studio.txt does not list it, so nothing else here would notice it quarantined, and training, export and inference all import it. - a foreign managed venv. verify_install now takes scan_paths, and the CLI hands over that venv's own site-packages; RECORD rows resolve against the distribution that declared them, so without it the scan answered for the external CLI's tree and deep was a no-op on that branch. - a distribution gone entirely. Every version check compares against the installed version, so an absent one passes all of them. - the scan budget. It was consulted every 64 rows, which let a mount taking a second per stat overrun 5 seconds by a minute. A clock read is ~68ns against ~1343ns for a warm stat, so the guard is cheaper than the call it guards. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments added by this branch Comments and docstrings only, verified with comment_tools.py check --strip-docstrings. 179 comment lines to 66. * Abort when the repair cannot restore the files, and scan the launcher A forced reinstall that cannot reach an index left the tree damaged and the pass carried on, where the ordinary upgrade audits the intact metadata as satisfied and write_manifest records a success nothing rechecks. The repair now rescans and the installer aborts. Judged on the tree rather than pip's exit code, so a reinstall that restored the files while exiting non-zero is not held against it. Console scripts and data files are recorded relative to site-packages, so `../../../bin/unsloth` was dropped by the blanket `..` rejection: an antivirus could take the launcher, leave the package tree intact, and the deep check would call the install healthy while the command was gone. Parent-relative rows are now resolved and required to land inside the venv, which is what keeps a row naming something outside it from being read as our damage. Measured across every distribution installed here: 647 such rows in 81 of 294 distributions, none flagged, and a quarantined `bin/flask` detected. A companion whose dist-info is gone left the scan nothing to walk, and no check above ever looked at its version, so unsloth-zoo could vanish whole. Only for the default install: a custom `--package` need not depend on it, and demanding it there would repair that environment on every run. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse an absent core package, and leave a custom package's neighbours alone A distribution that is not installed at all has no RECORD to walk, so the scan read it as undamaged. That is only safe before the core phase, where a fresh run has nothing installed yet, so a second gate runs after it with require_present: by then the phase has run or was deliberately skipped, and the install.sh handoff that sets SKIP_STUDIO_BASE=1 never reaches it at all. Without this, write_manifest recorded a finished install for an environment with no unsloth-zoo in it. unsloth-zoo is now claimed only by the default install, in the installer and in verify_install alike. `--package X` installs X on its own, so a damaged unsloth-zoo beside it belongs to something else: force-reinstalling it would mutate an unrelated distribution and fail the update on an unreachable index. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Check presence after a repair, not only before it --force-reinstall uninstalls before it installs, so a failure in between leaves no distribution at all, and a distribution that is not there has no RECORD for the follow-up scan to call damaged. The repair reported success on the state it had just created. Presence is now rechecked after every reinstall attempt, unconditionally rather than under require_present: we only reinstall what the scan already found, so it was installed a moment earlier. The manifest written in that state records no version, and verify_install compared against it with `bool(recorded) and not current`, so the deep check called an environment with nothing installed healthy. An empty version is only ever absence here, since two records or an unreadable one set local_conflict and report studio_install_metadata_conflict before this runs. * Bound a stat that never returns, not only many that are slow The deadline inside the walk is checked between calls, so it bounds an accumulation of slow stats but not one that hangs outright, which is what a wedged mount or a filesystem filter produces. Neither installer wraps its verify_install call in a timeout, so that case blocked setup rather than reporting the scan unfinished as documented. The walk now runs on a daemon thread and is abandoned past the budget. The interpreter does not wait for one at exit: measured at 1.04s to exit with a thread parked forever in a syscall. budget_seconds = 0 keeps the old inline call for the installer, which has already committed to a full pass and needs the real answer. The new test hangs until killed without this and passes with it. * Do not read our own staged launcher as a quarantined one _move_launcher_aside renames Scripts/unsloth.exe to .update-stale before setup runs, and setup.ps1's deep check runs inside that window, so scanning parent-relative rows made every ordinary Windows update see its own launcher missing: the fast path was lost and the core package force-reinstalled over the network on a healthy no-op update. A recorded file that is absent with one of the three siblings _recover_missing_launcher reads beside it is staged, not quarantined. Absent without one is still damage. _core_package_names compared the package name exactly while verify_install canonicalizes it, so `--package Unsloth` had the deep check scan unsloth-zoo and force a pass that neither repair gate would then act on. Two unit tests called the real installed_versions through the presence check added beside the scan, so they answered for the host rather than the code and failed on a checkout with no unsloth distribution installed. * Narrow the staged-launcher excuse, and refuse a helper that will not import The staged exemption applied to any recorded file and asked only whether a sibling existed. Both were too loose: nothing but Scripts/unsloth.exe is ever moved aside, so a stray .update-stale could hide a quarantine anywhere in the tree, and the transaction recovers through _is_valid_pe, so an orphaned or truncated copy is one it would reject while this accepted it and left the launcher missing with the fast path intact. Same two-byte test now, and only for that name. An install_manifest.py that is present but will not import is damage, not an old release, and it is the one file whose damage silences every check that follows it. Both installers now force the dependency pass for that. Absence keeps the old escape: setup.sh is resolved out of the installed studio package so it should be impossible, separating it from an old release here would need a RECORD walk inside the installer, and the CLI already reports that case as studio_install_manifest_missing. Verified by extracting the probe from setup.sh and running it: absent keeps the fast path, truncated and raising both force the repair. * Trim the comments added since the opening pass Comments and docstrings only, verified with comment_tools.py check --strip-docstrings. 120 comment lines to 73. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-31 04:01:34 -07:00
Daniel Han	d5b153c2f8	Studio: keep a Windows update from leaving a CPU-only PyTorch (#9857) ... * Studio: keep a Windows update from leaving a CPU-only PyTorch The in-app updater runs `unsloth studio update`, which runs studio/setup.ps1, which runs studio/install_python_stack.py. install.ps1 is never on that path, and install.ps1 holds the only PyTorch flavor repair there has ever been. Meanwhile the dependency steps in install_python_stack.py install with deps and without an --index-url, so the resolver's default source is PyPI, whose Windows torch is 2.11.0+cpu, and constraints.txt names no torch to stop it. Both Windows repair passes are disabled there: _ensure_cuda_torch returns early on Windows, and the step 13 repair block is gated on `not IS_WINDOWS`, even though its own comment says the steps above can pull torch from PyPI. The result is a cu124 venv that comes out of an update as 2.11.0+cpu with the update reporting success. The app then runs everything on CPU, Settings shows no GPU, and the bundled CUDA llama-server prints "Available devices: (none)" because a +cpu venv ships no cudart64_*.dll for it to load. Add a Windows torch flavor invariant as step 13w. It resolves the flavor the install actually selected (env var from setup.ps1, then the install manifest, then a live probe), compares it against what is in the venv, and reinstalls the bounded trio from the correct index on a mismatch. It prints install.ps1's line verbatim so a support log from either path reads the same. If torch is still CPU-only afterwards on a host that expected a GPU build, the install now fails rather than reporting "dependencies up to date". It is conservative by construction: no-torch mode, a deliberate UNSLOTH_TORCH_BACKEND, a non-CUDA or absent expectation, a hidden GPU via CUDA_VISIBLE_DEVICES, and a torch that is missing or unimportable all no-op. Also stop the update path deleting a healthy venv. setup.ps1 collapses the expected tag to "cpu" whenever its bounded nvidia-smi probe comes back empty. On a direct update none of the existing escapes apply, so a working cu124 venv was removed and setup then aborted telling the user to run install.ps1, with no rollback copy because only install.ps1 keeps one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: unset the published torch flavor portably The publication block relied on `$env:X = ""` deleting the entry. That held on Windows PowerShell 5.1 and on 7.0-7.4, but 7.5 picked up .NET 9's change: an empty string is now a real value and only $null removes the name. The comment asserted the old behaviour as fact. Both spellings still read as unset on the Python side, which tests truthiness rather than presence, so nothing was broken today. The point is that a setup script should not publish a value whose meaning depends on which PowerShell the user happens to have. Remove-Item behaves the same on every version. Deleting rather than blanking matters on its own too: a value inherited from the caller's shell must not survive a run that decided it cannot name this host's flavor, or the stack enforces a stale expectation against a fresh venv. Found by running the block under a real PowerShell 7.6 over the host matrix. * Studio: close three gaps in the Windows torch flavor invariant Three separate holes in the invariant added earlier in this PR, all reachable. Do not route around an explicit index pin. When a pin's leaf names no flavor (a corporate /simple mirror), setup.ps1 publishes nothing, so the expectation fell back to the manifest, which records whatever was installed BEFORE the pin was set. Repairing off that stale tag rejects the pinned URL, because its leaf does not match, and reinstalls from the public pytorch index: it overrides an administrator's deliberate package source and fails outright on a network-restricted host. Decline instead, on the same test _ensure_cuda_torch already uses. Enforce the published XPU flavor, not just CUDA. setup.ps1 publishes "xpu" for an Arc host and installs the XPU trio before handing over, so treating that expectation as a no-op left the invariant carrying an answer it refused to act on. The exposure is identical: the dependency steps re-resolve torch from PyPI, and _ensure_xpu_torch cannot clean up afterwards because step 13's whole repair set is gated off Windows. Repairs with the XPU trio, whose floor is 2.6 rather than 2.4, since unsloth raises at import for an XPU device below that and a 2.5 wheel would satisfy the CUDA range and be kept. ROCm stays with the path that owns it: its wheels come from a per-architecture repo.amd.com index that cannot be reconstructed from a generic "rocm" tag. Classify a CUDA venv from disk when the torch probe hangs. A wedged display driver faults or hangs `import torch` on CUDA exactly as a faulted HIP runtime does on AMD. XPU and ROCm both rescue that venv by reading torch/version.py, but CUDA had no such arm, so the probe-failure chain fell through with a null tag, the no-wipe escape could not see a cu* wheel to preserve, and a direct update deleted a healthy CUDA environment before aborting. Keeps the family rather than a flat "cuda", because the stale comparison it feeds is cu126-vs-cu128, and matches with [regex]::Match so $Matches in the caller's scope is untouched. One existing test asserted the XPU no-op and has been updated: it was pinning the behaviour above, not guarding it. * Studio: enforce the published ROCm flavor too, by delegation The invariant enforced cu* and xpu but treated a published "rocm" expectation as a successful no-op. That left the AMD Windows path with the same hole the whole change exists to close: _ensure_rocm_torch runs at step 2b, which is BEFORE the dependency steps that can resolve torch back to PyPI's CPU wheel, and nothing ran it afterwards, so the update could still write a successful CPU-only manifest. The reason given for excluding it was that AMD's Windows wheels live on a per-architecture repo.amd.com index that a generic "rocm" tag cannot name. That much is true, and it is why this pass still does not build a ROCm URL of its own: setup.ps1 hands over $TorchInstallIndexUrl, which on the AMD path still points at /cpu. But it is not a reason to leave the flavor unchecked, because _ensure_rocm_torch already owns the whole problem -- it detects the arch, maps it to an index, honours an explicit ROCm pin, and re-probes the stale UNSLOTH_ROCM_TORCH_INSTALLED marker rather than trusting it. So the ROCm arm delegates to it and then applies the same verdict as the other two: still CPU-only afterwards fails the update rather than reporting success. The warning block is now a helper shared by both exits, so the three flavors cannot drift apart in what they tell the user. * Studio: do not skip the flavor invariant on an explicitly pinned GPU family The two gates at the top of _ensure_expected_torch_flavor both fired on a host that had pinned XPU or ROCm on purpose, which is the opposite of what the XPU and ROCm enforcement added just before this was for. _TORCH_BACKEND is not only install.sh's export: when it is empty, this module derives it at import from UNSLOTH_TORCH_INDEX_URL / _FAMILY, so a pinned xpu or rocm index sets it to "xpu" or "rocm" and the `not in ("", "cuda")` gate then skipped the only post-dependency Windows invariant there is. Those two values now fall through and are re-tested once the expectation is known: a GPU backend that DISAGREES with what setup.ps1 published is still a no-op, because the pin is the newer and more specific instruction, while one that agrees is enforced. "cpu" and unrecognised values still exit before the expectation is resolved, so that path costs no GPU probe. The second gate was a flat veto on _explicit_unknown_family_torch_index_url(). That helper's known set is rocm*/gfx*/cpu/cuXXX -- it predates XPU and does not count "xpu" as a family -- so an explicit XPU pin read as "unknown" and skipped the very host it was set for. It is now compared against the expectation instead: a pin whose leaf IS the family being enforced is not an unknown pin, and it is also the correct index to repair from. Widening the helper itself would have changed _ensure_cuda_torch and _ensure_rocm_torch too and drifted from install.sh / setup.ps1 / install.ps1, which it is documented to match. The parametrized deliberate-backend test still passed throughout, because it pairs a rocm/xpu backend with a cu124 handover, which is the disagreement case. It is split so the two reasons are stated separately rather than one standing in for the other. * Studio: let an explicit torch index pin outrank the recorded flavor The manifest records what a previous install left behind; an index pin is the instruction for the run happening now. Resolving the manifest first meant a freshly set UNSLOTH_TORCH_INDEX_FAMILY=cu128 over a cu124 manifest resolved to cu124, and _expected_torch_index_url then rejected the cu128 pin as a family mismatch and repaired from the public cu124 index, undoing both the family and the source the user had just selected. The pin now resolves second, after the setup script handover and ahead of the manifest. AMD leaves collapse to "rocm" so the tag matches the installed flavor vocabulary, and a leaf naming no family still falls through to the manifest, so the unknown-pin escape added earlier is unchanged. * Studio: keep the Windows ARM64 no-torchaudio exception in the flavor repair No win_arm64 torchaudio wheel is published on any index. setup.ps1 drops it from all four of its install trios, but the new final repair asked for it unconditionally, so a cu* expectation reaching that path on Windows on ARM turned a repairable venv into a failed install. The repair now drops torchaudio on the same condition, for the CUDA and XPU trios alike, using the _is_windows_arm64 helper the script already carries. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: verify the delegated ROCm repair, and enforce a pinned CPU index Two ways the invariant could report success over an environment that never got what it asked for. The ROCm arm accepted any GPU build. _torch_build_is_gpu is deliberately family-blind, so a transient repo.amd.com failure inside _ensure_rocm_torch, which is non-fatal there, left the old cu124 or xpu wheel in place, passed the check, and the manifest recorded rocm over it. The arm now re-probes the flavor and requires the family it asked for, with a distinct warning for a wrong family and the existing CPU-only warning kept for a venv that came back CPU. A venv nothing could read still passes, because ambiguity must not fail an update by itself. An expectation of cpu was never enforceable, but the unpinned dependency steps honour UV_TORCH_BACKEND, so a venv deliberately built against /cpu can come out of them holding a GPU wheel. A pinned CPU index is now enforced, and only a pin: setup.ps1 also publishes cpu when its nvidia-smi probe returns nothing, which is the wedged-driver host whose healthy cu124 venv the no-wipe escape in that file exists to protect, and acting on that handover would force it down to CPU. The verification for that case compares the family rather than asking whether a GPU build is present, which is the opposite question. * Studio: verify the repaired family, and re-settle what the torch release pins Four ways the invariant could still finish over the wrong environment. An untagged wheel under a CPU pin. _torch_flavor_tag reads every untagged version as cpu, which is right for PyPI and wrong for a private index serving an untagged CUDA or ROCm build: that wheel compared equal to a cpu expectation, skipped the repair, and was recorded as cpu. The runtime probe already carries the markers _ensure_cpu_torch uses to tell them apart, so the comparison consults them. The non-ROCm tail accepted any GPU build. A misconfigured mirror can answer a /cu128 request with a cached cu124, rocm or xpu wheel, and _torch_build_is_gpu is family-blind, so the update exited 0 and recorded the requested tag over a build that never arrived. The tail now re-probes the flavor and requires the family it asked for, as the ROCm arm already does. A CPU-only outcome keeps its own warning, since the remedy differs, and an unreadable venv still passes. torchao and xFormers are re-settled, but only when the repair actually moved the torch release, since the common case restores the wheel the venv already had. torchao's cpp extensions are release-specific and step 4 chose its pin from the torch this repair replaced, typically 0.17.0 for a 2.11 that the <2.11.0 spec takes down to 2.10, whose matched build is 0.16.0. xFormers is stricter: its _C.pyd is linked against one exact torch and CUDA pair, and beside any other pair the load is downgraded to a log line, so the import succeeds with memory-efficient attention, SwiGLU and the sparse ops silently gone. This script never installs xFormers and cannot reconstruct install.ps1's wheel selection, so a mismatched one is removed, which is what the backend's own resolver concludes as well: torch SDPA beats an extension that cannot load. Neither half can fail the update, both being secondary to the defect the repair has just fixed. * Studio: stop the post-repair resync from undoing the repair torchao depends on torch, so the --force-reinstall I added last round resolved dependencies and could re-resolve torch from the unpinned default index, which on Windows is the 2.11.0+cpu wheel the repair had just removed, reinstalled moments after removing it and after the verification had already passed. The update would then exit 0 and record a GPU tag over a CPU-only venv, which is the exact defect this whole change exists to close, reintroduced by its own follow-up step. The reinstall now passes --no-deps, which is not an optimisation here but the safety of the call: torch and the rest are already present and correct at that point. The helper additionally reports whether it ran anything that could touch torch, and the caller re-verifies the flavor when it did, so a regression cannot be silent even if the flag is ever wrong. The xFormers half was gated on the release moving, but its _C.pyd is linked against the exact torch and CUDA pair, so 2.10.0+cu124 to 2.10.0+cu128 after an index family change keeps the release and still leaves an extension that cannot load. The two are now gated separately: torchao on the release, xFormers on the full build label. * Studio: re-pin torchao when the CUDA major moves, not just the release _select_torchao_spec branches on cuda>=13 as well as on the release, so a repair that takes a venv from cu124 to cu130 at the same torch release changes the matched build while the release comparison alone reports nothing happened. The 0.16.0 extension cannot load beside a cu130 wheel, which is the same silent drop to the slow fallback the release gate was added for. The gate now fires on either. A flavor move within one CUDA major, cu124 to cu128, still leaves torchao alone, since nothing about its matched build changed there. * Studio: five more gaps in the Windows flavor invariant A visibility mask no longer vetoes an explicit expectation. An update inheriting CUDA_VISIBLE_DEVICES="" or -1 disabled the invariant outright, so the dependency steps could still put PyPI's CPU wheel over a cu124 venv and the update recorded the CUDA expectation over it, which is the whole defect this change exists to close. An emptied mask is a reason not to CONCLUDE a flavor from an nvidia-smi probe; it is not a reason to ignore one setup.ps1 published, one the user pinned, or one the last install recorded. The gate now applies only to the inferred case. The post-repair classification uses the same rule as the pre-repair one. Both compare through _torch_flavor_tag, which reads every untagged version as cpu, and only the pre-repair side consulted the runtime markers, so a private index answering a /cpu repair with an untagged CUDA wheel was accepted on a rule the check that triggered the repair had just rejected it on. The resync notices its own failures. pip_install_try and _uninstall_distribution report by return value rather than by raising, and both results were ignored, so an unreachable PyPI behind a reachable torch index, or an xFormers whose files were locked, left the wrong build in place while the update reported success. Both now warn. Neither fails the update, which stays deliberate. A direct Windows run now performs the XPU Triton swap. setup.ps1 does it after this script exits, so the helper skipped Windows entirely, but a direct invocation has no such postlude and the core package install pulls triton-windows over torch's XPU triton, leaving torch.compile unable to use the XPU. The handover variable gates it, since setup.ps1 sets it immediately before invoking this file and clears it otherwise. * Studio: ARM64 exception on the delegated ROCm repair, and a broken torch is not a hung one _ensure_rocm_torch's Windows branch asked for the full torch, torchvision and torchaudio trio unconditionally, so on win_arm64, where no torchaudio wheel exists on any index, the whole install is unresolvable. That is worse on the delegated path than on the plain one: the failure there is nonfatal, so the CPU build the repair existed to replace stays in place and the family verification then fails the update. The same _is_windows_arm64 exception the flavor repair and setup.ps1 apply now covers it. setup.ps1's disk-label rescue treated a definitive import failure exactly like a driver timeout. Both leave a +cu* version.py behind, and keeping the venv is right in both cases, since deleting one does not fix a driver and a faulted display or HIP runtime raises at DLL load rather than timing out, which is the bug the rescue was written for. But only a timeout says nothing about whether the installation is sound: a truncated torch took the same path, the family-matched install below then ran with bare requirements and no reinstall flag, and the run could write a completion manifest over a torch that still could not import. Invoke-BoundedPythonProbe now reports TimedOut, so the two are distinguishable, and a definitive failure keeps the venv but forces a same-family reinstall in place. Restricting the rescue itself to timeouts would have resurrected the original wipe, so it is the reinstall that is gated, not the rescue. One existing assertion in test_pr5940_followups.py looked for the three companion names inside the call itself; the trio is now built just above it so the ARM64 exception can drop torchaudio, and the assertion follows it there. * Studio: the XPU triton swap now has a third repair point on Windows Step 13w migrates torch on Windows for the same reason the two Linux repair passes do, and a direct `python install_python_stack.py` there has no setup.ps1 postlude to swap triton back afterwards. The platform guard test still asserted the old contract, where Windows was skipped unconditionally, so it covered the case the invariant exists for and called the fix a regression. Windows is now covered on both sides: it defers when setup.ps1 published the handover variable, and does the swap itself when nothing else will. The wiring tests count the third call site and check that it follows _ensure_expected_torch_flavor, which is that platform's torch migration. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reduce comment volume across the branch Comment-only pass over the eight files this branch touches: delete restatements of the code, section banners and narration, and collapse multi-line blocks that did not carry a distinct fact per line. Added comment lines 409 -> 140. No code or behaviour change. Installer suites still pass (3123 passed). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: a repair whose wheel will not import must fail the update Two problems on the verification that runs after the flavor repair. _installed_flavor_tag_now fell back to reading torch/version.py whenever the runtime probe did not answer cleanly, which lumps together two different facts. A probe that could not RUN is the wedged-driver host the fallback exists for. A probe that ran and reported that torch does not import is a definitive answer, and version.py still names the requested +cu124 tag from a half-written or DLL-less wheel, so the caller read tag == expected and wrote a completion manifest over a torch nothing can import. That is the same "reported success while unusable" state this whole change exists to remove, one layer later. The probe now reports that case distinctly and the update fails with its own message, which does not name the family: the wheel on disk may well be the right one, and what is wrong is that it does not load. A torch that failed to import BEFORE the repair cannot reach this, since the pre-repair chain returns early. Second, when the CUDA major moved and the replacement torchao could not be installed, the old one was left in place with a warning about the slow path. Across a CUDA-major move it is not the slow path: _select_torchao_spec exists because a torchao compiled for CUDA 12 cannot load its cpp under cu130, so the update completed holding a package that may fail on import. Remove it, the way the xFormers arm two lines down already removes a build made for a torch that is gone. A release-only move keeps the old warning, because that one really is just slower. Fixing the second exposed a hole in the harness those tests share: its pip_install_try stub only appended to a list, and list.append returns None, so every test in that class had been exercising the could-not-install branch without meaning to. The stub now returns a bool and takes a knob, so success and failure are different runs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: a torch that will not import must force its own reinstall $script:TorchImportDefinitivelyFailed is raised when the probe reports an import error rather than a timeout, so the wheel is the suspect and not the driver. Two things then ignored it. The flag never cleared $SkipPythonDeps, and every consumer of it lives inside `if (-not $SkipPythonDeps)`. On a current core package with a valid manifest the fast path announced a reinstall it then skipped, and setup exited successfully over an unusable wheel. The XPU and CPU arms never read it. Their force gates fire on a flavour change or a pin change, and neither applies here: an unimportable +xpu wheel still reports its on-disk tag as "xpu", so the bounded range is satisfied and the resolver keeps it. The ROCm arm already forces every time. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-31 02:58:18 -07:00
Maheswar Kumar	84c8b9aad2	studio: fix audio ci tests (#9953) ... Drops the module-level torchaudio import from test_native_audio.py, which is not installed in the studio backend job, and injects a fake save proxy in the one test that needs it. Re-anchors five frontend audio assertions onto the reformatted source, and updates two that follow real refactors: the replay guard now also checks modeRef, and transitionMode calls invalidatePendingTtsSelection. Replaces the diffusers pin test. The pin is diffusers==0.40.0, a released wheel rather than a source archive, so forcing it through --no-binary would defeat the pin.	2026-08-29 19:56:52 -07:00
Daniel Han	5bd09e13e0	Fix three CI regressions: code-span anchor, load-dispatch anchor, recordless pip install (#9959) ... * Fix three CI regressions: code-span anchor, load-dispatch anchor, recordless pip install Three unrelated failures, all reproduced locally first. 1. tests/studio/test_update_release_notes.py::test_code_span_closers_ignore_backslashes #9796 moved the code-span scanner into scanCodeSpans, which is defined ABOVE `export function codeSpans`. The test sliced the file from that export and counted `escaped(text` in the remainder, so the count went from 1 to 0 and the assertion "only an opener can be escaped" fired on a file where exactly one opener is still the only escaped thing. Count over the whole module and check the surviving call guards the opening run, so a wrapper moving again cannot read as "no opener is escaped either". 2. studio/backend/tests/test_parallel_slots_per_load.py::test_route_resolves_slots_once_before_dedupe_guard_and_load The GGUF load moved into _run_gguf_load_attempt, a health-wait helper defined above _load_model_impl, so `llama_backend.load_model` left the function body and .index() raised ValueError. Every ordering the test guards still holds. Anchor on whichever call the body dispatches a load through instead of on one backend's method name. 3. startup windows-latest pip will not replace a distribution whose .dist-info carries no RECORD, and an interrupted install leaves exactly that. The job reuses its venv, so the stub is not transient: once written, every later run dies the same way on "Cannot uninstall pydantic-core / no RECORD file was found". pip_install now retries once after clearing only the distributions pip named as unremovable, and only when it named one. pip's own hint is --ignore-installed, which would apply to every package in the command and skip real upgrades with it. run() grows check=False so pip_install can inspect a failure instead of exiting inside it; every existing caller keeps the old exit-on-failure behaviour. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Mirror the StageRoot branch into install.ps1's shared copy #9890 added staging support to studio/setup.ps1 and put an early `if ($StageRoot)` return inside Get-ManagedLlamaCppDir, which sits in the block install.ps1 holds a byte-identical copy of. The copy never got it, so test_denied_llama_cpp_preflight.py failed on exactly the drift it exists to catch, taking Repo tests (CPU) down with it. install.ps1 never assigns $StageRoot, and it runs under Set-StrictMode -Off from line 25 onwards, so the read is $null there and the branch is inert. Parity is what the block is for: the copies have to match or the next person edits one of them and only finds out in production. --------- Co-authored-by: danielhanchen <unslothai@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-29 19:55:55 -07:00
oobabooga	90644db9e8	Studio: repair duplicate package metadata during updates (#8515) ... * Studio: repair duplicate package metadata during updates * Studio: clarify duplicate metadata recovery * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: preserve metadata repair across install paths * Studio: complete duplicate metadata repair * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden duplicate metadata repair * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: reject unreadable package metadata * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: simplify metadata conflict handling * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: protect malformed metadata repair on Windows * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: verify editable foreign environments * Studio: reject versionless foreign metadata * Studio: repair malformed package metadata * Studio: do not read one site-packages twice on a lib64 build purelib hardcodes lib while platlib follows sys.platlibdir, and venv creates lib64 as a symlink to lib. On Fedora and SuSE the two schemes are therefore two names for one directory, and deduplicating the scan roots by string kept both. installed_versions() then returned every record twice, so metadata_conflict() was true for every installed package in a perfectly healthy environment: update would exit 1 telling the user to recreate the venv, and the repair could never converge because the duplication was an illusion. Deduplicate on the real path instead. * Studio: apply the same scan rules to the CLI's own metadata checks Two changes to _studio_deps, both keeping it consistent with install_manifest: _scan_paths and _venv_site_packages deduplicate their site-packages roots by real path, so a lib64 build does not report every installed package as duplicated, locally or when inspecting a foreign managed venv. installed_metadata_conflicts grouped on METADATA's Name and dropped records without one, so a nameless or non-UTF-8 record was invisible to it while installed_versions() and _distributions_in() both called the same directory a conflict. It now applies the dist-info filename fallback the manifest already uses. Such a record is marked unreadable rather than trusted, so damaged_installed_files keeps skipping it: pip cannot parse it either. * Studio: fetch the replacement before removing any metadata record The uninstall loop removes every record it finds, and the reinstall went through pip_install, which routes failure into run() and exits the process. So a failure between the two left the venv with no unsloth at all, took the entry point with it, and never reached the function's own return False. An unreachable index was one way in; --package with a name that is not on an index was certain to hit it, since such a package can never be reinstalled by bare name. Stage the replacement with pip download --no-deps first, install from that directory, and report a failed uninstall or install instead of exiting, so the caller's return 1 runs and the existing install is left alone when the replacement cannot be obtained. * Studio: probe for duplicate metadata in install.sh too install.ps1 and both setup scripts were moved onto installed_version_probe, but install.sh still reported the installed version through importlib.metadata.version(), which answers from whichever record the finder yields first. The same duplicated venv was therefore reported as a conflict on Windows and as an ordinary version on Linux and macOS. Both installers also fall back to the plain version lookup when studio.install_manifest cannot be imported, which is the case when --package installs something that does not ship studio/: that used to print the version and would otherwise have started warning that it could not be determined. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make a failed metadata repair non-destructive in both directions Two gaps in the staging I added, both reproduced. An unreadable record has to be out of the tree before pip runs, since a non-UTF-8 METADATA makes pip list, show and uninstall raise for the whole environment. It was deleted outright, so a package whose ONLY record was unreadable lost it and was left with files and no install, while the message said the existing install had been left in place. It is now moved to a quarantine directory and put back on any outcome short of a completed repair. pip download leaves an sdist when the source has no compatible wheel, and the install that follows runs --no-index, so its isolated build could not fetch setuptools and the package stayed uninstalled. Stage with pip wheel instead, which builds while the index is still reachable, and require a wheel to be present before treating the package as staged. * Studio: apply the repo's kwarg-spacing formatter to the metadata repair changes * Studio: stage overlay sources too, and put the payload back on a partial uninstall Three gaps, each reproduced. --local pulls unsloth-zoo from git, so an overlay is a network fetch just as much as an index install is, yet the overlay path skipped staging entirely. An unreachable GitHub therefore left unsloth-zoo uninstalled after the loop had already removed every record. Stage whichever source will be installed, and let a failed overlay fall back to the staged artifact, which was built from that same source so it never substitutes a release. The uninstall loop returned as soon as one uninstall failed, even though an earlier successful one had already deleted the package tree, leaving a surviving dist-info that claimed an installed core package whose files were gone. It now reinstalls from the staged replacement first. installed_metadata_conflicts treated a record with a name but no version as readable, so a single such record was silently trusted while install_manifest.metadata_conflict called the same directory inconsistent. Readability now requires both, and the message distinguishes an unreadable record from a duplicated one. * Studio: build the foreign venv fixture at the platform's interpreter path The cross-venv tests write bin/python, which _venv_site_packages only looks at on POSIX. On Windows the probe has nothing to run, so the fixture falls through to the glob fallback and the resolution under test never happens: both new cases failed there while passing on Linux and macOS. Found by running these files on a windows-latest runner. The repo's Windows matrix covers three parity files only, so nothing else would have caught it. * Repair duplicate core metadata again before writing the manifest The repair pass runs before the core packages are installed, so an upgrade that itself leaves a superseded .dist-info behind survived it. write_manifest then recorded a null version and the installer exited successfully while every later check rejected the environment. Run the same repair a second time immediately before write_manifest. It is a no-op when nothing is ambiguous. * Keep the metadata repair working under uv index and hash policies Three ways a hardened or privately mirrored machine could defeat the duplicate-metadata repair: - pip wheel obeys require-hashes exactly as pip install does, so staging a replacement was refused before anything was built and the repair aborted on the conflict it exists to remove. Confirmed on pip 26.2. _relaxed_pip_policy_env now covers wheel alongside install and download. - The staged wheel was reinstalled through uv when uv is the package manager, and UV_REQUIRE_HASHES rejects the unpinned name after the uninstall loop has removed every record, leaving the package gone. Both staged reinstalls force pip; the wheel is already built. - uv has no wheel subcommand, so staging runs pip, which reads none of UV_DEFAULT_INDEX, UV_INDEX, UV_INDEX_URL, UV_EXTRA_INDEX_URL or UV_FIND_LINKS. A private build could be uninstalled and replaced by a public PyPI wheel. Those are now translated into the pip equivalents, and only where pip has no setting of its own. * Keep uv index priority and upload cutoff when staging a repair wheel Staging runs pip because uv has no wheel subcommand, and the previous commit handed uv's indexes across as PIP_INDEX_URL plus PIP_EXTRA_INDEX_URL. That restores the reach but not the semantics. uv's default index-strategy is first-index: it stops at the first index carrying the package, which is what stops a public release shadowing a private one. pip pools its indexes and takes the highest version, so a newer public wheel could replace a private build. Offer each index alone instead, in uv's own priority order, and take the first that can build the wheel. An inherited extra index is cleared for those attempts so the candidates stay separated. UV_EXCLUDE_NEWER was ignored the same way. pip's --uploaded-prior-to is the same upload-time filter and takes the same date spellings, so the cutoff is passed through. It only exists from pip 25.3; on an older pip staging refuses rather than install a wheel the cutoff forbids, which leaves the existing installation intact. * Only advance past an index that confirmed the package is absent Staging offers each uv index alone, in uv's priority order. The loop treated every failure as absence, so a private index that was merely unreachable or refused the credentials advanced to the public fallback, which is the dependency confusion first-index exists to prevent. pip cannot be asked directly: measured on pip 26.2, a refused connection and an unresolvable host both print the same two no-match ERROR lines a genuine absence does, above a WARNING about retrying after a broken connection, and never the Could not fetch URL string. So a no-match is only believed when no transport, authentication or TLS marker appears beside it. Anything else stops the loop with the install still intact. Also honour UV_INDEX_STRATEGY: unsafe-best-match really does pool every index and take the best version, which is pip's own default, so those staging attempts are pooled too. first-index and unsafe-first-match both exhaust one index before the next, which is what the loop already does. * Ask uv which release and index to stage instead of reconstructing them Staging must run pip, because uv has no wheel subcommand. Reading uv's index configuration out of the environment and translating it cannot be made correct: uv also discovers uv.toml, pyproject [tool.uv] and a user config, honours UV_CONFIG_FILE, applies an implicit PyPI default, and resolves under an index-strategy pip has no equivalent for. A private index configured any of those ways was invisible, so the repair could uninstall a private build and reinstall the public package. Ask uv instead. uv pip compile --emit-index-annotation reports the exact index each package resolved from, under uv's own discovery, priority, strategy and upload cutoff, and pip is then pointed at that one index with that one version, for this interpreter. An unreachable higher-priority index fails the compile outright, which is the behaviour first-index exists to give, so the pip output markers that approximated it are gone. UV_OFFLINE forbids network access and pip has no offline mode, so the repair refuses rather than reaching the network. Plain pip is unchanged: it already reads its own configuration, so there is nothing to ask. * Stage a direct reference as written and replace pip's candidate sources Two gaps in the uv-resolved staging path. The overlay paths hand staging a git URL or a local checkout rather than a bare name. Such a requirement is its own provenance, since no index chose it, and uv appends the resolved commit to what it emits, so the name comparison could never match and every local duplicate unsloth-zoo repair aborted before it started. A direct reference is now staged as written, without asking uv. Replaying uv's answer also has to replace pip's candidate sources rather than add to them. An inherited PIP_NO_INDEX blocked the index uv picked, and an inherited extra index or find-links directory could satisfy the same version from a source uv never looked at, which is the provenance swap this path exists to stop. pip.conf can carry the same three settings, so it is dropped for that one command exactly as the pinned-index branch already does. * Keep pip's transport config and replay uv's artifact policy when staging Dropping pip.conf to remove its candidate sources took proxy, cert, client-cert and trusted-host with it, and those are how a private index is reached at all, so uv could resolve and pip would then fail to fetch, leaving the repair blocked. The environment overrides cannot do this alone: measured on pip 26.2, an empty PIP_EXTRA_INDEX_URL does NOT suppress an extra-index-url set in pip.conf, and pip contacts that index exactly as it does with the variable unset. So pip's own configuration is asked for with pip config list and written back minus the four source keys, which leaves the transport settings in force. A no-binary or only-binary rule was also ignored, so the repair could download a wheel under a no-binary rule or build an sdist under an only-binary one. --emit-build-options carries the uv.toml form through. Measured on uv 0.10.7, it does not surface the environment-variable spelling, so UV_NO_BINARY and UV_ONLY_BINARY are translated by hand, and only where pip has no setting of its own. * Keep index credentials and let a local checkout stage while offline uv emits every configured index with its credentials, but strips userinfo from the annotation that says which one answered. Taking the annotation at face value handed pip an unauthenticated URL for a private index, which answers 401 and aborts the repair, and reading only --index-url named the wrong index entirely when the credentialed one was configured as an extra. The annotation is now matched back to the emitted URL that carries the credentials, preferring the credentialed form when one index is emitted both ways. The offline refusal also ran before the direct-reference branch, so --local could not repair a duplicate unsloth record even though its replacement is a checkout on disk that needs no network. It now applies only to sources that are not already local. A git reference is still a network fetch and is still refused. * Remove the old payload when every metadata record is unreadable Quarantining the only record left pip nothing to uninstall, so the staged wheel was laid over the existing tree and any module the new release dropped stayed on disk and importable while the repair reported success. Give pip a parseable METADATA beside the intact RECORD instead, so it removes exactly the files that record lists. Confirmed against a real venv: with the METADATA corrupted pip show raises UnicodeDecodeError for the whole environment, and after the rewrite pip uninstalls the package and its entire payload. Without a usable RECORD this fails closed and asks for the environment to be recreated, since nothing then knows which files belong to the package. A find-links origin also no longer displaces the index. uv annotates a flat source with a file:// URL, which belongs in PIP_FIND_LINKS, already set from the emitted find-links lines; an sdist taken from a flat directory still needs the real index for its build backend. * Build a local checkout without the network when uv is offline The offline exemption for a local checkout let staging through, but pip builds it in an isolated environment and fetches the build backend for that, which UV_OFFLINE does not reach. Measured: an isolated build of a local project with no index reachable fails at installing build dependencies, and this repository pins setuptools and setuptools-scm exactly, so they would be fetched unless already cached. Either way the no-network policy was broken or the repair failed for a reason the user could not see. Offline local staging now builds against the interpreter's own backend and forbids the index outright, so no-network means no network. Giving up isolation is how the pinned build requirements stop being honoured, so it happens only in that case; every other staging keeps it. If the backend is not importable the build fails and the installation is left intact. * Pin the staged artifact by hash and carry uv's no-index policy Suppressing pip's other candidate sources turned out not to be achievable. Measured: with PIP_CONFIG_FILE pointed at a sanitised copy, a venv's own pip.conf is still read and its extra-index-url is still contacted, and --isolated does not suppress it either. So pip may always consult a source uv never considered. Ask uv for hashes instead and let pip enforce them. Whatever source pip reaches, the artifact has to be the one uv resolved, which is a stronger guarantee than removing sources and does not depend on suppressing any configuration. Measured: pip verifies the hashes even with PIP_REQUIRE_HASHES=0, and rejects an artifact that does not match. They are only accepted from a requirements file, written into the staging directory so it is removed with it. A flat source resolved with no index in play is also what a configured no-index looks like on the way out: uv emits the find-links entry and no index line. PIP_NO_INDEX is now carried in that case rather than cleared, which had handed pip back the default PyPI. * Apply every usable RECORD and commit each package as it is repaired Quarantining an unreadable record drops its RECORD on the floor. Where one sat beside a readable duplicate, the uninstall loop then removed only what the readable record claimed, so a module existing solely in the older release stayed on disk and importable while the repair reported success. Every unreadable record that has a usable RECORD is now made parseable so pip uninstalls it normally; only what cannot be rewritten is moved aside, and if that leaves nothing to uninstall the repair fails closed as before. The quarantine was also shared across both core packages. If the first was reinstalled and the second then failed, the rollback restored the first package's stale record on top of the install that had already replaced it, reintroducing the conflict and leaving an old RECORD describing a payload that was gone. Each package now has its own, and it is discarded as soon as that package is back in place, so rollback only ever touches the package being repaired. * Back up rewritten metadata and accept a direct-reference pin from uv The rewrite has to happen before staging, and staging can still fail. Overwriting the original METADATA in place left no way back, and what remained parsed, so the next run would see one readable record, decide nothing was wrong, and never attempt the payload repair still owed. The quarantine now keeps a copy of every file it rewrites and puts it back on any unsuccessful exit, byte for byte, so the conflict is still detected. A record whose backup fails is quarantined instead of rewritten. uv also emits a PEP 508 direct reference rather than an == pin when an override redirects a package to a path, repository or URL. The whole line was being compared as the name, so the requirement stayed empty and every repair under that policy aborted. The name is now parsed from either spelling while the reference itself is staged as written. * Drop an import left unused by a replaced test The repo lint blocks on an import the PR adds but does not use. shutil was needed by a staging test that a later round replaced. * Pin the scan boundary so a record outside the venv cannot fail it _installed_version answered from all of sys.path before this branch and now scans purelib/platlib only. Simulated against a real pip --user install: the old lookup reports the version, the new one reports nothing. Unreachable in production, since every caller runs against the managed venv and Studio never creates one with system site-packages, but nothing pinned the intent. * Cover the two ways a duplicate record actually appears pip renames the outgoing distribution to a ~ prefixed sibling during an upgrade and a kill mid-operation keeps both, which is the documented real-world source of this state and had no test. The opposite case had none either: a setuptools editable install resolves to a dist-info in site-packages plus an egg-info in the checkout, and scanning purelib and platlib rather than all of sys.path is what stops that reading as a conflict. * Tighten four comment blocks without dropping any of their reasoning * Make an unusable METADATA repairable whether it is corrupt or absent invalid_metadata_paths returns Path, so _rewrite_minimal_metadata raised AttributeError on rstrip for every record whose METADATA was present but unparseable, which is the case the rewrite exists to handle. The surrounding tests passed str and hid it. An absent METADATA took a second route to the same end: back_up reported failure, the rewrite was skipped and the record was quarantined instead, so the uninstall loop removed only what the readable record claimed and a module shipped solely by the older release stayed on disk and importable while the repair reported success. An absent file is now nothing to back up rather than a failure, and restore removes the synthetic METADATA instead of reinstating one that never existed. * Carry uv's keyring provider into the staging pip command The staging plan already replays uv's chosen index and its userinfo, but an index authenticated through the keyring CLI reached pip as a bare URL, so pip could not fetch what uv had just resolved and every repair on a private index aborted. Verified against the installed tools: uv 0.10.7 exposes UV_KEYRING_PROVIDER with disabled and subprocess, and pip 26.2.1 accepts both as PIP_KEYRING_PROVIDER, so the existing artifact-policy translation covers it unchanged. A uv.toml keyring-provider is not emitted and stays out of reach, same as the artifact policy. * Move pip's abandoned backup aside so the uninstall loop can converge An interrupted pip upgrade leaves the outgoing distribution renamed to a ~ prefixed sibling. Its METADATA still names the project, so installed_versions counts it as a duplicate, but pip calls the directory an invalid distribution and skips it, so uninstalling by package name can never consume it. Measured in a real venv: pip logged Ignoring invalid distribution ~impkg, the loop hit remaining >= record_count, and the repair returned False with both records still present, which would abort every future update on the commonest real cause of this state. The backup is now found by directory name and quarantined before the loop, so a failure to stage still restores it. Reported by review. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fail closed on an unrecorded stale record even beside a readable one _rewrite_minimal_metadata already refuses a record with no RECORD, because nothing then knows which files that release owned. The repair only honoured that when no other record survived. With one surviving, the count stayed nonzero, pip uninstalled just the readable record, and the quarantine was discarded on success, so whatever the older release owned alone stayed importable and the directory that was the evidence was deleted for good. Measured before the change: repair returned True, the stale module was still importable, and the stale dist-info was gone. Now the repair stops with the directory untouched, so a later run can still see the conflict. Four tests built their malformed record without a RECORD and asserted the repair continued, which is the behaviour above. They use an unversioned directory name instead, so the rewrite still fails and the quarantine path they exist to cover is unchanged. Reported by review. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse the repair for any record that cannot be made pip-uninstallable The previous commit only covered records with no RECORD. Any unrewritable record leaks the same way: quarantining it hides it, pip removes just the readable records, the quarantine is discarded once the reinstall succeeds, and whatever that release owned alone stays importable while the repair reports success and deletes the directory that was the evidence. The reachable route is a METADATA that exists but cannot be read, as an elevated install leaves root-owned. Measured in a real venv with the file unreadable: repair returned True, the stale module was still importable and its dist-info was gone. It now stops with the tree as found. An unreadable METADATA that can be rewritten still repairs, as does an ordinary duplicate. Quarantine-and-proceed had no remaining purpose once nothing runs after an unusable record, so it is gone and its three tests collapse into one that pins the invariant. Quarantine itself is still used for pip's ~ leftover, which is what the restore-on-staging-failure test now covers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the rollback's own metadata when the quarantine unwinds When the rewritten record has already been uninstalled and a later uninstall fails, _restore_from_staged reinstalls the package before the finally block runs quarantine.restore(). If the staged wheel recreates the same dist-info path, restore then either overwrote the wheel's valid METADATA with the original corrupt bytes or, where the original was absent, deleted a record pip had just written, leaving the core package malformed after a recovery whose whole purpose was to make it whole. Reproduced as a unit: the repair returned False and reading the resulting METADATA raised UnicodeDecodeError. A successful staged reinstall now drops the backed-up copies, since the wheel's metadata is authoritative. The moved entries are kept, because a record pip cannot consume still has to go back exactly as found. Reported by review. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a sole pip backup as a metadata conflict for PR #8515 * Mark a foreign venv's sole pip backup as a conflict for PR #8515 * Tighten the docstrings this PR adds without dropping their reasoning --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-22 22:15:05 -07:00
karan yadav	dc26127a42	Studio: propagate required backend version to repair pipeline (#8610) (#8670) ... * Studio: propagate required backend version to repair pipeline (#8610) Fix second-launch infinite repair loop when installed backend version is outdated (#8610). When the Desktop App launches with an installed managed venv whose version is older than expected_backend_version() (e.g. 2026.8.4 < 2026.8.15), preflight flags the install as ManagedStale with desktop_backend_version_outdated. On auto-repair, unsloth studio update ran setup.sh/setup.ps1 from the old venv, which skipped python dependency installation because INSTALLED_VER == LATEST_VER on PyPI or PyPI timeout, leaving the venv unchanged. The installer fallback (install.sh/install.ps1) also lacked version floor pins on standard fresh paths, locking the user in a permanent repair error loop. Key changes: - Pass UNSLOTH_DESKTOP_BACKEND_VERSION from Tauri (update.rs & install.rs) to child process environments. - Force Python dependency pass in setup.sh & setup.ps1 when UNSLOTH_DESKTOP_BACKEND_VERSION is set and INSTALLED_VER < UNSLOTH_DESKTOP_BACKEND_VERSION. - Apply UNSLOTH_DESKTOP_BACKEND_VERSION floor constraint when updating core packages in install_python_stack.py. - Ensure standard fresh install paths in install.sh and install.ps1 use "unsloth>=2026.8.15". - Bump MIN_DESKTOP_BACKEND_VERSION in preflight/version.rs to "2026.8.15". - Add shell and Python unit tests for the fast-path escape and desktop backend version constraint. Fixes #8610 * Narrow desktop repair to version propagation * Handle version suffixes in repair fallback --------- Co-authored-by: imagineer99 <samleejackson0@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>	2026-08-19 19:12:22 +01:00
Daniel Han	88262f5901	Studio: keep the extras install working under a hardened uv.toml / pip.conf (#8579) ... * Studio: keep the extras install working under a hardened uv.toml / pip.conf A machine with security-hardened package-manager config could not install Studio. With `no-build = true` in ~/.config/uv/uv.toml, the "unsloth extras" step failed because extras.txt carries requirements that ship no wheel on PyPI at any version, and uv correctly refused to build them: x No solution found when resolving dependencies: `-> Because openai-whisper==20250625 has no usable wheels [...] hint: building from source is disabled for all packages (--no-build) The pip fallback then hit `require-hashes = true` in ~/.config/pip/pip.conf and rejected every requirement, since the shipped requirements files are pinned but unhashed. The wheel-less requirements are already audited and allowlisted for source builds in .github/scripts/clean-machine-assert.sh, so the installer now names them explicitly with a package-scoped --no-binary. That overrides a global no-build / only-binary policy for those four names only, and leaves the user's binary-only policy in force for every other requirement. A blanket --no-build or `--no-binary :none:` override would have discarded the policy entirely. Hash-required mode has no command-line equivalent, so it is switched off in the child environment of the installer's own pip commands. pip applies environment variables after config files, so index-url, trusted-host, cert and proxy settings from pip.conf all stay in force, and uv commands are untouched. The override never reaches os.environ, so the user's own later pip commands keep their policy. Pinned-index installs additionally drop the restrictive UV_* / PIP_* variables. That branch already neutralised the config files, but an environment variable outranks a config file, so a hardened shell could still fail a torch repair the pin was supposed to make deterministic. Fixes #8530 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the source-build exemptions into the later source-only installs Two paths still aborted under a user-level `no-build = true` once the extras step was fixed. The pinned Diffusers revision is a source ARCHIVE, and uv refuses to build one under no-build ("Building source distributions for `diffusers` is disabled"), so a hardened host died at "diffusers pin" instead. That step runs on every platform for python >= 3.10, which includes the macOS host in the report. The exemption is guarded on the version marker because python < 3.10 resolves a released wheel from diffusers-pin.txt that must not be forced through a source build. extras.txt pins MeCab==0.996.5 on macOS cp314 and up, the last release carrying an sdist, so no-build refuses it there too. MeCab is a C extension, so the exemption is conditional: every other host resolves 0.996.13 from a wheel and must not be pushed into a compiler-dependent build. Verified against uv 0.10 with a cold cache, which matters here: a warm cache reuses the wheel it already built for the archive and hides the failure. * Tighten the comments added by this PR * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-12 09:46:34 -07:00
Daniel Han	3a58fa5c41	Studio: apply base.txt on the install.sh and install.ps1 paths (#8195) ... * Studio: apply base.txt on the install.sh and install.ps1 paths install.sh and install.ps1 install unsloth and unsloth-zoo inline, then export SKIP_STUDIO_BASE=1 so setup.sh / setup.ps1 do not install the same two packages a second time. install_python_stack.py read that flag as "skip base.txt" and short-circuited the whole step: if skip_base: pass That was the same thing only for as long as base.txt held nothing but those two names. Add a third, pinned entry to base.txt and it reaches no fresh install on any platform: neither installer reads the file, and the one branch that does was skipped. It would only land later, if the user happened to run `unsloth studio update`. Every install.sh and install.ps1 path was affected, on every platform: CUDA, ROCm, XPU, CPU, macOS, local and non-local, fresh and migrated. Keep skipping the two core packages, which is all the flag was ever meant to avoid repeating, and apply whatever else base.txt asks for. When base.txt holds only the core packages, as it does today, there is nothing left to install and no extra subprocess runs. No-torch mode is untouched: it has its own list in no-torch-runtime.txt, which the installers do apply inline. The core-package filter parses the project name rather than matching on a prefix, so a future unsloth-<something> pin is not swallowed too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reconcile base requirements with current main * Preserve relative requirements includes across filters * Fix filtered requirements test cleanup * Separate core and shared base requirements * Preserve shared base requirement resolution * Keep the filtered-requirements and uv alias paths from aborting an install The adjacent temp copy raised PermissionError on a read-only requirements dir, and a symlink failure handed uv back the spaced path it cannot read. Fall back to the temp dir and to a copy respectively, and stop the real-extras tests leaving filtered files in the tree. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Count the MLX slot and survive an unusable base.txt Simulating every install path showed two gaps. base_total never counted the Apple Silicon MLX step, so `studio update` there ran 13 steps out of a declared 12 and recorded the wrong steps_total. And the new base.txt read happens before the manifest is dropped, so a missing or unreadable file aborted with a traceback where the old code reached pip; a BOM also read as content and scheduled an empty step. Progress coverage now spans both core paths on all four platforms. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the unreadable base.txt case independent of the mode bits chmod(0o000) denies nothing as root, which containerized test jobs run as, and Windows does not implement POSIX modes at all, so the case asserted None against a file it could still read. Raise from a patched read instead. * Tighten the comments this PR adds --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-08-11 00:06:23 -07:00
Daniel Han	310a4b5b40	Studio: keep the upgrade intent when the pip fallback runs (#8112) ... * Studio: keep the upgrade intent when the pip fallback runs pip_install runs uv and falls back to pip on any nonzero exit. The fallback built its command with _build_pip_cmd, which dropped --upgrade-package and its value because pip has no such flag. On the update path that made the fallback a no-op. install_python_stack passes --upgrade-package unsloth --upgrade-package unsloth-zoo with req = base.txt, and base.txt lists a bare unsloth-zoo and unsloth, so pip found both requirements already satisfied, installed nothing, and exited 0. _fail_if_install_damaged checks file integrity rather than versions, so nothing downstream noticed, and unsloth studio update reported success having upgraded nothing. This was not limited to the Windows in-use launcher that motivated #8109. Any uv failure reaches the fallback, including a network or index failure, so the same silent no-op was possible on Linux and macOS. Translate the flag instead of dropping it, and pin --upgrade-strategy to only-if-needed rather than relying on pip's default, since that default is what keeps the existing torch build from being re-resolved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-07 08:22:04 -07:00
oobabooga	5659a9c47e	Installer: keep status messages off the progress bar line (#8052) ... * Installer: keep status messages off the progress bar line * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix blank lines and stale progress state in PR #8052 Follow-ups on the centralised progress-line close: - Four ROCm messages still opened with a literal \n, which used to be the only line terminator. _safe_print() emits one too now, so inferred-gfx, Strix and gfx906 installs got a blank line. - _end_progress_line() caught only OSError, so a closed or detached stdout took down messages bound for stderr, including the manifest error paths. - install_python_stack() reset _STEP but not _PROGRESS_LINE_ACTIVE. Only _step() read it before; every _safe_print() does now, so an aborted run left a stray newline on the next run's first message. - _note() aligned to the value column in verbose mode, where there is no bar and no step line to align to. - Dropped the _end_progress_line() call that _safe_print() now makes itself. Tests: patch _HAS_COLOR in _render() so the layout assertions hold under FORCE_COLOR=1, plus AST guards for leading-newline messages and direct sys.stdout writes, and coverage for wrapping, verbose, colour, closed stdout and the entry-point reset. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten comments in the progress-line changes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-07 01:20:47 -07:00
Daniel Han	3ab8dce97a	install: let UNSLOTH_TORCH_INDEX_FAMILY / _URL override CUDA wheel detection (#6692) ... * install: let UNSLOTH_TORCH_INDEX_FAMILY / _URL override CUDA wheel detection get_torch_index_url (and the studio-update mirror _detect_cuda_torch_index_url) chose the torch wheel family solely by probing the host GPU, with no override. In a headless / container / CI build the host driver is visible via the /proc/driver/nvidia/gpus fallback but nvidia-smi cannot report a CUDA version, so the function fell back to its cu126 default and installed the wrong wheels (e.g. a cu128 image got cu126 torch). Add an explicit override checked before any probing, in both the shell installer and the Python studio-update path: - UNSLOTH_TORCH_INDEX_URL full index URL, used verbatim (wins) - UNSLOTH_TORCH_INDEX_FAMILY family (cpu, cu128, rocm6.4, ...) appended to the mirror base (UNSLOTH_PYTORCH_MIRROR still honoured) This matches how the published GPU images select CUDA -- vLLM and SGLang take the CUDA version from an explicit build ARG rather than detecting it, and the Unsloth Docker base image already pins the cu128 index directly. Desktop installs are unchanged: with no override set, detection runs exactly as before. Adds test_get_torch_index_url.sh cases for the override (family, full URL, precedence, mirror base, trailing-slash strip, empty-ignored). * install: make the torch-index override authoritative across ROCm paths Address review feedback on the override added in this PR so a pinned index is honoured everywhere, not just in get_torch_index_url: - Skip the WSL ROCm bootstrap (root privilege + large downloads, probes /dev/dxg) when UNSLOTH_TORCH_INDEX_URL / _FAMILY is set; it previously ran before the override was consulted. - Skip the Radeon/Strix rerouting (which re-probes the GPU and overwrites the resolved URL with repo.radeon.com / repo.amd.com) when the index is pinned, so an explicit ROCm override (e.g. UNSLOTH_TORCH_INDEX_FAMILY=rocm6.4) is kept. - install_python_stack.py: derive _TORCH_BACKEND from the override when UNSLOTH_TORCH_BACKEND is unset (standalone studio update), so _ensure_rocm_torch / _ensure_cuda_torch repair to the requested family instead of re-detecting. - Strip ALL leading/trailing slashes in the shell override to match the Python side (avoids 404s on strict pip proxies). Adds test cases for double-slash and leading/trailing-slash overrides. * install: honor pinned torch index in CUDA/ROCm repair paths Follow-up to the override work in this PR: the get_torch_index_url / install.sh reroute already respect a pinned UNSLOTH_TORCH_INDEX_URL / _FAMILY, but the Python repair helpers in install_python_stack.py still re-probed the GPU and could overwrite the pinned family. Make the pin authoritative there too: - _ensure_cuda_torch: an explicit cu* pin commits to CUDA wheels, so repair a ROCm-poisoned venv even when no NVIDIA GPU is visible here (headless / container / CI cross-install), instead of bailing on the GPU-presence gate. - _ensure_rocm_torch: skip the AMD per-gfx (Strix) reroute when a ROCm index is pinned, and in the generic reinstall path install from the pinned URL verbatim rather than re-detecting the host ROCm version. gfx*/rocm7.2 indexes serve torch 2.11+, so select the 2.11 package specs for a gfx leaf. - install.sh: raise the torch constraint to 2.11 for */gfx* indexes too, matching rocm7.2, so a pinned full-URL/family override that returns early keeps a valid constraint. Add _explicit_torch_index_url / _explicit_rocm_torch_index_url helpers and tests covering the no-GPU CUDA pin repair and the explicit gfx index honored verbatim. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: honor torch-index override on the Windows installers too The pinned-index work landed for install.sh and install_python_stack.py, but the Windows installers still picked the wheel index from GPU probing. Extend the same UNSLOTH_TORCH_INDEX_URL / _FAMILY contract so a pinned index wins on every platform: - install.ps1: Get-TorchIndexUrl returns the pinned URL/family before nvidia-smi probing; the AMD ROCm reroute is skipped when the index is pinned, so an explicit cpu/cu* pin on an AMD host is not overwritten. - studio/setup.ps1: add shared Get-PinnedTorchIndexUrl / Get-TorchIndexLeaf helpers; the stale-venv check, the install selection and the AMD reroute all honor the pin, and the CPU/CUDA install pulls from the resolved index URL. - tests: parity test that all four installers read both override vars and the two Windows installers gate the AMD reroute on the pinned flag. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: complete pinned-index handling for ROCm/Windows edge cases Follow-ups to the override work flagged in review: - install.ps1: a pinned gfx*/rocm>=7.2 index previously skipped the AMD reroute that sets the torch>=2.11 floor, so the generic install used torch>=2.4,<2.11 and could resolve the known-bad _grouped_mm wheel. Route a pinned ROCm index through the ROCm install path with the 2.11 floor + companions, and guard the companion-spec lookup so a skipped reroute block cannot null-deref. - studio/setup.ps1: the stale-venv check compared the installed flavor (cuXXX/cpu, with +rocm misread as cpu) against the raw pinned leaf (gfx1151 / rocm6.4), so a correct pinned ROCm venv was always marked stale. Classify +rocm wheels as the generic 'rocm' flavor and normalize a pinned rocm*/gfx* leaf to 'rocm' before comparing (cu* stays specific so cu126-vs-cu128 still rebuilds). - install_python_stack.py: _ensure_cuda_torch now also reinstalls from a pinned CUDA index when the venv carries a CPU wheel (headless CPU-venv-to-CUDA cross-install via 'studio update'), not only when it finds a ROCm build. - tests: parity assertions already cover all four installers honoring the override. * install: finish pinned ROCm/CUDA edge cases on Windows + repair path Follow-ups to the previous round: - studio/setup.ps1: a pinned gfx*/rocm>=7.2 index now routes through the ROCm install path with the 2.11 floor + companions (it previously fell through to the CUDA branch with bare torch/torchvision/torchaudio against the ROCm index). The CPU/CUDA fallback index is forced to the CPU wheel index when a ROCm index is active, so a failed pinned-ROCm install does not retry the ROCm mirror. - studio/setup.ps1: the stale-venv check no longer treats an unrecognized pinned URL leaf (e.g. a PEP 503 mirror ending in /simple) as a torch flavor tag, which was marking a correct venv stale; cu*/cpu/rocm/gfx leaves are still compared. - install.ps1: the post-failure CPU fallback uses an explicit CPU index instead of , which for a pinned ROCm index was the ROCm mirror itself (so the 'fallback' just retried the failing index and aborted the installer). - install_python_stack.py: _ensure_cuda_torch now also reinstalls when the venv's CUDA family differs from a pinned one (installed cu126 vs pinned cu128), not only CPU->CUDA; the probe reports the installed cuXXX tag for the comparison. * install: keep the ROCm to CPU fallback install inside the retry-helper window The pinned-ROCm CPU fallback computes an explicit CPU index, but the comment explaining why it cannot reuse $TorchIndexUrl pushed the actual Invoke-InstallCommandRetry / --force-reinstall call more than 600 chars past the "ROCm PyTorch install failed" message, so test_pr5940_followups's window check no longer saw the retry helper. Move the CPU-index computation and its comment above the failure substep so the retrying force-reinstall stays adjacent to the message. No behavior change: same explicit CPU index, same retry, same --force-reinstall. * install: address #6692 review round 5 (ROCm/CPU pin edge cases) setup.ps1: - Stale-venv check: treat an AMD/ROCm host (HasROCm or a resolved gfx arch) with no explicit pin as expecting "rocm", not "cpu", so a healthy +rocm venv is not flagged stale (which made installer-managed setup exit and direct update rebuild). - Pinned-ROCm install failure now routes into the force-reinstall CPU branch: CuTag stays the rocm/gfx leaf on failure, so the condition also checks ROCmCpuFallback; otherwise the CUDA branch installed from the CPU index without --force-reinstall and kept the partial ROCm torch. - Explicit ROCm pin compare no longer collapses gfx*/rocm* to a generic "rocm": it compares the +rocmX.Y version (and the torch 2.11 line for gfx pins) so changing the pinned family (e.g. rocm6.4 -> gfx1151) rebuilds and applies it. install_python_stack.py: - _ensure_rocm_torch: an explicit ROCm wheel-index pin now bypasses the NVIDIA-present / no-AMD-GPU / unreadable-ROCm gates (headless/container/CI cross-install), mirroring the explicit-CUDA-pin bypass in _ensure_cuda_torch. - Add _ensure_cpu_torch: an explicit CPU pin (FAMILY=cpu or /cpu URL) now has a repair path that reinstalls CPU torch over an existing CUDA/ROCm build on a standalone update (which skips install.sh's flavor enforcement). install.sh: - Pin torchvision/torchaudio companions alongside torch for the rocm7.2 / per-gfx index and the Strix reroute (those AMD indexes publish companions independently and a bare name can resolve a torch-2.12-built wheel, an ABI mismatch). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * torch-index override: classify CUDA pin by leaf; trim blank shell overrides _ensure_cuda_torch only overrode the NVIDIA-presence gate for *any* pinned index, so a non-CUDA mirror URL (or a ROCm/CPU pin) on a non-NVIDIA host with ROCm torch could force a CUDA reinstall over a working ROCm venv. Add _explicit_cuda_torch_index_url() (leaf cu*), matching the ROCm/CPU helpers, and gate on it instead. install.sh::get_torch_index_url treated a whitespace-only UNSLOTH_TORCH_INDEX_URL / _FAMILY as authoritative (yielding an invalid index), unlike the Python .strip() and PowerShell IsNullOrWhiteSpace paths; trim leading/trailing whitespace first. * install: honor pinned torch index over CVD/GPU gates and fix leaf-based ROCm classification - install_python_stack.py: an explicit cu* pin now clears the CUDA_VISIBLE_DEVICES empty/-1 hide gate as well as the NVIDIA-presence gate, so CVD=-1 UNSLOTH_TORCH_INDEX_FAMILY=cu128 studio update repairs to CUDA wheels (parity with install.sh's get_torch_index_url override, which skips all GPU probing). Unpinned CVD=-1 still skips. - install_python_stack.py: _ensure_cpu_torch installs the bounded _CPU_TORCH_PKG_SPEC instead of a bare torch/torchvision/torchaudio trio; the /cpu index now also serves torch 2.11+, which is outside the supported <2.11 range. - install.sh: the torch>=2.11 constraint case matches the index leaf (rocm7.2|gfx*) instead of the whole URL, so a mirror base path containing a gfx/rocm7.2 segment with a cu*/cpu family is not false-matched onto the 2.11 line. - setup.ps1: the stale-venv check expects rocm torch only for arches the install path maps to a repo.amd.com wheel index; an unmapped/unreadable arch installs CPU, so a correct CPU venv is no longer marked stale. - Tests for each of the above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: tighten pinned torch-index override edge cases - install.sh: trim whitespace-only UNSLOTH_TORCH_INDEX_URL/_FAMILY before the _torch_index_pinned guard, matching get_torch_index_url, so a blank override no longer skips the WSL bootstrap and Radeon/Strix reroutes while detection still picks the normal index. - install.sh / install.ps1 / setup.ps1 / install_python_stack.py: force the torch 2.11 floor only for the gfx families with the <2.11 _grouped_mm bug (gfx120X-all, gfx1151, gfx1150). A pinned override to gfx110X-all/gfx90a/gfx908 stays on the default range, matching the automatic AMD path. - install_python_stack.py _ensure_cuda_torch: treat an untagged CUDA build under a CUDA pin as a family mismatch (reinstall), and match cuXXX pins narrowly (cu + digits) so a custom/current mirror leaf no longer forces CUDA over a CPU/ROCm venv. - install_python_stack.py _ensure_rocm_torch: reinstall when an explicit ROCm pin names a different ROCm family than the already-installed ROCm torch (the ROCm analogue of the CUDA cuXXX mismatch repair). Adds tests for each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: fix second-order edge cases in pinned torch-index ROCm/CUDA handling Parse the ROCm torch probe positionally so an empty HIP marker is kept: CPU/CUDA torch no longer reads as HIP, so the ROCm reinstall is not skipped. Emit one "<marker>|<version>" line (like the CUDA probe) for a robust parse. Limit the gfx torch 2.11 expectation to the install allowlist (gfx120X-all/gfx1151/gfx1150). A pinned gfx110X-all/gfx90a/gfx908 index stays on the default <2.11 specs, so a correct 2.10+rocm wheel is no longer judged a mismatch and force-reinstalled every update. Distinguish an AMD per-arch wheel (three-part +rocmA.B.C) from a generic pytorch.org wheel (two-part +rocmA.B): a gfx per-arch pin over a generic 2.11 wheel now reinstalls the per-arch wheel, while an already-installed per-arch wheel is not re-flagged (no reinstall loop). Mirror all of the above in setup.ps1 via new Test-RocmGfx211Leaf / Test-CudaFamilyLeaf / Get-RocmPinStaleTags helpers, reused by both the install-spec path and the stale-venv check so they cannot diverge again. Require a digit after "cu" (^cu[0-9]) in setup.ps1, install.ps1 and install.sh so a mirror leaf like /custom or /current is not branded CUDA and does not rebuild the venv every run. Add tests: CPU/CUDA probe -> has_hip_torch False; gfx110X-all pin + 2.10 wheel not stale; gfx1151 pin + generic 2.11 wheel stale; gfx1151 pin + per-arch wheel not stale; /custom and /current not CUDA; plus cross-language allowlist and cu-digit parity guards, and a PowerShell unit test for the new setup.ps1 helpers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix ROCm/gfx pin case normalization, ROCm-tag requirement, and CUDA-leaf classification Normalize torch-index leaves to lowercase before the gfx*/rocm*/cu* allowlist matches so the canonical gfx120X-all (capital X) gets the torch 2.11 floor in install.sh (leaf, flavor and repairable helpers). Require an installed +rocm local tag before a rocmX.Y or non-2.11 gfx pin is judged satisfied in setup.ps1 Get-RocmPinStaleTags and the Python _rocm_pin_family_mismatch, so an untagged CPU/CUDA wheel never leaves the pin unapplied. Classify a leaf as CUDA only via ^cu[0-9]: the Python _TORCH_BACKEND derivation now uses _is_cuda_family_leaf, and install.sh brands cuda only on cu[0-9]* (unset on an unknown /current /custom mirror leaf) so the stack probes the GPU instead of skipping ROCm repair. Add bash, Python and PowerShell tests for capital gfx120X-all floor, current/custom not-cuda, and untagged-wheel ROCm pins. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: converge torch-index pin detection via a per-venv marker Introduce a torch-index MARKER that records the exact wheel --index-url used after each successful torch install, so `unsloth studio update` / repair makes the "did the pinned index change?" decision by an EXACT string compare rather than inferring it from the wheel +rocm/+cu version tag. The tag cannot encode the AMD per-arch gfx family (two 2.11 gfx indexes both install +rocm7.13.0), so the tag heuristic missed a gfx1151 -> gfx120X-all switch and a custom-URL swap. Marker path is per-venv (.unsloth-torch-index), one line = the resolved index URL, written atomically (temp + rename). Path, format and normalization are shared across all four installers (install.sh, install_python_stack.py, setup.ps1, install.ps1). - Reapply gfx pins on a per-arch target change: the marker's exact compare reinstalls when the pinned index differs, even when both wheels share a tag. - Honor custom ROCm URL pins during repair: an explicit index whose leaf is not rocm/gfx/cu/cpu (e.g. simple, current) now reinstalls torch VERBATIM from the pin when it differs from the marker ("URL wins verbatim"). - Align the KNOWN-2.11 rocm/gfx set to exactly rocm7.2 plus the gfx allowlist gfx120x-all/gfx1151/gfx1150 in every language; stop treating an unknown newer rocm (rocm7.3, which does not exist) as the 2.11 line speculatively. Backward compatible: with no marker (old venvs, torch installed out-of-band) the existing +rocm/version-tag heuristics still decide, and a matching marker never reinstall-loops. A cu128 CUDA pin stays a CUDA pin; custom and current leaves are not CUDA. Adds marker tests (py/sh/ps) plus cross-installer parity checks. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: keep the torch-index marker additive to flavor validation Three narrow fixes in the marker-based stale-venv detection: - setup.ps1: a matching marker no longer overwrites the detected installed flavor. The marker compare is now an additional rebuild trigger, so a stale wheel (torch swapped to a +cpu build while the marker still records a cuXXX pin) is still caught by the flavor check instead of being masked as up to date. - setup.ps1: a supported AMD arch carrying CPU torch is no longer marked stale and wiped. The downstream AMD Windows ROCm override upgrades CPU torch to ROCm in place, so wiping first would delete the venv and abort with "Virtual environment not found". Only a genuinely wrong CUDA wheel still rebuilds. - install.sh: the Radeon --find-links path records its repo.radeon.com base in the marker instead of the generic pytorch.org ROCm fallback index, so a later pin to that generic family correctly reinstalls rather than comparing equal. Mirrors install.ps1/setup.ps1, which already record the real AMD index. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: honor custom pins and repair pinned venvs in place Four follow-ups to the torch-index marker work: - install_python_stack.py: _ensure_cuda_torch/_ensure_rocm_torch now bail when an explicit custom-index pin names no known torch family, so a verbatim URL override (a private/simple mirror) is not clobbered by auto-detected CUDA/ROCm wheels before _ensure_verbatim_torch_index applies it. - install_python_stack.py: the ROCm marker is additive, not a substitute -- a matching marker still runs the family/version check so a wheel swapped after the marker was written is caught. Mirrors setup.ps1. - setup.ps1: a stale venv under an explicit pin, whose torch still imports, is repaired in place (force-reinstall torch from the pin in the dependency pass) instead of wiped. The wipe path only delegates to install.ps1, so on a direct update it stranded the user at "Virtual environment not found" instead of applying the new pin. A broken venv or unpinned drift still wipes/delegates. - install.ps1: when a pinned ROCm install fails over to a CPU base, the marker now records the CPU index actually used instead of the ROCm pin, so the next managed setup does not see CPU torch under a ROCm pin and abort as stale. * setup.ps1: keep the ROCm CPU-fallback force line the pr5940 test guards 5c93ffd4 folded the pin-change force-reinstall into the ROCm CPU-fallback condition on one line, so the exact literal that test_pr5940_followups.py checks (if ($ROCmCpuFallback) { $cpuForce = @("--force-reinstall") }) no longer appeared and the test failed. Split the two conditions into separate if lines: the ROCm fallback line is restored verbatim and the pin-change force is its own line. Both still set $cpuForce to the array, so @splat passes one arg. * install: honor exact CUDA/custom index URL pins in the torch-index marker Address three Codex review findings on the torch-index marker mechanism: - install.sh: after the ROCm CPU repair reinstalls torch from the generic $TORCH_INDEX_URL, record that as the marker source. A Radeon --find-links install set _TORCH_MARKER_INDEX_URL to its repo.radeon.com base earlier, so leaving it made the marker misreport Radeon wheels and a later Radeon pin would compare equal and skip a needed reinstall. - install_python_stack.py: _ensure_cuda_torch now consults the exact-URL marker (_marker_pin_mismatch) when the installed +cuXXX tag matches the pinned leaf, so a same-leaf CUDA mirror change (official cu128 to an internal cu128 mirror) is reinstalled and re-recorded instead of skipped. - _normalize_index_url / _normalize_family_leaf (install.sh, setup.ps1, install_python_stack.py): lowercase only KNOWN wheel-family leaves (rocm/gfx/ cpu/cuXXX) so gfx120X-all still matches gfx120x-all, while a custom (unknown-family) leaf keeps its case so a verbatim URL pin like /Current does not compare equal to /current. Tests updated to assert the refined behavior. * install: fix 3 torch-index marker edge cases (CPU mirror pin, Radeon leaf, migrated venv) Addresses three review findings on the torch-index override path: 1. CPU index URL change on an already-CPU venv. _ensure_cpu_torch returned early whenever torch was already a CPU build, so a standalone update that moved the pin (official /cpu -> a private UNSLOTH_PYTORCH_MIRROR /cpu, same +cpu tag) never reinstalled. It now consults the exact-URL marker and reinstalls only when _marker_pin_mismatch reports a different index, mirroring the CUDA/ROCm same-family handling. A matching marker (or none) still leaves CPU torch untouched, so there is no reinstall loop. 2. Radeon find-links directory misclassified as a pip ROCm family. A repo.radeon.com/.../rocm-rel-7.2.1 leaf starts with "rocm" but is a find-links listing, not a pip --index-url. The old startswith(("rocm", "gfx")) test routed it into a --index-url reinstall that fails against find-links. New _is_pip_rocm_family_leaf gates on ^rocm\d / gfx (matching install.sh's rocm[0-9]* and setup.ps1's ^(rocm[0-9]|gfx)), so a Radeon URL routes to the verbatim/marker path instead. 3. Migrated venv rewriting its marker to a pin it did not install. install.sh and install.ps1 write the marker unconditionally, so a migration that preserves existing torch recorded the newly requested pin and a later update then found a matching marker and skipped the reinstall the pin needs (e.g. a per-arch gfx1151 -> gfx120X-all switch, identical +rocm tag). Both now track _TORCH_INSTALLED_THIS_RUN and write the marker only when torch was actually installed or repaired this run. Also add Get-NormalizedFamilyLeaf to the setup.ps1 helper-extraction list in test_torch_index_marker.ps1 (it was added to setup.ps1 and the shell test in an earlier round but missed here) and add two unit tests covering findings 1 and 2. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: keep pinned torch repairs on the pinned index Two fixes for explicit index pins (UNSLOTH_TORCH_INDEX_FAMILY / _URL): 1. install_python_stack.py's repair paths ran uv without clearing the inherited uv index env vars. uv resolves the default index (--index-url or --default-index) at the LOWEST priority, so a UV_INDEX or UV_EXTRA_INDEX_URL mirror in the environment won for any package it served: a cu128-pinned repair could install torch from the mirror and then record the cu128 marker it never used. Verified empirically: with UV_EXTRA_INDEX_URL=.../cu126 exported, uv pip install torch --index-url .../cu128 resolves torch 2.13.0+cu126. Strip the four uv index env vars for pinned-index commands only, mirroring the gate install.sh, install.ps1 and setup.ps1 already have; non-pinned installs keep the user's mirror. 2. install.ps1 routed any pinned leaf matching rocm* through the ROCm --default-index path, so a custom find-links leaf like rocm-rel-7.2.1 was treated as a PEP 503 ROCm index and could silently fall back to CPU torch on resolution failure. Require a digit after rocm, matching install.sh's rocm[0-9]* and install_python_stack.py's ^rocm\d. Adds parity + unit tests for both (11 new tests). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: keep pinned repairs off UV_TORCH_BACKEND and narrow setup.ps1's rocm pin match Round 2 of the pinned-index hardening: 1. _build_uv_cmd converted UV_TORCH_BACKEND into --torch-backend before the new env isolation could act, and uv's torch backend redirects torch resolution to its own per-backend index even when --index-url is given (verified: a cu128-pinned dry run with UV_TORCH_BACKEND=cpu resolves torch 2.13.0+cpu). Pinned-index commands now never receive the flag and UV_TORCH_BACKEND joins the stripped env vars, so uv cannot re-read it. 2. setup.ps1's pinned reroute had the same bare rocm* glob install.ps1 had: a custom find-links leaf like rocm-rel-7.2.1 was routed through the ROCm --index-url path instead of the verbatim unknown-pin path. Now requires a digit after rocm, matching install.ps1, install.sh and _is_pip_rocm_family_leaf. 3. The marker test's case-normalization checks used -eq, which is case-insensitive in PowerShell, making them vacuous, and the unknown-leaf expectation was written lowercased while the implementation deliberately preserves custom-leaf case. Tightened to -ceq with the case-preserving expected value. Adds unit + parity tests for 1 and 2 (5 new tests). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: extend the pinned-index guards to every remaining surface Round 3 of the pinned-index hardening, closing the same holes on the surfaces the earlier rounds missed: 1. install.sh's pinned-install env scrub now clears UV_TORCH_BACKEND (uv's torch backend redirects torch resolution to its own per-backend index even against --default-index), and both PowerShell wrappers clear it in their pinned-install scrubs, matching install_python_stack.py. 2. setup.ps1's marker stale check still classified any rocm* leaf as a PyTorch ROCm family while the install selection is digit-gated, so a custom rocm-current / rocm-rel-7.2.1 pin stale-compared as not-rocm vs rocm and force-reinstalled on every studio update. The stale check now uses the same ^rocm\d gate. 3. install_python_stack.py's pinned-command scrub also strips PIP_EXTRA_INDEX_URL for the pip fallback: pip adds the env extra index in addition to --index-url, so an inherited mirror could satisfy torch off the pin while the marker recorded the pinned URL. PIP_INDEX_URL needs no strip since the explicit --index-url flag overrides it. Parity + unit tests extended (4 new tests). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: scrub find-links and carry the pinned scrub through pip fallbacks Round 4 of the pinned-index hardening: 1. UV_FIND_LINKS joins every pinned-install scrub (install.sh, install.ps1, setup.ps1, install_python_stack.py): uv's --find-links locations can satisfy torch off the pinned index the same way an extra index does. 2. setup.ps1's Fast-Install restored the scrubbed vars in its finally BEFORE the pip fallback ran, and never touched the pip env vars at all, so a failed uv attempt fell back to python -m pip with an inherited PIP_EXTRA_INDEX_URL / PIP_FIND_LINKS able to win over the pinned --index-url. The scrub now wraps the whole function (uv attempt + pip fallback) and includes the pip vars; restore happens after both. 3. install_python_stack.py's scrub also strips PIP_FIND_LINKS for its own pip fallback, completing the PIP_EXTRA_INDEX_URL fix from round 3. Parity tests extended (2 new tests). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: digit-gate rocm leaves in marker normalization and ROCm side effects Round 5 of the pinned-index hardening (three custom-rocm-leaf edge cases): 1. _normalize_family_leaf lowercased every leaf starting with rocm, so a custom mirror leaf like rocm-Current compared equal to its lowercase form and a case-only pin change was skipped. URL paths can be case-sensitive. The rocm prefix is now digit-gated (rocm[0-9]*, matching _is_pip_rocm_family_leaf) in install.sh, setup.ps1 and install_python_stack.py, so only true family leaves (rocm7.2) are lowercased; a custom rocm-* leaf keeps its case. 2. setup.ps1 Test-MarkerPinMismatch compared normalized URLs with -ne, which is case-insensitive in PowerShell, so a case-only marker change (Simple vs simple) was treated as matching and the reinstall skipped. Now -cne. 3. install.sh gated the AMD bitsandbytes install and the "repair ROCm torch" --default-index reinstall on a bare whole-URL rocm glob, so a custom CPU/CUDA/private index whose leaf merely starts with rocm (rocm-current) was force-repaired from the wrong ROCm-only path whenever torch.version.hip was empty. Both now gate on _torch_index_is_rocm_family, computed once from the digit-gated leaf (rocm[0-9]*/gfx*). Tests: 4 new parity assertions plus 2 case-sensitivity marker checks. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: apply an explicit custom torch-index pin on the first update Round 6: an explicitly-set custom (unknown-family) UNSLOTH_TORCH_INDEX_URL was silently ignored on the first `studio update` of a venv that predates the marker feature, on both platforms, because the no-marker case was treated as "do nothing" and the version-tag heuristics cannot judge an unknown leaf. 1. install_python_stack.py _ensure_verbatim_torch_index now reinstalls verbatim when the marker is ABSENT (None), not only when it differs, and short-circuits only when the marker already records this exact pin. It then writes the marker, so every later update is a no-op. A user who did not set the override gets pin=None and is untouched, so an out-of-band torch install is never clobbered. 2. setup.ps1: for an unknown-family pin on a marker-less venv the stale-venv check now sets PinChangedForceReinstall so the torch block reinstalls in place from the pin. It deliberately does NOT set shouldRebuild, which would wipe the venv and strand a direct `studio update`. 3. setup.sh (the Linux `studio update` entry point) skipped install_python_stack.py entirely when unsloth was already current, so the marker-driven reinstall (both the verbatim custom pin and the cu/rocm flavor and family-change repair, e.g. gfx1151 to gfx120X-all) never ran. It now forces the dependency pass when a torch-index pin env var is set; the pass is idempotent and no-ops when the marker already matches. This mirrors setup.ps1's stale-venv pre-check. Tests: 3 new parity assertions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test: expect first-update reinstall for a no-marker custom index pin Follow-up to d671d8fb2: _ensure_verbatim_torch_index now applies an explicit unknown-family URL pin verbatim on the first update when the marker is absent (instead of no-op), so the old test_verbatim_custom_url_no_marker_is_noop assertion was stale. Rewritten as test_verbatim_custom_url_no_marker_reinstalls_once: asserts the one verbatim reinstall from the pinned URL, that the marker is written, and that a second call with the pin still set is idempotent (no reinstall loop). * install: gate the pinned update pass on the marker and record a pin baseline Round 8, two follow-ups to the round-6 first-update pin fix: 1. setup.sh forced the full dependency pass on EVERY `studio update` while a torch-index pin stayed exported, even after the marker already recorded the same pin, turning quick updates into the expensive pass every time. It now probes install_python_stack.py --torch-pin-needs-apply (which reuses the exact marker normalization) and forces the pass only when the pin is not yet applied (marker absent or different); an already-applied persistent pin keeps the fast path. A probe error fails safe toward running the pass. setup.ps1 gets the same probe in its fast path for parity. 2. A known-family full-URL pin on a venv predating the marker (e.g. an installed cu128 build and UNSLOTH_TORCH_INDEX_URL pointing at a same-family mirror) left the marker absent forever: the _ensure_* helpers deliberately do not force a multi-GB reinstall of identical-family wheels on an old venv, so nothing recorded the pin and every update re-entered the pass. _record_torch_index_pin_baseline now records the resolved pin as a baseline after the ensure sequence when the family already matches and no marker exists, so the pin is tracked (a later genuine change is detected and applied) and the update loop is broken, without the redundant reinstall. Tests: 3 new baseline unit tests, 4 new parity assertions, and the CLI probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * setup.sh: keep the pin probe's exit 1 from killing the update under set -e The --torch-pin-needs-apply probe deliberately exits 1 for the common steady-state answer (pin already recorded, keep the fast path), but it ran as a bare command under set -euo pipefail, so the whole studio update aborted before the exit code was even captured. Absorb the status with || _PIN_NEEDS_APPLY=$? and pre-seed 0 so all three outcomes route as documented: 0 runs the pass, 1 keeps the fast path, anything else fails safe into the pass. Parity test asserts the guard. * install: strip pin credentials, disable uv config discovery, bound verbatim installs Four verified fix groups from a 12-reviewer audit of the torch-index override feature, each reproduced before fixing: 1. Credential persistence: all four marker writers stored the raw pin URL, so an authenticated pin (https://user:token@mirror/simple) persisted its credentials in .unsloth-torch-index (mode 0644 under a default POSIX umask) and install_python_stack.py printed pin URLs verbatim in repair messages. Userinfo is now stripped before persisting and in every log/substep that interpolates a pin, via lockstep helpers (_strip_index_url_credentials in install.sh / install_python_stack.py, Remove-IndexUrlCredentials in install.ps1 / setup.ps1). The three normalizers strip too, so an OLD marker that already carries credentials still compares equal to the same pin: no reinstall loop on upgrade. Query strings deliberately stay in the marker; two indexes distinguished only by query must not compare equal. 2. uv configuration discovery beat the explicit pin: with a discovered uv.toml declaring torch-backend = "cpu" or a [[index]] entry, uv 0.10.12 resolves torch 2.13.0+cpu against an explicit --index-url/.../cu126 pin; UV_NO_CONFIG=1 restores +cu126 (reproduced both ways). The pinned-install scrub in all four installers now sets UV_NO_CONFIG=1 and drops UV_CONFIG_FILE. 3. The verbatim custom-index update path installed a bare, unconstrained torch trio while fresh installs from the same unknown-leaf pin apply the supported range; _ensure_verbatim_torch_index now installs the bounded trio spec, closing the fresh-vs-update asymmetry. 4. Query-bearing pins (.../cu128?token=x) classified by raw leaf split and force-reinstalled on every update (the installed cu128 never equals cu128?token=x). Query/fragment are now stripped before leaf classification in all four implementations; the marker comparison keeps the query per (1). Rejected after verification (no change): the pin-baseline record cannot produce a wrong later decision (every pin change still mismatches and reinstalls from the new pin); the venv temp-file symlink scenarios require an attacker who already owns the environment; pathological inputs like " / cu128 / " have no realistic caller and fail loudly. Parity, stack, rocm-support, marker (sh + ps1), pin-stale, index-url and flavor suites all pass (455 python + full shell/ps1 batteries). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: harden custom-pin repair against clobber, broken torch, and pip config Four follow-ups to the pinned-index audit fixes: 1. setup.ps1 routed an unknown-leaf custom pin through the CUDA branch with a bare torch trio while install.ps1 (fresh) and the Python verbatim path bound the supported range; the pinned unknown-leaf route now applies the same torch>=2.4,<2.11.0 bound. Known cu* leaves and unpinned runs are unchanged. 2. The final torch safety pass could not repair a clobbered unknown-family pin: intermediate dependency steps can pull torch from PyPI (the pass exists for exactly that reason), but the verbatim helper short-circuited on marker==pin and no flavor tag exists to probe. The helper now keeps a per-run snapshot of the installed trio (taken after a verbatim reinstall or on the first matching-marker pass) and reinstalls from the pin when the final pass sees the trio drifted. Probe failure skips the comparison; a reinstall refreshes the snapshot, so no loop. 3. _record_torch_index_pin_baseline could freeze a known-family pin as applied on a venv whose torch is missing or broken (every family helper returns without reinstalling when its probe fails), making --torch-pin-needs-apply report done forever. The baseline now probes the installed flavor and records only on a match: a cuXXX pin requires the matching +cuXXX tag, cpu requires a cpu build, rocm/gfx requires hip; probe failure records nothing. 4. The pinned pip fallback stripped PIP_* env vars but user/site pip config files still applied (a configured global.extra-index-url can satisfy torch off the pin). PIP_CONFIG_FILE is now pointed at the null device for pinned commands (pip loads no config files then), in _install_env_for_cmd and setup.ps1's Fast-Install pinned scrub. install.sh / install.ps1 have no pip fallback (uv-only), verified. Tests: 7 new rocm_support tests (snapshot reset fixture), 1 stack test, 2 parity tests. Full battery green (464 python, sh and ps1 suites). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: complete the pin-repair coverage across the fast path and platforms Three cross-platform follow-ups to the round-2 pin-repair fixes: 1. The --torch-pin-needs-apply probe only compared marker==pin, so a torch trio clobbered to the wrong family (a cpu wheel replacing cu128 via a later pip install) with a still-matching marker reported "already applied" and the _ensure_{cuda,rocm,cpu} repair never ran on the Linux fast path. The probe is now a testable _torch_pin_needs_apply() that also checks the installed flavor against a known-family pin (via a shared _torch_flavor_matches_pin() helper, so the baseline and the probe cannot drift). An unknown-family pin has no flavor to validate and a failed probe cannot prove drift, so both keep the fast path. 2. macOS ARM (real CPU/MPS torch, not NO_TORCH) never applied an unknown- family custom pin on update: both the verbatim path and the baseline returned on IS_MACOS while fresh install.sh honors the pin, so the marker was never written and setup.sh forced the dependency pass on every update forever. The guards are now IS_MAC_INTEL (Intel mac is already NO_TORCH), and the final pass applies the pin on macOS ARM. 3. The round-2 final verbatim repair sat in the step-13 sequence guarded not IS_WINDOWS, so on Windows a dependency step that clobbered torch after the pin was applied was masked by the matching marker (setup.ps1 does not re-validate the main venv's torch after calling this script -- verified). Step 13 now runs the verbatim snapshot-drift repair on Windows and macOS ARM too; the Linux-oriented cuda/rocm/cpu family helpers stay Linux-only. Tests: 13 new rocm_support cases (flavor drift, macOS ARM, Windows repair), parity updates. Full battery green (475 python, sh and ps1 suites). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: strip query tokens from the marker and tighten the pin-drift probe Four follow-ups to the round-3 pin-repair fixes: 1. The credential stripper feeding the torch-index marker and the logged repair messages dropped only user:pass@ userinfo, so a private feed that carries its auth token in the query string (.../simple?token=SECRET) persisted the token in the world-readable marker (mode 0644 under a default umask) and printed it in substep output. All four strippers (install.sh, install.ps1, studio/setup.ps1, install_python_stack.py) now drop the query and fragment before building the sanitized URL. A query is not part of a PEP 503 index's identity, so this also stops a rotated token from spuriously mismatching the marker and forcing a needless reinstall. 2. The --torch-pin-needs-apply fast-path probe accepted an untagged CUDA build (no +cuXXX local tag) under a specific cuXXX pin, but _ensure_cuda_torch reinstalls exactly that build to enforce the pin. The probe was more lenient than the repair, so the repair pass was skipped on the fast path. _torch_flavor_matches_pin now reports a mismatch for an untagged build under a cuXXX pin, forcing the pass. 3. The probe's ROCm branch accepted any HIP build for a rocm/gfx pin, while _ensure_rocm_torch decides a reinstall with the per-arch _rocm_pin_family_mismatch predicate (a generic +rocm7.2 wheel under a per-arch gfx pin, or a wrong ROCm version, is a mismatch). The probe now reuses that predicate, so it is as strict as the repair. This needs the installed torch version, so _probe_torch_flavor now returns (marker, cutag, version) and _torch_flavor_matches_pin takes the pin URL (extracting the leaf internally). 4. On Windows a known-family cu*/cpu pin is applied to the main venv by setup.ps1 before install_python_stack.py runs; a later dependency step can clobber it, and the GPU-aware _ensure_{cuda,cpu}_torch self-skip on Windows while the verbatim helper handles only unknown-family pins, so nothing repaired the clobber (setup.ps1 does not re-validate the main venv's torch afterward, verified). New _ensure_pinned_known_family_torch reinstalls a drifted cu*/cpu pin in the step-13 Windows/macOS-ARM branch; rocm/gfx per-arch specs stay owned by setup.ps1, unknown-family by the verbatim helper. A speculative ROCm 2.11 floor was also raised but is unreachable: the rocm7.2 index publishes no 2.x wheel below 2.11.0, and an unknown newer rocm is not floored speculatively. Tests: query/fragment strip cases in the sh + ps1 marker suites and the Python strip/marker tests; the tri-state helper and the probe/baseline harnesses moved to the (marker, cutag, version) flavor with matching versions; new probe cases (untagged CUDA, generic-rocm-under-gfx) and 8 _ensure_pinned_known_family_torch tests; a four-way query-strip parity assertion. Full battery green (1150 python, sh 26/26 marker, ps1 marker/flavor/pin-stale). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: reinstall markerless gfx pins and cap custom-index updates at torch 2.11 Two follow-ups from the pin-marker audit: 1. A markerless venv with a gfx per-arch 2.11 pin trusted the wheel version tag, which is byte-identical (+rocm7.13.0) across gfx120X-all / gfx1151 / gfx1150. A pre-marker install holding one gfx arch's wheel that is now pinned to a DIFFERENT gfx index was therefore never switched: _rocm_pin_family_mismatch returns no-mismatch for any three-part +rocm 2.11 wheel, and _ensure_rocm_torch's absent-marker branch fell through to that heuristic. _ensure_rocm_torch now forces a one-time reinstall when the marker is absent AND the pin leaf is a 2.11 gfx per-arch index; the reinstall writes the marker, so the next update compares exactly and does not loop (the correctly-pinned no-reinstall guarantee then comes from the exact marker compare, not the ambiguous tag). Non-gfx-2.11 pins (rocmX.Y, non-2.11 gfx) stay on the tag heuristic -- their tags are distinguishable. 2. The verbatim custom-index update path used _CUDA_TORCH_PKG_SPEC (torch <2.12.0) while a FRESH install of the same unknown leaf caps torch at <2.11.0 (install.sh's default TORCH_CONSTRAINT, and setup.ps1's custom-pin branch), so a private /simple mirror publishing torch 2.11 could upgrade a `studio update` to a state the fresh installer never produces. Added _CUSTOM_INDEX_TORCH_PKG_SPEC (torch>=2.4,<2.11.0), used only by the verbatim path; companions stay pinned for the same exclusive --index-url ABI reason as _CUDA_TORCH_PKG_SPEC (a bare name could pull a torch-2.12-built torchvision). _CUDA_TORCH_PKG_SPEC is unchanged (known-family cu/cpu repair correctly tracks install.sh's widened cu ceiling). Tests: 2 new markerless-gfx cases (one-time reinstall + marker write + no-loop second run, and the rocmX.Y absent-marker no-op), the pre-existing markerless gfx no-reinstall test flipped to assert the one-time reinstall (it had encoded the old tag-trusting behavior), and the custom-index bound assertions. 488 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: a matching marker must not mask a broken, clobbered, or misclassified torch Four round-6 follow-ups, all closing cases where a matching torch-index marker wrongly vouched for a torch that is not actually the pinned one: 1. _is_cuda_family_leaf matched cu+digits by PREFIX (^cu[0-9]), so a custom mirror leaf like cu128-private classified as CUDA family; the flavor check then compared the installed cu128 tag to the whole leaf cu128-private and forced a reinstall on EVERY update (never converging). The cu family is now matched EXACTLY (re.fullmatch cu[0-9]+), so a cu-suffixed custom leaf routes through the verbatim/unknown path with a stable marker. Mirrored in install.sh (_normalize_family_leaf: strip cu, require an all-digit remainder) and setup.ps1 / install.ps1 (^cu[0-9]+$). 2. _torch_pin_needs_apply returned False on a failed torch probe (missing or unimportable) under a matching marker, so setup.sh kept the fast path and a broken torch was never repaired. A failed probe now forces the pass: the marker cannot vouch for a torch that does not import, forcing is idempotent, and once torch imports again the probe succeeds and the forcing stops (self-resolving). Reverses the round-4 conservative choice for this case. 3. _ensure_verbatim_torch_index snapshotted the installed trio on the first pass with a matching marker and treated an unimportable torch (snapshot None) as "no drift, skip", so a torch clobbered to a broken state before the run was masked. A None snapshot now reapplies the pin. A torch clobbered to a WORKING-but-wrong build under an unknown-family pin remains undetectable from metadata (no flavor tag; reinstalling every update would be the loop this avoids) and is documented as a known limitation. 4. The step-13 Windows final repair reran only the verbatim (unknown-family) and known-family cu*/cpu paths, so a clobbered explicit rocm/gfx pin (the wheel setup.ps1 installed from AMD's per-arch index) was left in place. The branch now also runs _ensure_rocm_torch on Windows for an explicit rocm/gfx pin; it has a Windows path and no-ops when torch already links HIP, so it only reinstalls a genuinely clobbered ROCm venv (loop-safe). Tests: the round-4 failed-probe-trusts-marker test flipped to force the pass; new cases for the cu-suffix no-loop, the broken-torch verbatim reinstall, and the Windows rocm final-repair structure; item-2 exact-cu parity assertions. 490 passed. sh/ps1 marker + flavor + pin-stale suites all green. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: repair Windows ROCm pins from the pinned URL and honor NO_TORCH Four round-7 review items, two of them regressions in the round-6 work: 1. _torch_pin_needs_apply ignored UNSLOTH_NO_TORCH. With a torch-index env var set and no marker, the failed-probe branch forced the dependency pass on every `studio update`, and the pass (which also honors NO_TORCH) never installs torch or writes a marker, so nothing could ever stop the forcing. It now returns False immediately under NO_TORCH: the pin only matters once torch is actually installed. 2. The step-13 Windows final repair (round-6) restored a clobbered explicit rocm/gfx pin by calling _ensure_rocm_torch, whose Windows path reinstalls from the arch AUTO-DETECTED via hipinfo, not from the pin. A user pinning a different gfx family or a private mirror was restored from the wrong source (and the wrong marker written), and a headless box was skipped entirely (the arch probe returns nothing). The repair now goes through _ensure_pinned_known_family_torch, which reinstalls from the PINNED url with the same per-arch floor setup.ps1 uses (2.11-line gfx leaves) or a bare trio (older arches, rocmN mirrors). It is gated on IS_WINDOWS since macOS ARM has no ROCm, and the existing flavor check keeps it loop-safe (a matching HIP wheel is left alone). 3. _ensure_verbatim_torch_index's broken-torch check (round-6) used "_installed_trio_snapshot() is None", but that helper reports a REMOVED torch as "torch==absent" (a non-None tuple) and a broken import as the stale on-disk version, so a missing or unimportable torch under a matching marker was read as "no drift" and skipped. The matching-marker path now confirms torch health with an import probe (_probe_torch_flavor): a torch that does not import reapplies the pin, while a healthy torch keeps the snapshot-based intra-run drift detection. 4. A unit test for _ensure_cpu_torch did not pin NO_TORCH False like its siblings, so a suite run with UNSLOTH_NO_TORCH=1 in the environment made the guard return early and the reinstall assertions fail spuriously. Tests: the round-6 broken-torch verbatim test re-encodes the non-None "torch==absent" snapshot case (the exact state the old "is None" check missed); new Windows-ROCm pinned-repair cases (reinstall from the pin, per-arch floor vs bare spec, matching-wheel no-op, off-Windows no-op); a NO_TORCH fast-path probe case; the parity test now asserts the Windows final branch does not auto-detect the ROCm index and that the helper reinstalls from the explicit pin. 494 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: floor the rocm7.2 index in the Windows pin repair; isolate marker tests Three round-8 review items, two of them downstream of the round-7 changes: 1. _ensure_pinned_known_family_torch gave a rocm<d> index leaf a bare torch/torchvision/torchaudio trio while flooring only gfx* leaves, so a Windows venv clobbered under an explicit rocm7.2 pin could reinstall an unbounded or ABI-mismatched trio from that exclusive --index-url. It now mirrors the spec the initial ROCm paths pin: the rocm7.2 floor for 2.11-line gfx leaves and rocm<d> leaves that serve torch 2.11, the <2.11 default for older rocm versions, and a bare trio only for older gfx per-arch leaves (which publish no floor), matching _ROCM_TORCH_PKG_SPECS / _ensure_rocm_torch. 2. test_verbatim_custom_url_no_marker_reinstalls_once called _ensure_verbatim_torch_index twice; the second call now hits the matching-marker health probe, and with pip_install mocked torch never becomes importable, so in a no-torch environment _probe_torch_flavor returned None and forced another reinstall, failing the idempotence assertion. The test now pins a healthy flavor so the idempotence check is about the marker, not ambient torch. 3. The TestEnsureRocmTorchMarker fixture patched os.environ per test but not _TORCH_BACKEND, which install_python_stack.py computes once at import from UNSLOTH_TORCH_BACKEND. A runner starting with a cuda/cpu backend made _ensure_rocm_torch early-return and skip the mocked repair these tests exercise. The fixture now neutralizes _TORCH_BACKEND so the marker tests are independent of the caller's installer-pin environment. Tests: the Windows floor-spec test now asserts a rocm7.2 mirror pin uses the rocm7.2 floor (not bare), plus a new rocm7.1 case that must fall back to the <2.11 default; the marker suite passes under a hostile UNSLOTH_TORCH_BACKEND=cuda / UNSLOTH_TORCH_INDEX_URL env. 495 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: apply same-flavor pin repoints, keep ROCm fallback nonfatal, bound custom companions Four round-9 review items, two of them regressions in the round-7 pin helper: 1. _ensure_pinned_known_family_torch returned as satisfied whenever the installed flavor matched the pin, so a same-flavor SOURCE change (one /cpu or /cu128 mirror to another, or a gfx1151 -> gfx120x-all per-arch switch, both carrying the same wheel tag) was never applied, while _torch_pin_needs_apply kept forcing the pass on the marker mismatch forever. It now also reinstalls when the marker records a DIFFERENT index of the same flavor, rewriting the marker so the next update matches (no loop), exactly as the Linux _ensure_{cuda,cpu}_torch helpers do. An absent marker on an already-matching venv is still left to the baseline recorder (no forced reinstall of a correct pre-marker venv). 2. That helper reinstalled a Windows ROCm pin with the FATAL pip_install, so when setup.ps1 had taken its CPU fallback (the pinned AMD index unavailable), the final repair re-hit the same missing index and aborted the whole install. The ROCm reinstall is now nonfatal (pip_install_try): on failure it leaves the CPU base in place and writes no ROCm marker, so the install completes -- matching _ensure_rocm_torch's Windows path. cu*/cpu pins stay fatal (authoritative source). 3. install.sh left torchvision/torchaudio bare for a pinned custom/unknown-leaf index (a private /simple mirror), unlike the Python update path's _CUSTOM_INDEX_TORCH_PKG_SPEC, so a mirror also exposing newer companion wheels could resolve a torch-2.12-built torchvision against the capped <2.11 torch. It now bounds the companions (torchvision>=0.19,<0.26.0 / torchaudio>=2.4,<2.11.0) for a custom leaf, gated on an empty _expected_torch_flavor_tag so known families keep their curated bare/floored companions. 4. install.sh's _expected_torch_flavor_tag matched cu[0-9]* by prefix, so a custom leaf like cu128-private classified as the cu128 family and force-reinstalled a correct +cu128 wheel on every run. It now requires exact cu+digits (routing the suffixed leaf to the custom path), matching the Python re.fullmatch(cu[0-9]+) and PowerShell, and feeding item 3's custom-leaf detection. Tests: new cases for the same-flavor marker-change reinstall, the nonfatal ROCm fallback (no marker on failure), the rocm7.2/older-rocm floor selection now split across the nonfatal path, cu-suffixed custom leaves in test_torch_flavor.sh, and the custom-leaf companion bounds in test_torch_constraint.sh. 497 python + 143 shell assertions pass; the marker suite still passes under a hostile UNSLOTH_TORCH_BACKEND=cuda env. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: bound custom-pin companions on the Windows setup path; isolate pin-probe tests Two round-10 review items: 1. setup.ps1's custom/unknown-leaf pin branch capped only torch ($cudaTorchSpec) and still asked the exclusive index for bare torchvision/torchaudio, so a private mirror that also serves newer companion wheels could install a torch<2.11 wheel alongside a torchvision>=0.26 / torchaudio>=2.11 built for a newer torch ABI, after which the marker records the pin as applied. It now bounds the whole trio (torch>=2.4,<2.11.0 / torchvision>=0.19,<0.26.0 / torchaudio>=2.4,<2.11.0) for a pinned non-cu-family leaf, matching install.sh, install.ps1's fresh pinned install, and install_python_stack.py's _CUSTOM_INDEX_TORCH_PKG_SPEC. This completes the companion-bounds fix across all three installers; known cu* leaves keep bare specs (the family index bounds them). 2. The _torch_pin_needs_apply probe tests did not pin NO_TORCH False, so a test process launched with UNSLOTH_NO_TORCH=1 short-circuited the probe (the round-7 guard) and returned False for cases that expect the pass to run. The _needs_apply helper now patches NO_TORCH (default False) around the call, and the dedicated no-torch case passes no_torch=True explicitly. Tests: the cross-platform parity test now asserts setup.ps1 bounds the full trio (not just torch) for a custom leaf; the pin-probe suite passes under a hostile UNSLOTH_NO_TORCH=1 environment. setup.ps1 parses clean; 497 python + shell suites green. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: bound custom rocm-* pins, redact diag tokens, snapshot custom pins before base update Three round-11 review items, all reproduced before fixing: 1. install.sh's custom-index companion bounds gated on _expected_torch_flavor_tag returning empty, but that helper returned "rocm" for ANY rocm* leaf, so a custom mirror whose leaf starts with rocm but is not a pip family (a private rocm-current mirror, a Radeon find-links rocm-rel-7.2.1) escaped the bounds and installed bare torchvision/torchaudio. It now digit-gates rocm to rocm[0-9]* (matching the Python _is_pip_rocm_family_leaf ^rocm\d), so those custom leaves return "" and the <2.11 companion caps apply; real rocm7.2 / gfx per-arch indexes still classify as rocm. 2. _tauri_torch_index_family classified by the raw last path segment, so a pinned URL carrying auth in the query (.../rocm7.2?token=SECRET) had the token echoed verbatim into the emitted [TAURI:DIAG] line. It now strips query/fragment before classifying (mirroring the marker/log credential stripping), so no token reaches the diagnostic output; as a side effect .../cu128?token=x now classifies as cu128 instead of auto. 3. On studio update, the core package step (a newer unsloth can require a torch the custom pin does not satisfy, pulling a default PyPI trio) runs BEFORE the step-2b verbatim check, which then recorded the already-clobbered trio as the baseline for a matching marker and left the pin unapplied. A new _capture_verbatim_baseline() records the pre-clobber trio before the core step, so the verbatim pass detects the drift and reapplies the pin. Captures only for a matching custom pin with importable torch; a mismatched/absent marker or broken torch is left to _ensure_verbatim_torch_index. Tests: _expected_torch_flavor_tag rocm-current / rocm-rel cases; _tauri_torch_index_family token/fragment redaction with a no-leak regression guard; _capture_verbatim_baseline record/skip cases plus an end-to-end clobber-detection scenario; a structural guard that the capture runs before the core step. 501 python + shell suites pass; install.sh bash -n clean, shellcheck unchanged from base. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: match rocm family leaves exactly, enforce the rocm7.2 torch line, repair a broken pinned torch A pinned index is a pip ROCm --index-url family only when its leaf is an exact rocm<digits> / rocm<digits>.<digits> (rocm7.2) or a gfx* per-arch leaf. The prior ^rocm[0-9] prefix match also caught suffixed private-mirror leaves (rocm7.2-private, rocm7-current), routing them through the ROCm/companion-family path instead of the verbatim pin: the companion bounds were skipped and, on a pre-marker venv with a compatible +rocm wheel, the pin was never applied. Match the family exactly through one shared helper at every site: - install_python_stack.py: _is_pip_rocm_family_leaf (re.fullmatch), plus the two other loose gates it feeds (_normalize_family_leaf, _torch_flavor_matches_pin). - install.sh: a new _is_pip_rocm_family_leaf routes _expected_torch_flavor_tag, _torch_index_repairable, _normalize_family_leaf and the ROCm side-effect gate. - setup.ps1: a new Test-PipRocmFamilyLeaf routes Get-NormalizedFamilyLeaf and both pinned reroutes; install.ps1 anchors its reroute regex. _rocm_pin_family_mismatch (and its setup.ps1 mirror Get-RocmPinStaleTags) compared only the ROCm version, so a +rocm7.2 wheel whose torch release drifted off the 2.11 line (2.12/2.13 from an out-of-band upgrade or a custom rocm7.2 mirror) satisfied the family check while violating _ROCM_TORCH_PKG_SPECS['rocm7.2'] (torch>=2.11,<2.12). Flag it stale so the repair reinstalls to floor; >=2.11 alone is not enough, so the release is compared exactly against the 2.11 line for a KNOWN-2.11 rocm pin. _ensure_pinned_known_family_torch returned on a failed import probe, but _torch_pin_needs_apply forces the dependency pass on that same failed probe: a broken torch under a known-family pin was left in place and the pass was forced on every update. Treat an unimportable torch as drift and reinstall the pinned trio (the spec and marker derive from the pinned leaf, not the absent flavor); once it lands the probe succeeds and the fast path returns. Tests: exact-match cases across test_torch_flavor.sh, test_rocm_support.py, test_cross_platform_parity.py and the two .ps1 helper suites; the rocm7.2 release-line and broken-probe-reinstall cases; extraction lists updated for the new helpers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: anchor the PS pinned-ROCm floor gate and bound install.ps1 custom-pin companions Round 12 made every family CLASSIFIER exact, but the Windows install-flow floor gate reads $_pinRocm211 directly from the raw pinned leaf with an unanchored -match '^rocm(\d+)\.(\d+)' BEFORE any exact classification runs. A suffixed custom leaf (rocm7.2-private) matches that rocm7.2 prefix, so it takes the 2.11-floor branch and is force-routed through the ROCm install path before the exact-match elseif can send it to the verbatim install. Anchor the match ($) in both install.ps1 and setup.ps1 so only an exact rocmX.Y leaf is floored; a suffixed or newer-suffix leaf falls through to the verbatim path. The Python floor selection is already exact (dict lookups gated on _is_pip_rocm_family_leaf), so only the two PS scripts needed this. install.ps1's custom (non-cu-family) pinned-torch install bounded torch>=2.4,<2.11.0 but left torchvision/torchaudio bare, so a private mirror serving newer companions could pull a wheel built for a newer torch ABI while the marker records the pin as applied. Bound both companions (torchvision>=0.19,<0.26.0 / torchaudio>=2.4,<2.11.0) when the leaf is not a cu<digits> family index (a cu index bounds its own resolution), matching setup.ps1's Test-CudaFamilyLeaf gate and _CUSTOM_INDEX_TORCH_PKG_SPEC. Tests: parity guards for the anchored floor gate in both PS scripts and for install.ps1's bounded custom-pin companions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: tighten comments in the torch-index-override paths Collapse the verbose comment and docstring blocks added across the installer scripts and their tests to fewer, clearer lines without changing behaviour. Remove a duplicated CUDA-spec comment block. Comments/docstrings only; no code changes (AST-verified). * install: repair a broken pinned torch on Linux, strip trailing slash in tauri family, count the final step _ensure_cuda_torch / _ensure_cpu_torch returned on a failed import probe (torch present but unimportable). With an explicit CUDA/CPU pin, _torch_pin_needs_apply forces the dependency pass on that same failed probe, and the base package update does not force-reinstall an already-installed torch distribution, so the broken torch was left in place and the pass reran every update without repairing it. Treat a failed probe under a pin as drift and reinstall from the pinned index (the reinstall rewrites the marker and the next probe imports, so no loop). This is the Linux counterpart of the known-family repair fix. _tauri_torch_index_family stripped the query/fragment before classifying but not a trailing slash, so a token-authenticated pin like .../cu128/?token=x collapsed to .../cu128/ and fell through the exact-suffix */cu128 and */cpu arms to "auto". Strip a trailing slash too, mirroring _torch_index_url_leaf. The Windows / macOS-ARM final torch-repair step (_ensure_pinned_known_family_torch) runs a progress step that base_total never counted (the final-step increment was gated to Linux), so _STEP ran one past _TOTAL on those platforms. Add the missing increment. Tests: broken-probe reinstall for the CUDA (family and URL pins) and CPU paths; trailing slash / slash+token cases for _tauri_torch_index_family; a full-flow progress-count guard asserting _STEP == _TOTAL on Windows and Linux. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: tighten comments in the torch-index-override paths * install: harden the torch-index pin across all four installers Redact index-URL credentials from captured install logs before they print on failure. uv/pip failure text embeds the failing --index-url verbatim, so a user:token@ or ?token= secret could leak into the console. Add a shared redaction pass (_redact_install_output / Redact-InstallOutput) wired into the error-output dump in install.sh, install.ps1, setup.ps1 and install_python_stack.py. Verbose mode still streams live uncaptured output, so it is intentionally left unredacted (developer opt-in). Trim trailing slashes on the PATH only for a verbatim UNSLOTH_TORCH_INDEX_URL override, preserving a ?query/#fragment token. A whole-URL rstrip corrupted a base64 token ending in "/", and a single-slash strip left .../cu128// classifying as an empty leaf. Add _trim_index_path_slashes / Trim-IndexPathSlashes and route the override through it; strip ALL trailing slashes in the backend-branding leaf classifier so a double slash still yields the real leaf. Reject a trailing-dot ROCm leaf (rocm7.) in the bash family validator so it matches Python re.fullmatch(rocm\d+(?:\.\d+)?) and the PowerShell regex: both the major and the minor must be non-empty digits, so rocm7. is a custom verbatim pin, not a pip ROCm family. Scrub PIP_NO_INDEX and PIP_INDEX_URL for a pinned install in the two installers that have a plain-pip fallback (install_python_stack.py, setup.ps1): PIP_NO_INDEX=1 makes the fallback ignore every index including the pinned --index-url, and PIP_INDEX_URL replaces it. install.sh and install.ps1 install via uv --default-index (which ignores pip config/env), so they are unaffected. Add unit tests (bash, Python, PowerShell) and cross-platform parity tests covering credential redaction, path-only slash trimming, the rocm7. validator, the double-slash leaf, and the PIP_NO_INDEX/PIP_INDEX_URL scrub. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: redact captured torch-install output and warn on a failed pinned ROCm repair Close a redaction gap the earlier pass missed: setup.ps1's direct `Fast-Install ... | Out-String` branches (ROCm from $ROCmIndexUrl, CPU/CUDA from $TorchInstallIndexUrl, plus the Triton and T5 sub-venv installs) printed the captured $output verbatim on failure, bypassing Redact-InstallOutput. A private index carrying userinfo or a ?token= in the pin could leak into Windows Studio setup logs. Route every `Write-Host $output` through Redact-InstallOutput. Warn on a failed pinned Windows ROCm reinstall in _ensure_pinned_known_family_torch: the branch printed "reinstalling from it" then called pip_install_try, but had no else, so a failure continued silently and left the user believing the pin was applied while the old CPU/wrong torch survived. Mirror the auto-ROCm Windows path and warn, telling the user to retry. * install: redact captured output on the pip fallback and optional-install failure paths The uv install path already redacted its captured output, but pip_install's pip fallback runs through run(), which printed result.stdout verbatim on failure, and _print_optional_install_failure did the same. A pinned --index-url carrying userinfo or a ?token= could still leak there when uv is unavailable or the pip fallback also fails. Route both through _redact_install_output. The verbose pip_install_try path stays raw (developer opt-in), matching the other installers. * install: split the survive-updates marker subsystem into a follow-up The torch-index override PR grew a persisted per-venv marker plus repair machinery (stale-pin detection, verbatim re-apply, update-time reinstall triggers) that roughly doubled it. That subsystem is orthogonal to the core feature and is being reworked in a follow-up (versioned/hashed marker, full-URL pin baseline), so it moves there wholesale instead of shipping twice. What this PR still does: UNSLOTH_TORCH_INDEX_URL / UNSLOTH_TORCH_INDEX_FAMILY pick the torch wheel index at install time in all four installers, with the exact rocm/gfx/cpu/cu leaf classification, the torch 2.11 floor for the per-arch AMD indexes, bounded companions for custom leaves, credential redaction of captured installer output, path-only slash trimming, and the uv/pip index env scrubs. Flavor-based repair keeps honoring the pin: a wrong family under an explicit pin still reinstalls from the pinned URL, and setup.ps1 repairs a pinned stale venv in place instead of wiping it. What moves to the follow-up: the .unsloth-torch-index marker file and its writers/readers/normalizers, exact-URL pin-change detection on update (same-tag gfx switches, custom-mirror repoints), the verbatim trio snapshot and clobber re-apply, the pin-baseline recorder, and the --torch-pin-needs-apply fast-path probe in setup.sh / setup.ps1. Their tests (the marker sh/ps1 suites, the stale-pin suite, and the marker classes in the rocm/cuda/parity suites) move with them; the removed code is preserved on a local archive branch to seed that PR. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: re-apply a ROCm pin over an existing HIP wheel via the version tag The subsystem split left an explicit ROCm/gfx pin unenforced on `studio update` whenever the venv already imported ANY ROCm torch: the pinned reinstall lived inside the `elif not has_hip_torch` branch, so a rocm6.4 to rocm7.2 switch, a gfx1151 pin over a generic +rocm7.2 wheel, or a broken 2.12+rocm7.2 drift never re-applied the pin. Restore the markerless half of that detection: _rocm_pin_family_mismatch compares the pinned leaf against the installed wheel tag (exact rocmX.Y compare, the 2.11 gfx per-arch allowlist, the untagged-wheel rule), the HIP probe emits "<hip_marker>|<version>" again so the installed tag is available, and _ensure_rocm_torch reinstalls from the pinned URL when the tag mismatches even though HIP torch is present. setup.ps1 mirrors it: the stale-venv check routes a pinned rocm/gfx leaf through Get-RocmPinStaleTags instead of collapsing it to a generic "rocm" flavor, and the existing pinned in-place repair (no wipe) applies the change. What still waits for the follow-up marker PR, by design: pin changes the wheel tag cannot see -- a per-arch switch between two 2.11 gfx indexes (identical +rocm7.13.0 tag), a custom-mirror URL repoint under the same family leaf, and unknown-family verbatim pins. Those need the persisted index record. Tests restored with the code: the _rocm_pin_family_mismatch table, the five update-path cases (older-rocm reinstall, gfx-over-pre-2.11 reinstall, matching-pin no-reinstall, non-2.11 gfx no-reinstall, gfx-over-generic-2.11 reinstall), the "|" probe-format guards, and the AST-extracted Get-RocmPinStaleTags suite for setup.ps1. * install: compare major-only rocm pins, redact URL fragments, bound pinned CPU trio Three review fixes on the restored pin-repair path. The family classifier accepts a major-only rocm<d> leaf (rocm7), but the mismatch comparators only parsed rocmX.Y, so a rocm7 pin fell through to the 2.11-line fallback and INVERTED both verdicts: an installed +rocm6.4 wheel compared as satisfied (pin never re-applied) while a matching +rocm7.2 wheel compared as stale (reinstall loop). Major-only pins now compare on the major alone in _rocm_pin_family_mismatch and Get-RocmPinStaleTags: rocm6.x under a rocm7 pin is a mismatch, any rocm7.x satisfies it, an untagged wheel never does, and a bare +rocm tag with an unreadable version is accepted (matching the existing lenient unreadable fallback). The output redactors scrubbed userinfo and ?query= values but not #fragments, so a pin like https://mirror/whl/cu128#token=secret leaked the secret in captured uv/pip failure text -- inconsistent with the URL handling itself, which already treats fragments as sensitive. All four redactors gain a URL-anchored fragment rule (anchored so a bare "# comment" line in tool output is never touched). setup.ps1's CPU branch installed a bare torch/torchvision/torchaudio trio; fine for the unpinned host default, but a PINNED cpu index routes through the same branch and the /cpu index serves newer torch, so a fresh pinned CPU install could land an unsupported trio that _ensure_cpu_torch then keeps (it accepts any CPU build). Under a pin the branch now installs the bounded trio mirroring _CPU_TORCH_PKG_SPEC (torch>=2.4,<2.12.0 and matching companions); the unpinned path is unchanged. Tests: major-only rows in the Python mismatch table and the AST-extracted setup.ps1 suite; fragment + query-plus-fragment + bare-hash-comment cases in all four redactor suites; a parity check that the pinned CPU trio bounds exist, are gated on the pin, and mirror the Python repair spec. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: tighten comments in the torch index override paths * tests: track the moved pass-through inheritance in the gguf order check Main moved the llama_extra_args pass-through inheritance out of the GGUF branch into _resolve_inherited_extra_args, which runs before it, so the source-order assertion's "if request.llama_extra_args is None" anchor no longer exists inside the branch and the check failed after the main merge. The test now asserts the same property in the current shape: inheritance before the GGUF branch (a carried --no-mmproj still shapes the hub guard's companion requirement), and marker, hub guard, unload in order within the branch. Full file passes (32 tests). * tests: anchor the inheritance order check on the call, not the definition source.index("_resolve_inherited_extra_args(") matched the function definition, which always precedes the endpoint, so the ordering assertion was vacuously true. Anchoring on "= _resolve_inherited_ extra_args(" pins the first call site inside the load endpoint (line 4505), which is the statement whose position relative to the GGUF branch the test is meant to guard. 32 tests pass. * tests: align the gguf order test with main Main fixed the stale ordering assertion in PR 7252; adopting its version verbatim removes this file from the branch diff entirely and avoids a conflict on the next main merge. 32 tests pass. * install: bound the companion constraints to torch's window everywhere A full platform x vendor validation matrix over this branch surfaced a real trio mismatch on the cpu/mac paths: torch is capped <2.11 (installs 2.10.0+cpu) but the bare torchaudio companion resolves 2.11.0+cpu, because torchaudio 2.11 dropped its exact torch pin. Reproduced in a sandboxed end to end cpu install. torchvision still exact-pins torch and self-corrected. The default companion constraints are now bounded to torch's window (<0.26 / <2.11) and widen together with the cu* torch window (<0.27 / <2.12), so every leaf resolves a paired trio. Verified with uv dry-runs on the cpu, cu130, and rocm6.4 leaves (2.10.0/0.25.0/2.10.0, 2.11.0/0.26.0/2.11.0, 2.9.1/0.24.1/2.9.1) and a rerun of the sandboxed cpu install, which now lands torch 2.10.0+cpu with torchaudio 2.10.0+cpu. The Strix WSL reroute now also forwards UNSLOTH_TORCH_INDEX_URL and UNSLOTH_TORCH_INDEX_FAMILY into the rerouted 24.04 distro; dropping them silently reverted the child install to auto-detection, defeating the pin this branch introduces. test_torch_constraint.sh updated: the bounded companions must appear at the defaults and the custom-leaf block, no bare companion may remain, and the cu* widen must carry the companions with it. * install: harden the override path against reroute drift and credential leaks Review sweep focused on default-path idempotency found no defects on the unset path; these fixes cover the override path and failure reporting. install.sh: - The early WSL Strix Halo distro reroute now honors an explicit index pin (UNSLOTH_TORCH_INDEX_URL / _FAMILY): the pin is used in the current distro instead of probing the GPU and re-entering another distribution, matching the contract of the later Radeon and Strix guards. Whitespace only values do not gate, in parity with get_torch_index_url. - Verbose mode now streams installer output through the credential redactor; it previously bypassed the redaction the quiet path applies. The exit code survives the pipe via an rc file since the script runs under plain sh with no pipefail. - The kept-release fallback warning now strips credentials from the index URL before printing it. install.ps1: - Bounded torchvision and torchaudio next to every capped torch install (custom pin, ROCm CPU fallback, CUDA flavor repair). torchaudio 2.11 dropped its exact torch pin from the wheel metadata, so a bare companion beside torch<2.11 can resolve a mismatched 2.11.0 build, cu family indexes included. Mirrors the install.sh companion bounds. studio/install_python_stack.py: - The verbose failure path now redacts index URLs in pip and uv output before printing, matching every other output site in the file. All sh, ps1 and python installer test suites pass (the host-defaults suite has a known pre-existing failure unrelated to this change). * install: redact verbose Windows installer output and repair the parity tests Follow-ups to the override-hardening commit, from review: - install.ps1 Invoke-InstallCommand and setup.ps1 Invoke-SetupCommand now pipe verbose output through Redact-InstallOutput per record, and the three verbose Fast-Install torch call sites (ROCm, CPU, CUDA) do the same: uv and pip echo the pinned index URL, credentials included, in their errors, and verbose mode previously bypassed the redaction the quiet paths apply. ForEach-Object and Out-Host leave $LASTEXITCODE untouched, verified with a native command exiting 7 behind the pipe. - test_cross_platform_parity.py: the install.ps1 companion-bounds assertion now matches the implemented behavior (bounds on every index, no cu-family exemption, since torchaudio 2.11 dropped its exact torch pin) instead of requiring the removed $_pinCuLeaf gate. - test_rocm_support.py: the WSL reroute guard test slices the whole function body to its closing brace instead of a fixed 1200-character window, which the new pin-gate preamble had outgrown. 428 tests pass across the parity, install stack and rocm support suites; the sh and ps1 installer suites pass unchanged. * install: tighten comments in the torch-index and ROCm/CUDA repair paths * install: digit-gate the gfx family leaf and honor ROCm pins in the Windows repair Two review follow-ups on the override path: - The pip ROCm family predicate accepted ANY gfx-prefixed leaf, so a custom verbatim pin like /gfx-private classified as a ROCm family and enabled the ROCm-only side effects (AMD bitsandbytes, ROCm torch repair) on a mirror that may serve CPU/CUDA wheels. gfx now requires a following digit (gfx90a, gfx1151, gfx120X-all), consistently in install.sh, install_python_stack.py, install.ps1 (family gate and expected-flavor classifier) and setup.ps1, matching the strictness the rocm side already had (rocm7.2-private stays verbatim). The broader backend BRANDING globs are unchanged on purpose: radeon repo leaves (rocm-rel-X.Y) must still brand the rocm backend without being force-repaired as a family. - The Windows branch of the ROCm torch repair always installed from the public per-arch index, ignoring an explicit ROCm-family pin: after a pinned setup.ps1 install failed to a CPU base, the repair retried repo.amd.com instead of the pinned index. The branch now resolves _explicit_rocm_torch_index_url() first, uses it as the install index when set, and mirrors the Linux pin contract by skipping the NVIDIA and gfx-detection gates a pin is documented to override. Source-assertion tests updated to the tightened predicate and the new repair label. 1165 tests pass across the parity, install stack and studio install suites; the sh and ps1 suites pass; both PowerShell installers parse clean. * Remove scratch archives accidentally committed with the comment pass The temp/ archive copies of installer and test files were working scratch, not PR content, and inflated the diff by about nine thousand lines. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-20 00:58:52 -07:00
Saicharan Ramineni	9d53656614	Make _uv_safe_path space-safe on macOS/Linux (#6503) (#6534) ... * Copy uv `-c`/`-r` paths to a space-free temp dir on macOS/Linux uv 0.11.x truncates a constraints/requirements path passed via `-c`/`-r` at the first space, so `unsloth studio` setup from a repo cloned under a path containing a space (e.g. `/Users/me/Open Source/unsloth`) fails with: error: File not found: `/Users/me/Open` _uv_safe_path() already worked around this on Windows via the 8.3 short path but returned the space-containing path unchanged on macOS/Linux, which have no 8.3 equivalent. Extend it to copy the (small, flat) constraints/requirements file into a space-free temp dir and hand uv the copy; the temp dirs are removed at process exit. Falls back to the original path on any error, so it is never worse than before. Refs unslothai/unsloth#6503 * Route UV_OVERRIDE through _uv_safe_path and fix temp-dir leak (#6503) The -c/-r fix did not cover UV_OVERRIDE, which uv also truncates at the first space. On Apple Silicon the overrides file is handed to uv via UV_OVERRIDE at install time (install_python_stack.py) and during the MLX self-heal (utils.mlx_repair), so a repo under a path containing a space still broke every uv call there. Move _uv_safe_path into backend.utils.uv_path_safety so both sites share it, and route UV_OVERRIDE through it. Also stop leaking the temp dir when shutil.copyfile fails after mkdtemp, and add tests for the UV_OVERRIDE channel, the TMPDIR-with-space fallback, the atexit cleanup, and the no-leak path. --------- Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-06-24 04:02:24 -07:00
Daniel Han	a6dc10dad2	Reduce and tighten comments and docstrings across the test suite (#6429) ... * Reduce and tighten comments and docstrings in tests Shorten verbose comments and docstrings across the test suite without changing any test logic. Remove narration that restates the next line, collapse long module and test docstrings to a single line, and drop banner separators. Keep regression context (issue and PR references, run ids), skip reasons, mocking and timing rationale, license headers, lint and type directives, and commented-out code. Comments and docstrings only: an AST signature check confirms no code, assertions, or string literals changed, and the suite byte-compiles cleanly. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-18 01:07:09 -07:00
Daniel Han	187144d4e7	Reduce and tighten code comments and docstrings repo-wide (#6095) ... Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.	2026-06-08 23:09:51 -07:00
Roland Tannous	19e9c60a8e	Consolidate dual venvs and separate install from update (#4530) ... * refactor: consolidate dual venvs into single ~/.unsloth/studio/unsloth_studio * refactor: separate install.sh (first-time) from setup.sh (smart update with PyPI version check) * fix: install.sh calls setup.sh directly, keep both setup and update CLI commands * fix: use importlib.resources.files() directly without _path attribute * fix: bootstrap uv before pip upgrade to handle uv venvs without pip * fix: frontend 404 when launched via CLI, add global symlink to ~/.local/bin * feat: add --local flag to install.sh and unsloth studio update for branch testing * fix: resolve repo root from script location for --local installs * feat: add --package flag to install.sh for testing with custom package names * feat: add --package flag to unsloth studio update * fix: always nuke venv in install.sh for clean installs * revert: remove Windows changes, will handle in separate PR * fix: error when --package is passed without an argument * revert: restore Windows scripts to current main * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: always explicitly set STUDIO_LOCAL_INSTALL and STUDIO_PACKAGE_NAME env vars * fix: pass explicit STUDIO_LOCAL_REPO env var for --local installs * fix: align banner box for Setup vs Update labels * deprecate: hide 'unsloth studio setup' command, point users to update/install.sh * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: check stdout not stdin for auto-launch detection (curl pipe fix) * fix: update install URL to unsloth.ai/install.sh * fix: update install.sh usage comments to unsloth.ai/install.sh * fix: use --upgrade-package for base deps to preserve existing torch/CUDA installs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: --local install now also installs unsloth-zoo via base.txt before editable overlay * fix: don't skip base packages for --local installs (editable needs unsloth-zoo) * refactor: move --local full dep install to install.sh, keep SKIP_STUDIO_BASE for all paths * feat: add migration support for old .venv and CWD-based installs in setup.sh * Revert "feat: add migration support for old .venv and CWD-based installs in setup.sh" This reverts commit 301291d002. * feat: migrate old .venv layout in install.sh instead of always nuking * feat: validate old .venv with torch CUDA test before migration, recovery message on launch failure * fix: try CUDA then fall back to CPU for migration validation * fix: upgrade unsloth/unsloth-zoo with --reinstall-package on migration to preserve torch * remove: delete unused unsloth ui command (use unsloth studio instead) * Fix Windows venv path mismatch between install.ps1, setup.ps1, and studio.py install.ps1 was creating the venv CWD-relative ($VenvName = "unsloth_studio"), setup.ps1 was using an absolute path to ".unsloth\studio\.venv", and studio.py looks for ".unsloth\studio\unsloth_studio". All three paths were different, so the Windows installer would never produce a working Studio setup. install.ps1: - Use absolute $StudioHome + $VenvDir matching the Linux install.sh layout - Add 3-way migration: old .venv at STUDIO_HOME, CWD-relative ~/unsloth_studio from the previous install.ps1, or fresh creation with torch validation - For migrated envs, upgrade unsloth while preserving existing torch/CUDA wheels - Set SKIP_STUDIO_BASE=1 before calling setup.ps1 (matches install.sh behavior) - Fix launch instructions to use the absolute venv path setup.ps1: - Change $VenvDir from ".unsloth\studio\.venv" to ".unsloth\studio\unsloth_studio" - Add SKIP_STUDIO_BASE guard: error out if venv is missing when called from install.ps1 (which should have already created it) - Differentiate "Setup" vs "Update" in banners based on SKIP_STUDIO_BASE * setup.ps1: unconditionally error if venv missing, matching setup.sh setup.sh always errors out if the venv does not exist (line 224-228), telling the user to run install.sh first. setup.ps1 was conditionally creating a bare venv with python -m venv when SKIP_STUDIO_BASE was not set, which would produce an empty venv with no torch or unsloth. Now setup.ps1 matches setup.sh: always error, always point to install.ps1. * Fix --torch-backend=auto CPU solver dead-end on Linux, macOS, and Windows On CPU-only machines, `uv pip install unsloth --torch-backend=auto` falls back to unsloth==2024.8 because the CPU solver cannot satisfy newer unsloth's dependencies. install.ps1 already solved this with a two-step approach; this applies the same fix to install.sh and install_python_stack.py. install.sh: add get_torch_index_url() that detects GPU via nvidia-smi and maps CUDA versions to PyTorch index URLs (matching install.ps1's Get-TorchIndexUrl). Fresh installs now install torch first via explicit --index-url, then install unsloth with --upgrade-package to preserve the pre-installed torch. All 5 --torch-backend=auto removed from primary paths. install.ps1: add fallback else-branch when TorchIndexUrl is empty, using --torch-backend=auto as last resort (matching install.sh). install_python_stack.py: remove unconditional --torch-backend=auto from _build_uv_cmd. Torch is pre-installed by install.sh/setup.ps1 by the time this runs. Callers that need it can set UV_TORCH_BACKEND. Both install.sh and install.ps1 now share the same three-branch logic: migrated env (upgrade-package only), normal (torch-first + index-url), and fallback (--torch-backend=auto if URL detection fails). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Use --reinstall-package for migrated envs on both Linux and Windows For migrated environments (moved from legacy venv location), --reinstall-package is better than --upgrade-package because it forces a clean reinstall even if the same version is already installed. This ensures proper .dist-info and .pyc state in the new venv location. --upgrade-package remains correct for the fresh install path where torch is already installed and we just want to add unsloth without re-resolving torch. * Address review findings: portability, parity, and stale comments - Replace grep -oP (GNU Perl regex) with POSIX sed in get_torch_index_url() so the script works on BSD grep (macOS is already guarded by the Darwin early-return, but Alpine/BusyBox would silently get the wrong CUDA tag) - Add LC_ALL=C before nvidia-smi invocation to prevent locale-dependent output parsing issues - Add warning on stderr when nvidia-smi output is unparseable, matching install.ps1's [WARN] message - Add explicit unsloth-zoo positional arg to install.ps1 migrated path, matching install.sh (--reinstall-package alone won't install it if it was never present in the migrated env) - Fix stale comment in install_python_stack.py line 392 that still claimed --torch-backend=auto is added by _build_uv_cmd - Add sed to test tools directory (function now uses sed instead of grep) * Add --index-url to migrated env path to prevent CPU torch resolution The migrated path runs uv pip install with --reinstall-package for unsloth/unsloth-zoo. While uv should keep existing torch as satisfied, the resolver could still re-resolve torch as a transitive dependency. Without --index-url pointing at the correct CUDA wheel index, the resolver would fall back to plain PyPI and potentially pull CPU-only torch. Adding --index-url $TORCH_INDEX_URL ensures CUDA wheels are available if the resolver needs them. Applied to both install.sh and install.ps1. * Revert --index-url on migrated env path The original install.ps1 on main already handles the migrated path without --index-url and it works correctly. --reinstall-package only forces reinstall of the named packages while uv keeps existing torch as satisfied. No need for the extra flag. * Fix unsloth studio update --local not installing local checkout studio.py sets STUDIO_LOCAL_REPO when --local is passed, but install_python_stack.py never read it. The update path always installed from PyPI regardless of the --local flag. Add a local_repo branch that first updates deps from base.txt (with --upgrade-package to preserve torch), then overlays the local checkout as an editable install with --no-deps. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-03-25 05:24:21 -07:00