mirror of
https://github.com/unslothai/unsloth.git
synced 2026-10-27 17:18:23 +00:00
Windows on ARM: install the native ARM64 CUDA stack on NVIDIA hosts (#10282)
* Windows on ARM: install the native ARM64 CUDA stack on NVIDIA hosts NVIDIA's Windows-on-ARM parts (GB10 / N1X, the RTX Spark laptops) run CUDA natively, but the installer treated every ARM64 Windows host as "no GPU wheels here" and fell back to x64 Python under emulation. That fallback looks like it works: torch loads and sees the GPU. It then fails on the first kernel, because triton-windows' bundled ptxas is built for the x64 CUDA 12.8 toolkit and cannot target the sm_121 these parts report, so every Unsloth kernel fails to compile. llama.cpp separately picked a CPU-only bundle regardless of the GPU. Detection is evidence based rather than arch based: the installer asks the wheel indexes what actually exists for this interpreter, and only takes the native path when both a win_arm64 CUDA torch and an obtainable pyarrow are found. Anything else keeps the historical x64 path, so a host that cannot be served natively is never left with a half configured venv. The torch index is probed in order: download.pytorch.org first, so this special case ends by itself the day PyTorch publishes these wheels, then NVIDIA's GA channel, then its nightly. Requirement pins with no win_arm64 wheel are split on platform_machine == "ARM64" rather than loosened. Every other platform (linux x86_64 and aarch64, win x64, macOS arm64 and x86_64) evaluates to the pins it had before, checked by evaluating the markers per platform as well as by the test suite. Also: * llama.cpp prebuilt selection learns a windows-arm64-cuda install kind and falls through to upstream's win-cuda-13.4-arm64 asset, so these hosts get a CUDA bundle instead of the CPU one. When neither is hash gated the fallback is still taken, but says so in the log rather than silently. * setup.ps1 keeps an installed cu134 torch on win_arm64 instead of "repairing" it to a cu130 that has no wheel for this platform, and asks the chosen index whether it publishes torchaudio instead of assuming it does not. * install.ps1 checks before the studio handoff that the installed package understands this host, and stops with an explanation if it does not. The two halves ship through different channels (the script is fetched from unsloth.ai, the package comes from PyPI), and an older setup.ps1 handed a working +cu134 torch will reinstall a cu130 that does not exist here, leaving the environment with no torch at all. UNSLOTH_WOA_NATIVE=0 opts back out to the x64 path. Verified on an RTX Spark N1X laptop: clean install into an empty UNSLOTH_STUDIO_HOME ends EXIT=0 with torch 2.14.0+cu134, torchvision 0.29.0, torchaudio 2.11.0, triton-windows 3.8.0.post28 and the llama-b10715-bin-win-cuda-13.4-arm64 bundle; LoRA and QLoRA both train; Studio serves and reports CUDA; Qwen3.8-27B-GGUF UD-Q4_K_XL answers at 8.1 tok/s with the GPU at 94-95 percent through generation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: key the new filters on the interpreter, not the machine Review of the branch turned up one real regression and several sharp edges. All of them are on paths this PR added; none change what a host did before it. The regression. `_is_windows_arm64()` answers for the MACHINE, and says yes even under an emulated x64 Python -- its docstring says so, since that is what it is for. But the requirement markers this PR adds key on `platform_machine == "ARM64"`, which is the INTERPRETER's arch, and every Windows-on-ARM install predating native support is running an emulated x64 interpreter, because install.ps1 deliberately fetched one there. So on exactly those existing installs the two mechanisms disagreed: the marker rows correctly selected the x64 pins while the new skip list dropped ten packages (xformers, tiktoken, tensorboard, librosa, openai-whisper, hf_transfer, sqlite-vec, mecab and the rest) that have working win_amd64 wheels and install fine today. Anyone taking the new UNSLOTH_WOA_NATIVE=0 escape hatch would have hit it too. `_is_win_arm64_interpreter()` answers the other question, from `sysconfig.get_platform()` -- the same authority pip and uv tag wheels with -- and the two call sites this PR added now use it. The pre-existing uses of the machine predicate are left alone, so nothing outside this PR shifts. The rest: * setup.ps1 answers `Test-WinArm64Venv` from the machine arch before launching any interpreter. The callers sit in the stale-venv path, which on x64 previously executed nothing at all; probing a venv's python there to learn something already known gave a corrupt or lock-blocked interpreter a chance to hang the install. * `$expectedKinds` gains windows-arm64-cuda only on the arch that can carry it. Accepting it on x64 would keep an install that machine cannot run, where today it is removed as mismatched. * The wheelhouse reuses a staged wheel only if it opens as a zip. An interrupted run leaves a truncated file, and skipping on mere existence handed the resolver that same corpse on every retry. Both downloads also take a timeout now. * UV_OVERRIDE / UV_FIND_LINKS / PIP_FIND_LINKS are cleared when they point into this StudioHome's own woa directory, so a second `irm | iex` in one shell cannot inherit a previous run's wheelhouse. Matched by path rather than cleared outright: an inherited UV_FIND_LINKS that a user set for their own wheelhouse is left alone, since clearing it would break air-gapped installs on every Windows host. * hf_transfer in no-torch-runtime.txt and cryptography in studio.txt (and its mirror in pyproject) get the same marker split as the other rows. That file installs --no-deps and the extra is installable directly, so neither was covered by the constraints file. * `windows_cuda_arch_for_host` tests is_windows as well as is_arm64. Every caller is already inside a Windows branch, but `is_arm64` is also true on macOS arm64 and Linux aarch64, so the answer should not depend on the caller remembering that. * The published legacy-minor scan uses the arch being resolved instead of a hardcoded x64, which was harmless only while no arm64 CUDA artifact existed. * No `windows_cuda_selection` log line on x64, where the arch is the unspoken default and the line would be new output in every existing install log. Verified: requirement markers resolve identically to origin/main across 65 environments (linux x86_64/aarch64/arm64/armv7l/ppc64le/s390x, win32 AMD64/x86/ARM64EC, darwin arm64/x86_64, freebsd, python 3.10 to 3.14), differing only on win32/ARM64; the studio-extra mirroring guard passes; both PowerShell files parse; and a simulated interpreter matrix confirms the win_arm64 skip list applies on a native ARM64 interpreter and on nothing else, including the emulated x64 case that regressed. * Keep the cross-platform parity greps matching Two parity tests read install.ps1 and setup.ps1 as text, and both changes in this PR moved the strings they look for. The behaviour they guard is intact in each case, so the fix is on this side rather than in the assertions. test_install_ps1_bounds_unknown_leaf_pinned_torch pins the literal "@_torchSpecs --default-index $TorchIndexUrl", which is how it checks that the bounded companion specs actually reach uv. The Windows on ARM branch inserted @_torchExtraArgs between the two halves. Those arguments are all flags and their values, never positional packages, so uv accepts them anywhere on the line; moving them after the index restores the literal and changes nothing about the resolve. test_setup_ps1_scrub_covers_pip_fallback slices a fixed 2500 character window from "function Fast-Install" and requires the pip fallback to appear before the finally that restores the scrub. The flag stripping loop added here is sixteen lines and pushed the finally out of that window: the fallback is still inside the try, but the test could no longer see it. Lifting the loop into Remove-UvOnlyResolverFlags puts Fast-Install back near its original size, with the finally at offset 1695 against a 1629 baseline on main. The helper was exercised against the inlined version it replaces on all six shapes: no flags, --index-strategy in both spellings, --prerelease in both spellings, and an empty list. Output is identical, including the empty-in empty-out case that every caller in the tree takes today. * Exempt the Windows on ARM drop list from the xFormers pin check test_installer_never_installs_an_unpinned_xformers flags every quoted xformers literal in install.ps1 that is not the exact pin. The Windows on ARM block added one, but it is a removal list: those names are written into the requirement overrides so the package vanishes from the resolve, since win_arm64 has no xFormers wheel and no buildable sdist. A version pin there would be meaningless, and the name has to match the requirement line exactly for the override to bind. The test already carves out wheel filenames on the same grounds, that a filename is not a spec. This adds the drop list beside it rather than loosening the assertion. Checked that it stays narrow: a bare "xformers" appended at end of file is still caught, and so is a loose "xformers>=0" placed on the line immediately above the drop list, while the real file is clean. * Windows on ARM: fix six ways the native stack picks the wrong thing Six defects found in review, all on the new WoA path and none reachable on any other host. Probe the interpreter that was actually selected. Initialize-WoaNativeCudaTorch answered for $PythonVersion, but Find-CompatiblePython falls back to another supported minor whenever the requested one is not installed. Every native decision is keyed to an interpreter tag, so a 3.13 answer carried into a 3.12 venv staged a cp313 pyarrow wheel and then pinned pyarrow== to it, forcing the Arrow sdist this path exists to avoid. Re-probe once the interpreter is settled, key the staged tags on the venv's own minor, and clear every script-scope answer on entry so a second probe cannot inherit the first one's. Redact the selected torch index. The status line printed $script:WoaTorchIndexUrl verbatim, and it can be UNSLOTH_TORCH_INDEX_URL with userinfo or a token query. It was the only index line in the file that did not redact. Remove-IndexUrlCredentials moves up to the other helpers, since PowerShell resolves a function only once its definition has executed. Hand uv a space-free override path. uv reads UV_OVERRIDE as a space-separated list of files, which this script splits on \s+ itself, and the default StudioHome sits under %USERPROFILE%. An overrides file under a path with a space was read as two paths, neither of which exists, and every later uv call failed with "File not found". Same defect and same remedy as backend/utils/uv_path_safety.py (issue #6503), which the Python half already applies. pip splits its repeatable options the same way; UV_FIND_LINKS is comma-separated and is untouched. Do not enable the native stack for an x64 venv. The migration branches keep a healthy legacy environment as it is, and on this host those are x64 by design. Get-TorchIndexUrl returns the win_arm64 index on the flag alone while the spec lift is gated on the venv's platform tag, so a migrated venv was asked for the bounded x64 specs from an ARM64-only index and aborted with no fallback. Ask the venv the same question the spec branch asks. Accept every spelling of a local wheelhouse. UNSLOTH_WOA_WHEELHOUSE is documented as taking a directory or a base URL, but only drive-plus-backslash and UNC counted as local, so C:/wheels went to Invoke-RestMethod and the native path was dropped on a host whose wheels were present. Ask the filesystem instead. Count a wheelhouse wheel only for the interpreter it was built for. Staging copies every win_arm64 wheel, cp311 through cp314, but both availability checks keyed on the distribution name alone. A cp311 tiktoken in a cp313 venv cleared its skip and its requirement override, and the resolve fell to an unbuildable sdist. Filter on PEP 425 tags on both sides. WINDOWS_ARM64_SKIP_UNBLOCKED_BY takes the whole blocker set rather than one name, since librosa needs numba as well as llvmlite and whisper needs tiktoken too; its openai-whisper key was also never matched, being looked up canonically. Let an explicit CPU pin through the CUDA preservation guard. It ran ahead of every pin check on both sides, so UNSLOTH_TORCH_INDEX_URL=.../cpu was dropped even though download.pytorch.org publishes win_arm64 /cpu torch and torchvision. In setup.ps1 it also left PinChangedForceReinstall false, which skips the dependency pass entirely, so the run kept the CUDA build and reported success. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin that an emulated x64 interpreter still gets the torch flavor repair The CUDA preservation shortcut reads the interpreter, not the machine, and nothing in the suite held it to that. Swapping _is_win_arm64_interpreter for _is_windows_arm64 at the guard leaves 223 of 224 tests here passing, which is the whole margin between a correct guard and one that strands every Windows on ARM install predating native support. Those run an emulated x64 python against ordinary win_amd64 wheels from download.pytorch.org, so the repair has somewhere to go and must still reach them. The fixture drove both predicates from one flag, which is right for every case already here, since a native ARM64 venv is the only place both are true. They separate on an ARM64 machine running an x64 interpreter, so win_arm64_interpreter overrides just that half and defaults to the existing flag. Verified by mutation: with the guard reading the machine instead, the new test is the only one of the 224 that fails. * Windows on ARM: pick the retained llama.cpp kind from the venv arch $expectedKinds asked Get-HostMachineArch, which answers for the machine. A Windows-on-ARM host on the x64 fallback runs the prebuilt helper under an emulated x64 python and installs windows-cuda, so the machine's answer would call that bundle mismatched and discard a working runtime. Test-WinArm64Venv is the same predicate the torch path uses, and it short-circuits free on x64. The block stays inert either way: write_prebuilt_metadata persists no install_kind, so $existingKind is always null. * Windows on ARM: pin the marker split, drop a write-only variable Cross-platform verification of the requirement splits, plus one dead variable the earlier fix left behind. WoaPythonMinor was written in three places and read in none. It was the variable the staging tag was meant to use; that moved to WoaVenvMinor, which is keyed on the venv's own interpreter. Leaving a write-only "the minor we probed" next to the real one only invites a future reader to take the stale answer. test_windows_arm64_requirement_markers pins the mechanism the whole split rests on. Every package needing a different version on win_arm64 is two rows whose markers are exact complements, so exactly one is live everywhere. Both ways of breaking that are silent: an overlap makes pip intersect two specifiers and can render the row unsatisfiable, and a gap drops the package on a platform nobody tested. The tests evaluate every row of every requirements file and every pyproject extra across 60 environments. Two of the checks exist because a scope-limited test skips itself when the bug appears. Spell the machine "arm64" rather than "ARM64" and the conjunction with sys_platform == "win32" becomes unsatisfiable, the split stops existing, and any check that looks for a split finds nothing to look at; Windows on ARM quietly takes the x64 pins and source-builds. That is caught semantically, by asserting no row is dead in every environment, rather than by matching marker text -- torchcodec legitimately pairs 'arm64' with sys_platform == 'darwin' in a separate disjunct. And the expected rows are asserted per source file, because studio.txt and pyproject[studio] mirror each other, so a global check stays green when only one of them loses a row. Verified by mutation: overlapping rows, a lowercase arm64 in either file, aarch64 for ARM64, a deleted WoA row, and either deliberate exclusion (xformers, hf_transfer) losing its marker are each caught, and each by a different assertion. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: close seven gaps the last round left open All seven are on code the previous rounds added, which is where a fresh pass should look. The resolver paths have to be readable by uv or the native stack is not viable. uv splits UV_OVERRIDE on whitespace and accepts no quoting of any kind -- bare, "quoted", 'quoted' and back\ slashed all fail identically on uv 0.10.7 -- so the 8.3 short form was the whole mitigation, and 8.3 generation can be disabled on the volume. A spaced StudioHome then has no space-free spelling and every later uv call dies on a truncated path. Decide that up front and take the x64 path, which never sets these variables, rather than staging wheels into a venv that cannot resolve. Same rule the pyarrow probe already applies. That short form then has to be recognised on the way back in. The block that drops a previous run's resolver variables matched only the long spelling, so after a native install under a spaced home it no longer recognised its own values and the next run in that shell inherited them. Both spellings are checked; a wheelhouse the user set for themselves is still left alone. The probe follows the interpreter that ends up owning the venv. Re-probing for the minor Find-CompatiblePython settled on was not enough: the ARM64 bootstrap below it installs $PythonVersion, which can be a third minor again, and the index and pyarrow source would both have been confirmed for one no longer in play. setup.ps1 is told which index install.ps1 used. It otherwise derives the CUDA index from the driver, and download.pytorch.org publishes no win_arm64 CUDA wheel at all, so a repair or a missing companion on a native venv resolved against an index that cannot serve it. Passed as UNSLOTH_WOA_SELECTED_TORCH_INDEX rather than UNSLOTH_WOA_TORCH_INDEX_URL, which is the user's input override: a second run in the same shell would otherwise read this run's choice as a pin. A wheel's ABI has to agree, not just its interpreter tag. cp313-cp313t is the free-threaded build and uv rejects it on a regular cp313, so counting it as available dropped the package's override and sent the resolve to the sdist the override existed to avoid. The Python half already checked this; the PowerShell half did not. Blockers decide even when the package's own wheel is hosted. tensorboard and librosa publish py3-none-any wheels that the staging step copies, so a wheelhouse can hold the package and still lack grpcio or numba; unskipping it there walks straight into the blocker's sdist. UNSLOTH_LLAMA_ARM64_CUDA=0 is honoured on the direct-upstream path too. It was checked only in the fork resolver, so the documented escape hatch silently did nothing with --published-repo ggml-org/llama.cpp. Each fix is mutation-tested: reverting either Python-side change fails a specific new test, and the PowerShell changes are asserted structurally and by executing the extracted helpers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: build probe URLs with Join-UrlPath A tokenized mirror is a supported form for UNSLOTH_TORCH_INDEX_URL and for the wheelhouse, and "$base/$leaf" puts the leaf inside the token value: https://mirror/whl?token=abc became .../whl?token=abc/torch/, so the preflight read the mirror root or an error and disabled the native stack on a host whose wheels were there. Join-UrlPath already solves this for the xFormers URLs; it moves up next to the other Windows-on-ARM helpers, since PowerShell resolves a function only once its definition has executed and these probes run far above its old position. Applied to the torch/torchaudio preflight and to all three wheelhouse URLs, which had the same shape. * Windows on ARM: abandon the native route before building, and carry the index across shells Two review findings on the previous head. Abandon the native route earlier. The check for a StudioHome uv cannot read -- a path containing a space on a volume with 8.3 short names disabled, which uv splits UV_OVERRIDE on -- sat next to the resolver files it guards, which is after the ARM64 venv has been created. The install therefore paid for a venv it then tore down. Test-WoaResolverPathsUsable now runs inside Initialize-WoaNativeCudaTorch, immediately after the GPU probe, so the route is never chosen in the first place and every re-probe honours it. The verdict is cached, so a user with several interpreters reads the explanation once. Carry the chosen index across shells. install.ps1 exports the index it probed, but a later `unsloth studio update` runs in a fresh shell where that variable is gone, and Windows on ARM is the one platform whose CUDA wheels are not on download.pytorch.org at all, so the index cannot be re-derived from the driver. The manifest now records it, and setup.ps1 falls back to it when the handover variable is absent. That is a deliberate exception to the rule install_manifest.py documents for itself -- the flavor, never the index URL it came from, since a pinned index can carry a token -- so it is narrowed to the case that motivated it. Only credential-free https://pypi.nvidia.com URLs are written, with no userinfo, query or fragment, and setup.ps1 re-checks the same thing on read so a hand-edited manifest cannot redirect a torch install to another host. A mirror the user pinned is deliberately not persisted; they supply it through their own environment as before. tests/studio/install/test_woa_torch_index_persistence.py covers both halves of the guard, the Python that writes and the PowerShell that reads, including the older manifests that carry no such key and the truncated ones a killed installer leaves behind. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: restore the generated requirement overrides on a fresh-shell update The torch index was not the only thing install.ps1 hands over through a process-scoped variable. It also writes StudioHome\woa\overrides.txt, stages a win_arm64 wheelhouse beside it, and exports both through UV_OVERRIDE / UV_FIND_LINKS / PIP_FIND_LINKS. A direct `unsloth studio update` runs in a fresh shell where all three are gone, so the dependency pass resolved without them. That matters because studio.txt pulls ddgs, which requires httpx[brotli,http2,socks], whose brotli extra is Brotli on CPython, and neither Brotli nor brotlicffi publishes a win_arm64 wheel. Confirmed against the published metadata and against the resolver: uv pip compile for httpx[brotli]==0.28.1 targeting aarch64-pc-windows-msvc resolves 8 packages including brotli==1.2.0, and 7 with the overrides applied. With no wheel to install, those 8 mean an sdist build of a C extension -- the exact failure this path exists to avoid. Only reachable since this PR: an emulated x64 venv has a win_amd64 brotli wheel. Restore-WoaResolverEnvironment sits next to the torch-index recovery so both halves of the lost handover are put back in one place, before the torch installs and the dependency pass, which is where install.ps1 has them set for its own run. It restores rather than re-derives: both files persist under StudioHome\woa\, and the generated file is the only accurate record of what was dropped, since the drops depend on what the wheelhouse turned out to hold. No-op unless Test-WinArm64Venv, and it returns untouched if any of the three variables is already set, so install.ps1 driving the resolve in the same process is never clobbered. A native venv whose overrides file was deleted warns and points at install.ps1 rather than guessing the list. Get-UvSafePath is duplicated from install.ps1 because neither script can dot-source the other; a test asserts the two copies stay identical, since a copy that drifts would hand uv a path install.ps1 had already ruled unreadable. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: re-export the recovered index, restore each resolver variable, and cover no-torch Three follow-ups on the fresh-shell recovery. Re-export the recovered index. install_python_stack.py rewrites the manifest at the end of every run from UNSLOTH_WOA_SELECTED_TORCH_INDEX, so recovering the index into a local variable only meant the first fresh-shell update wrote a manifest without it -- erasing the one thing the manifest exists to carry, and making the whole mechanism work exactly once. It is now put back in the environment right after it is recovered, guarded so an empty recovery exports nothing. The value is always one install.ps1 chose or one write_manifest's credential-free allowlist already accepted, so re-exporting widens nothing. Restore each resolver variable independently. Returning when ANY of the three was set was aimed at the install.ps1 case, where all three arrive together, but it also fired for a shell carrying an unrelated UV_FIND_LINKS or PIP_FIND_LINKS -- a corporate wheel mirror has nothing to do with the win_arm64 drop list, and should not cost the user the brotli exclusions. Each is now restored on its own and a caller's value is still never overwritten, so the install.ps1 path is unchanged: every branch no-ops there. Cover no-torch mode. Both recoveries sat inside `if (-not $NoTorchMode)`, and neither is about torch. install_python_stack.py installs studio.txt at function scope with no NO_TORCH gate, and ddgs is not in NO_TORCH_SKIP_PACKAGES, so UNSLOTH_NO_TORCH=1 resolves the same brotli chain; the manifest is rewritten in that mode too. Both have moved above the guard. The torch-index USES stay inside it, where they belong. The new tests assert the placement rather than the text, and one asserts the studio.txt pass is still unconditional so this reasoning cannot go stale unnoticed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: read the manifest before dropping it, publish the index torch came from Three follow-ups, the first of which made the persistence added earlier a no-op. Read the index before the manifest is deleted. setup.ps1 drops unsloth_install_manifest.json before it replaces pip, torch and triton, so a run killed in those leaves the venv marked half-built rather than behind a marker that verifies. The recovery was placed below that drop and read a file that no longer existed, so it returned empty every time -- on precisely the fresh-shell path the persisted index exists to serve, and silently, since empty is also the correct answer for an older install that recorded nothing. The recovery now runs as soon as the dependency guard opens, before the drop. It stays inside that guard: a run that installs nothing has nothing to recover for. Publish the index torch was actually installed from. The native path resolves from the channel install.ps1 probed, while $TorchInstallIndexUrl still names the driver-derived family, so the handoff sent install_python_stack.py's repair to an index with no win_arm64 CUDA wheel and named the flavor cu130 when the installed wheel is +cu134 -- a repair could have tried to correct a working venv to a build that does not exist for this platform. $_effectiveTorchIndexUrl is initialised to $TorchInstallIndexUrl and overwritten only by the branch that installs torch, so every other path publishes exactly what it published before. On the NVIDIA channel the leaf is nvtorch_oot, not a CUDA family name, so no flavor tag is published at all -- the honest answer, and what the block already documents for an unknown leaf. Gate the llama.cpp ARM64 CUDA branch on the opt-out. resolve_upstream_asset_choice is reached through resolve_asset_choice on the fallback paths, so leaving it ungated meant UNSLOTH_LLAMA_ARM64_CUDA=0 worked in direct_upstream_release_plan and silently did not work here. The same gate is already on the equivalent branches at lines 1126 and 3897. The new tests assert ordering and wiring rather than the function in isolation, which is what let the first of these through. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: rebuild a migrated x64 venv as ARM64, and widen the llama.cpp CUDA opt-out Rebuild a migrated legacy environment. The migration branches keep a healthy ~/.unsloth/studio/.venv exactly as it is, and on a Windows-on-ARM host those are x64 by design: every install predating the native stack deliberately bootstrapped an emulated x64 interpreter. The venv-platform guard then saw win-amd64 and stood down to that same stack, so an upgrade left an NVIDIA user on a Triton that cannot compile for sm_121. Only a SECOND installer run recovered, because migrating makes the layout new and the new-layout branch already does preserve-and-recreate. Do that in the first run instead. Just above the venv-creation block, a migrated environment that is not win-arm64 is preserved with the same Start-StudioVenvRollback and the existing creation path builds an ARM64 venv. Scoped to a venv THIS run migrated: a new-layout venv was already moved aside, and one created moments ago came from the interpreter this run chose. $_Migrated is cleared on success. It gates an upgrade-in-place far below that installs unsloth with --no-deps --reinstall-package; left set, that would run against a freshly created empty venv and leave an unsloth with no dependencies. If the rollback throws, the environment is kept and the existing guard disables native exactly as before -- a user's environment is not worth trading for a native stack, so the platform guard keeps the final say. Widen UNSLOTH_LLAMA_ARM64_CUDA to every ARM64 CUDA bundle. This is a deliberate scope change, not only a bug fix. The helper documented itself as governing the upstream bundle when the fork publishes no checksum, but what the flag promises the user is the CPU bundle, and gating only the unverified tail meant the escape hatch would switch itself off the day the fork published an approved windows-arm64-cuda artifact, since the published branch returns first. The docstring now says what the code does. The inner check inside that branch is unreachable once the branch is gated, so it is gone rather than left reading as a fallback that no longer exists. The opt-out tests walk the AST rather than grepping, because the same attribute guards the x64, Linux and macOS paths, which must not be gated: they assert both that every Windows-ARM64 NVIDIA branch is gated and that no other one is. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: gate the pandas 3 row on Python 3.11, and keep the CPU llama bundle when CUDA is opted out pandas 3 needs Python 3.11. Every pandas 3.0.x is requires-python >=3.11 and its win_arm64 wheels start at cp311, which is why the Windows-on-ARM row exists at all -- but the row carried no python_version bound while requires-python still admits 3.9 and 3.10, so `pip install unsloth[studio]` was unsatisfiable on an ARM64 3.9 or 3.10 interpreter. Split on python_version as well as platform, which keeps the partition exact: below 3.11 an ARM64 host takes 2.3.3 and source-builds, slow but resolvable, as it did before win_arm64 wheels existed. Narrowing requires-python instead would drop 3.9/3.10 everywhere to fix one platform. The same two rows appear in studio.txt and single-env/constraints.txt with the same defect, so all three move together. The 603-case marker suite passed throughout this bug's life: the partition was exact and every environment had exactly one live row -- that row just could not install anywhere. So the suite gains the axis it was missing, asserting that a live row is installable on the interpreter it was selected for, checked against the package's own requires-python, plus the complement that no ARM64 Python is left with no pandas row at all. Keep the ARM64 CPU llama.cpp bundle when CUDA is opted out. With UNSLOTH_LLAMA_ARM64_CUDA=0 the selector installs a windows-arm64 CPU bundle on an NVIDIA Windows-on-ARM host, but the mismatch check expected only windows-arm64-cuda there, so it deleted a correct install on every setup and update -- and an update that cannot download then leaves no llama.cpp at all. The previous commit made this reachable in practice by widening the opt-out to the published artifact. The expected kind now follows the flag, INSTEAD of the CUDA kind rather than as well as it: accepting both would keep a bundle installed before the flag was set, so the opt-out would appear not to work on exactly the machine that already had CUDA. Falsy spellings are _upstream_arm64_cuda_allowed's, with a test asserting the two stay identical, since this is now one contract in two languages. The x64 arm is untouched and covered as a negative control. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: pair torchaudio with torch, prove wheels are CUDA, gate prereleases on nightly Take torchaudio only as a matched pair. The GA channel serves torch 2.14.0+cu134 beside torchaudio 2.11.0+cu134, torchaudio 2.11 dropped the exact torch== pin that 2.10 still carried, and the native specs are torch>=2.4 / torchaudio>=2.4 -- so nothing stopped the resolver installing that pair and leaving torchaudio's extension to load against a libtorch three minors newer. install.ps1 already names this hazard on the bounded-trio branch; the native branch was not covered by it. The probe now reads versions rather than existence and enables torchaudio only when its major.minor pairs with torch's, the relationship the old pin encoded. No matching pair means no torchaudio, which is what every Windows-on-ARM install did before this channel published one; it lights up on its own the day NVIDIA ships a matching pair. Establish CUDA positively. The probe rejected a literal "+cpu", which misses two things: PEP 503 hrefs encode "+" as %2B, and PyPI's own win_arm64 torch wheels carry no local version at all, so they are not "+cpu" and were read as CUDA -- sending the host native on CPU-only torch, which is worse than staying emulated because the GPU then goes unused with no fallback. Filenames are unescaped and must carry a +cuNNN local version. A private index whose CUDA wheels carry no +cu tag now reads as unknown and takes the x64 path: a behaviour change, and the safe direction, since that path works. Gate --prerelease=allow on the nightly channel, as install.ps1 already does. setup.ps1 passed it unconditionally and its comment ("the wheels are nightlies") predated the GA channel. "allow" means every prerelease, and it rode on a command carrying unsafe-best-match and an explicit PyPI extra index, so a prerelease of torch or of any shared dependency could outrank the stable GA build on every setup and update. The other two flags stay unconditional; the GA channel needs them. Verified against both live NVIDIA channels (temp/sim10282/probe_test.ps1) and covered offline with stubbed index pages for the parsing and pairing rules. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: five resolver and validation fixes Reject abi3 wheels on free-threaded interpreters. The exact-minor branch accepted "abi3" unconditionally, shadowing the guarded branch below it, so cp313-abi3 was installable on 3.13t while cp311-abi3 was correctly refused. Free-threaded builds do not implement the stable ABI (CPython #111506, PEP 703), so this marked a blocker available, dropped its skip and sent the resolver at a wheel it cannot use. PEP 803's abi3t is 3.15 and outside requires-python, so it is documented rather than matched. The test simulates both builds instead of reading whichever interpreter runs the suite. Validate UNSLOTH_PYARROW_WHEEL. Every other branch of Get-WoaPyarrowSource checks the interpreter and platform tags; this one took any file that existed, so an x64 wheel, a wheel for another minor or a truncated download selected the native path and then failed at resolution, having already given up the working x64 stack. Same checks as the siblings plus the zip signature, read two bytes off a stream rather than loading tens of megabytes. Keep the caller's resolver configuration. The purge above deliberately preserves anything that is not this StudioHome's own, and the assignments then overwrote it. Folded, not appended: uv combines override files rather than letting a later one win, and two files naming one package without distinguishing markers is an error -- and this file declares torch and torchvision. Same shape as New-UnslothTorchOverridesFile. Find-links have no such conflict semantics, so those are appended with the caller's kept and ours first. install_python_stack.py split find-links on os.pathsep alone, which would have read an appended "dirA,dirB" as one unusable path; it now splits on comma, whitespace and os.pathsep. The generated file is UTF-8 now that a caller's line can be non-ASCII. Accept the automatic ARM64 CPU fallback. resolve_asset_choice falls through to the published windows-arm64 bundle when no ARM64 CUDA asset exists, so demanding the CUDA kind deleted a healthy fallback every run. Widening strands nobody: the installer's already-satisfied check is per candidate and CUDA is attempted first, which a test now pins. The opt-out arm stays exclusive. Make the manifest writer and reader accept the same URLs. urlsplit().hostname strips the port, so https://pypi.nvidia.com:443/... was persisted and then refused by setup.ps1, losing the index on the next fresh-shell update. The writer now requires netloc == hostname, admitting exactly what the reader admits and dropping userinfo with it. Declining is preferred to silently rewriting a caller's URL; the default channels carry no port. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: probe and stage the interpreter's ABI tag, not just its minor A free-threaded build installs cp313-cp313t wheels, but every probe here derived its tag from the minor alone and searched for cp313-cp313. The two halves then failed in opposite directions: the index probe found the ordinary GIL wheels and enabled the native stack, while the wheelhouse staging kept exactly those wheels and discarded the cp313t ones the venv could have installed -- all after the x64 fallback had been given up, so the run failed at resolution with nowhere to go. Reachable: Find-CompatiblePython's python3/python PATH scan and its `py -0p` enumeration both accept any interpreter whose --version matches, which a free-threaded build does, and the native path ranks ARM64 builds first. Test-PythonFreeThreaded asks the selected interpreter for Py_GIL_DISABLED and Get-WoaAbiTag turns the answer into cp313 or cp313t, threaded through the torch, torchaudio and pyarrow probes and through the staging scan, which also records the venv's own ABI. A free-threaded host therefore declines the native path today, because no index publishes those wheels, and starts working by itself the day one does -- better than refusing such interpreters outright. Unknown answers GIL, the historical assumption, so nothing moves for a GIL host: there the two tags are the same string. The staging scan also stops accepting abi3 on a free-threaded venv, the rule already applied to _wheel_matches_interpreter, so the two now agree. Verified against the live GA channel: a GIL 3.13 still resolves torch 2.14.0+cu134 and a free-threaded 3.13 finds nothing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: re-probe on ABI change, merge caller overrides, honour an explicit pin Re-probe when the ABI changes, not only the minor. The previous round threaded the interpreter's ABI through every probe but left the re-probe guard keyed on the minor alone -- so the one case where the minor cannot change, a 3.13t chosen for the default 3.13 request, was exactly the case that never reached the call passing -FreeThreaded. The initial probe's GIL assumption stood, native mode was enabled on cp313 wheels, and the venv was then built from an interpreter that installs cp313t. $WoaProbedFreeThreaded is tracked beside $WoaProbedMinor and either changing re-probes, including after Install-PythonFromPythonOrg, where the ABI can change while the minor does not. Detection is scoped to an ARM64 host so no other Windows host spends a subprocess on an answer that cannot matter there. Merge caller overrides on an update instead of standing down. install.ps1 folds; setup.ps1 skipped the generated file entirely whenever UV_OVERRIDE was set, which keeps the caller's file and loses the win_arm64 drop list -- so ddgs -> httpx[brotli] -> Brotli went back to an sdist with no win_arm64 wheel. uv combines override files rather than letting one win, and errors on a duplicate package, so the two cases differ: disjoint files are both handed over, needing no rewriting and leaving each file's relative references resolving against its own directory, and only an actual conflict is merged, with this platform's entries winning because they name what it cannot build. Get-RequirementName(s) do PEP 503 normalisation so Brotli and brotli_cffi compare as one name, and a merge that cannot be written falls back to the caller's file with a warning. Let an explicit pin outrank the persisted index. UNSLOTH_TORCH_INDEX_URL and UNSLOTH_TORCH_INDEX_FAMILY both land in $PinnedTorchIndexUrl, and the recovered WoA index was winning over both -- so a user switching to another CUDA mirror was silently still served by the previously recorded channel. The recovery is a memory of what install.ps1 chose, not a decision the user made, so it is now consulted only when nothing is pinned. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: keep caller resolver entries, survive a bad handoff URL, drop a stale torchaudio, check wheelhouse versions - install.ps1: the resolver purge removes only the entries pointing into this StudioHome's woa directory instead of the whole variable, so a caller's mirror appended behind ours survives a second run in one shell. - install_manifest.py: parse the WoA index URL inside a try/except ValueError; a malformed authority no longer aborts the whole manifest write. - setup.ps1: on a native ARM64 install that leaves torchaudio out of the trio, uninstall an already-installed torchaudio whose major.minor no longer matches torch. - install_python_stack.py: the find-links scan records versions, and a hosted wheel clears a win_arm64 skip only when one of its versions satisfies the pin in the requirements file being installed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: match wheel tags as fields, evaluate markers, bound the purge, version the drop list - install.ps1: one Test-WoaWheelTags parses the PEP 425 fields and expands each dot-separated tag set, replacing seven substring probes. "*cp313-cp313*" also matched cp313-cp313t, so a free-threaded wheel could select and then be staged for a GIL interpreter, failing only after the ARM64 venv was committed to. - install.ps1: the resolver purge treats only the prefix itself or a descendant past a path boundary as owned, so <StudioHome>\woa-mirror survives. - install.ps1: the override drop list records wheel versions and consults the floor the released metadata puts on the name (xformers>=0.0.22.post7) before removing a drop; a lower hosted version keeps it. - install_python_stack.py: requirement markers are evaluated for the running interpreter, so the inactive MeCab row no longer overwrites the active one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: honour any explicit pin, preflight PyAV, keep caller override files in place - setup.ps1: the CUDA-preservation shortcut exempts any explicit torch index pin, not only a /cpu one. It distrusts the inferred host-family comparison; a pin at a cu129 mirror is a stated instruction, and the narrower test skipped the dependency pass and kept cu134 while reporting success. - install.ps1: on a free-threaded interpreter, native mode requires an installable win_arm64 av (PyAV) wheel. The wheel constraints.txt asks for is cp311-abi3, which no free-threaded build can use; PyAV publishes cp314-cp314t but no cp313t, so the index is asked rather than assumed. GIL builds are unchanged and never run the probe. - install.ps1: a caller UV_OVERRIDE file that declares none of our packages is passed to uv where it sits, so its relative -r and wheel paths keep resolving. Only a conflicting file is folded, and its relative references are rebased onto the directory they were written for. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: read the machine arch from the registry, not from the process Fresh install of this branch on the GB10 laptop went down the x64 path: plain "NVIDIA GPU detected" and cu130 from download.pytorch.org, which publishes no win_arm64 wheel, with no line saying why. Running the probe's gates one at a time found it: Get-HostMachineArch answered x86_64, so Initialize-WoaNativeCudaTorch returned on its first check before printing anything, and every later gate, nvidia-smi, the NVIDIA index, the pyarrow wheel, would have passed. Every signal it trusted follows the process, and the install ran from an x64 emulated shell. There PROCESSOR_ARCHITECTURE is AMD64, PROCESSOR_ARCHITEW6432 is empty because it is a WOW64 variable and x64 emulation on ARM64 is not WOW64, and .NET Framework's OSArchitecture reports X64, a bug fixed only in .NET 5. VS Code's integrated terminal is often x64 on these machines, so this is a real user path, not a test artefact. The machine-scope PROCESSOR_ARCHITECTURE from the registry is the one value emulation cannot misreport, and it now leads the signal list in install.ps1, in setup.ps1, and in setup.ps1's Get-UvHostArch so the uv an update picks matches the one install.ps1 chose. The per-process signals stay as fallbacks; on x64 hardware every answer is unchanged. setup.ps1 is the one that made this urgent. Test-WinArm64Venv short-circuits on the same function, so from an x64 terminal an update of a native ARM64 venv would have skipped the CUDA preservation guard and repaired a working cu134 torch to a cu130 that cannot install there. The python side had the identical hole: _is_windows_arm64 read the same three signals and its docstring promised ARCHITEW6432 would be set under emulation, which it is not. It reads the registry first now. Two of its three callers drop torchaudio on the grounds that no win_arm64 wheel exists, which is a property of the interpreter, not the machine: an emulated x64 venv on an ARM64 box installs win_amd64 wheels and download.pytorch.org publishes torchaudio for those. Both move to _is_win_arm64_interpreter. The third caller prints the Vulkan note, a machine fact, and stays. Verified with the real functions lifted from install.ps1 and run under both an x64 emulated and a native ARM64 PowerShell: both now answer arm64 and select the nvtorch_oot index. Under the x64 and the ARM64 CPython on the same box the machine predicate is True for both and the interpreter predicate is True only for the ARM64 one, which is the disagreement the wheel filters rely on. Tests: the arch-probe cases patch the registry helper so they answer the same on an ARM64 host as on x64 CI, and gain the emulated case with every per-process signal saying AMD64. The Vulkan-advice fixture takes the host as a parameter instead of reading the box it runs on; three of its tests had failed on every ARM64 dev box for that reason. * Make the marker-evaluation test independent of the host running it test_pins_are_read_canonically_and_markers_evaluated wrote a row under `sys_platform != 'win32'` and asserted the pin survived because "its marker holds on this host". It holds on a Linux runner and on no Windows machine, which is every machine this file is about; on the GB10 laptop it fails with a KeyError. The reader answers True, False or None for a marker, and only False drops the row, so the test now uses one marker that holds everywhere and one that holds nowhere, and asserts both directions. Same coverage, no dependence on the box. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: prepend the managed wheelhouse, rebase merged overrides, tolerate a self-copy - setup.ps1: the resolver recovery prepends $StudioHome\woa\wheels to an existing UV_FIND_LINKS / PIP_FIND_LINKS instead of standing down. find-links are additional search locations, so skipping ours because a caller had set a mirror left the staged win_arm64 wheels out of the search and sent pyarrow to an sdist. - setup.ps1: Resolve-WoaOverrideLine is carried over from install.ps1 and every line folded into overrides.merged.txt is rebased against the caller file's directory, so a relative -r, -c, -f or wheel path keeps resolving. - install.ps1: both wheel staging copies compare normalised paths first. A wheelhouse pointed at the managed wheel directory made Copy-Item overwrite an item with itself, which under ErrorActionPreference Stop aborted the install. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: open the supplied wheel, and honour a configured win_arm64 mirror - install.ps1: UNSLOTH_PYARROW_WHEEL is validated with Test-ZipArchiveReadable rather than a two-byte PK sniff. An interrupted download still carries that signature, and accepting one selected the native path, staged the broken file and failed the pyarrow resolve with the x64 stack already given up. - setup.ps1: UNSLOTH_WOA_TORCH_INDEX_URL is read ahead of the handover variable and the manifest. write_manifest persists only NVIDIA's own channels, since any other URL could carry a credential, so a corporate win_arm64 mirror survives a fresh shell nowhere else; without this a direct update fell back to the driver-derived index, which publishes no win_arm64 CUDA wheel. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: guard the third staging copy against a self-copy UNSLOTH_PYARROW_WHEEL may point at the already-cached wheel under $StudioHome\woa\wheels, which is how an offline run reuses the managed cache. Copy-Item refuses to overwrite an item with itself; here the throw is caught, so instead of aborting it disabled native mode after the ARM64 interpreter and venv had already been chosen, leaving the run on a CUDA index with no win_arm64 torch. Test-WoaSamePath now guards all three staging copies, and a counting test keeps a fourth from being added without one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the new marker and path tests independent of the host running them Same defect as the pin-reading test: a fixture marker of sys_platform != 'darwin' or python_version < '3.14' picks the other row on a macOS 3.14 box, and literal POSIX expectations for Resolve-WoaOverrideLine are wrong on Windows, where GetFullPath returns a drive-qualified backslash path. The marker fixtures now use one marker that holds everywhere and one that holds nowhere, and the rewriter expectations are derived with os.path.abspath. * Windows on ARM: trim the commentary Comment-only pass over the branch: collapse wrapped explanations, drop restatements of the line below them, and shorten test docstrings that carried more than their first sentence. Every fact a comment was carrying is kept. The comment-only property is checked two ways: comment_tools.py for the Python files, and a PowerShell tokeniser comparison for install.ps1 and setup.ps1 (32293 and 29539 non-comment tokens, identical before and after). One test changes: test_no_staging_copy_is_left_unguarded anchored its search on a comment, so a comment pass could break it. It now anchors on the code the region starts with, and still fails when a guard is removed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: six review fixes - _version_satisfies now answers with packaging when it is importable, with prereleases off unless the specifier asks for one, which is the policy uv resolves under. A wheelhouse tiktoken-0.13.0rc1 reduced to (0, 13, 0) and read as satisfying ==0.13.0, so tiktoken was unskipped and the resolve fell to the ARM64 sdist. The numeric fallback now refuses a version it cannot model rather than guessing. - Every pyarrow candidate is floored at the ARM64 constraint (>=21.0.0). A tag-compatible 19.x in the wheelhouse selected the native path, and staging then wrote an exact pyarrow== override that the constraint rejects, after the ARM64 venv had been built. A test keeps the two numbers in step. - --prerelease=allow is decided by the version of the wheel the probe selected, not by whether the index URL says nightly. A mirror of NVIDIA's prerelease channel need not say so, and without the flag uv takes the stable win_arm64 CPU torch from the PyPI extra index instead of the CUDA build. install.ps1 hands the answer to setup.ps1, which cannot probe. - The ARM64 CUDA-preservation shortcut in install_python_stack.py exempts every explicit pin, not only a /cpu one. A user asking for cu129 by URL or family was silently left on the cu134 build. setup.ps1 already did this. - Both requirement scanners follow a nested -r. uv reads those includes, so a conflicting package one level down made two override files look disjoint; handing uv both is an error, and the caller's capped torch could win. The fold flattens the include, rebasing each line onto the file it was read from. - install.ps1's torchaudio and prerelease flags apply only while the index they were measured on is still in force. Change UNSLOTH_TORCH_INDEX_URL and re-run an update in the same shell and the pin outranks the handover for the install, while a stale 1 asked the new index for a torchaudio wheel it does not publish and aborted the whole torch update. Each fix is mutation-tested. Suite: 5902 passed, four pre-existing failures unrelated to this branch (managed node runtime, tokenizers negative control). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: a version floor has to order prereleases Test-WoaVersionAtLeast compared the numeric release and then .postN, so 21.0.0rc1 cleared a 21.0.0 floor. Staging writes an exact pyarrow== override from whichever candidate it picks, and pyarrow>=21.0.0 rejects that rc, which is the failure the floor was added to prevent. A marker attached to the release now sorts below it. A larger release is unaffected, so a wheelhouse nightly still clears an older floor: the pyarrow 24.0.0.dev260 an end-to-end GB10 run staged is named in a test, since a floor that rejected it would break a verified install. The eleven-row ordering table is cross-checked against packaging in the same class rather than asserted from memory, because the PowerShell cannot import it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GPU driver: an all-[N/A] compute-apps listing is "cannot attribute", not "nothing appeared" Ran the Kaggle GPU payload against a fresh native install on the GB10 laptop. Every workflow passed except cli_run, which said `unsloth run` served from the CPU because no process appeared on the card holding more than 0 MiB. It had: with the exact flags the CLI passes (-ngl -1 --threads 2) the same llama-server generates at 7.7 to 8.3 ms per token, inside the --gpu-layers 99 band (6.5 to 7.1) and far from a two-thread CPU control (18.4). llama-server --list-devices shows CUDA0 with 46 GB. The verdict was blind, not wrong about the rule. On Windows (WDDM) and on unified-memory parts nvidia-smi lists every CUDA process with used_gpu_memory [N/A], and parse_compute_apps drops those rows, so a server that did appear during the window looked like nothing appeared and the device-delta fallback never engaged, because that branch only runs when nvidia-smi cannot enumerate at all. Listing alone is not evidence either: a -ngl 0 server is listed too, since a CUDA build creates a context regardless. Measured here: the device counter reads 132 MiB idle, 272 MiB with that bare context, 560 MiB with the 270M model offloaded, so the existing 200 MiB fallback threshold separates the two. count_listed_pids tells listed-but-unattributed apart from empty, and nvidia_compute_apps returns None for it, which is what the verdict already treats as "judge on the device delta". Readable listings are unchanged, and parse_compute_apps keeps dropping [N/A] rows, so its tests stand. Tests added for the count, for the None, and for the verdict on the numbers measured on this box. test_a_box_without_nvidia_smi_reports_no_gpu_rather_than_crashing fails on this host on unmodified main too, since a real nvidia-smi leaks through its simulation; not touched here. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: sqlite-vec cannot resolve in the studio extra pyproject pinned sqlite-vec==0.1.9 unconditionally, so a native win_arm64 interpreter running pip install "unsloth[studio]" hit a row it cannot satisfy: sqlite-vec publishes win_amd64 wheels only and no sdist at all, so there is nothing to build and no version to floor to. The runtime skip list already covers the name, but that filtering never runs for a direct extra install, which resolves pyproject's rows. Dropped on win_arm64 with the same negative marker cryptography and pymupdf use, mirrored into studio.txt so the two stay in step. Nothing regresses: rag_db.py guards the import and the router mounts either way, reporting RAG unavailable. Guarded generally for the studio extra: no skip-list package may be left live on win_arm64, and a marker that drops one must drop it nowhere else. Scoped to studio on purpose, since the other extras are x64 recipes pinning win_amd64 wheel URLs and were never a claim about this platform. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: three review fixes - The resolver purge split every variable on [,\s] and rejoined each with its own separator. uv reads UV_FIND_LINKS comma-separated, so a caller path like C:\private wheels is ONE directory there: the shared split tore it into two fragments, dropped the managed entry, and rejoined the pieces with commas, quietly losing an air-gapped mirror. Each variable is now split the way the tool that reads it splits it, in install.ps1 and in _find_links_wheel_versions, where the same class of split made wheels in a space-containing directory invisible to the skip list. - The native ARM64 index arguments were gated on $UseUv, so when uv could not be obtained at all the pip fallback got only --index-url pointing at a channel that publishes just the torch trio, leaving their shared dependencies unresolvable. Remove-UvOnlyResolverFlags already exists to make that list safe for pip, dropping --index-strategy and rewriting --prerelease=allow as --pre, so the arguments are built either way. - The manifest was the only durable record of the WoA index, and the dependency pass deletes it before rebuilding it. A run that died in that window left the next update with no handover and no manifest, so it fell to the driver-derived cu130 and failed on an index with no win_arm64 wheel, on every retry. A marker beside overrides.txt, written before the manifest is dropped, now carries it. It persists exactly what the manifest persists, checked on write as well as on read, so a mirror pinned with a token is not written to disk and a hand-edited file cannot redirect a torch install. Seven mutations, all killed. The first attempt at the marker tests passed with the write-side guard removed, since the read guard alone defends a round-trip; a test that inspects the file on disk covers it now. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: two review fixes - New-UnslothTorchOverridesFile kept relative references working by writing the merge into the caller's override directory, which only helped when every override file shared one. Round 19 made two the normal case on the native path, since a non-conflicting caller file is now kept where it sits, so the merge fell through to %TEMP% and every relative include or wheel path resolved against nothing. Each folded line is rebased against the file it came from instead, and nested includes are flattened for the same reason, so where the merge lands no longer changes its meaning. - The dependency pass fell back to pip whenever uv failed or was missing. On the native stack that does not recover: the overrides lift the released torch cap, which no win_arm64 CUDA wheel satisfies, and drop the packages with no win_arm64 build, and pip has no override mechanism to translate them into. Constraints cannot stand in, they narrow a requirement rather than replace one. Both fallback routes now refuse with the reason and name the remedy instead of resolving the wrong stack. Five mutations, all killed. Restored one behaviour the new scanner had changed: comment lines are carried through a fold again, which test_unsloth_torch_override.ps1 was right to be checking. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: four review fixes - An include's inline comment is not part of its path. "-r nested.txt # corporate pins" is a valid line, and capturing the comment meant the file never opened, so a torch conflict inside it went unseen and the two override files were handed to uv as disjoint. Stripped only where whitespace precedes the hash, so "-r a#b.txt" keeps its filename. - An index the marker may not record now CLEARS it rather than leaving the last one in place. A host that moved from NVIDIA's channel to a credentialed corporate mirror kept a marker naming the old public channel, and since the manifest does not record the mirror either, the next fresh-shell update read the marker and put torch back on a source the user had left. An unrecordable index inherits nothing. - The local wheelhouse mirror counted a wheel on its filename, which is all _find_links_wheel_versions reads, so a truncated archive took its package off the ARM64 skip list and uv failed the whole dependency pass on it. It is opened before it is copied now, the same check the remote branch already applied. - The GPU harness read a mixed nvidia-smi listing as proof of CPU: a readable row on one card keeps the mapping nonempty, so the all-[N/A] guard stays quiet and the newly launched server, reported as [N/A] on a WDDM or unified-memory card, is simply missing from it. A pid that appeared but carries no figure now defers to the device-wide delta, as the all-[N/A] case does. Both samples come from one nvidia-smi call, so the attributed mapping and the listed pids describe the same moment. Seven mutations, all killed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: two review fixes - The local wheelhouse pyarrow was selected and staged on its filename. Last round's archive check covered the OPTIONAL mirror, where a bad wheel costs one feature; this is the mandatory one, and staging writes an exact pyarrow== override from whichever file it picks, so a truncated wheel chose the native route and then failed the resolve with x64 already given up. The probe and the staging now apply the same filter, tags and readability, so they cannot pick different files. - The device-wide VRAM total is only a measurement of this process when this process owns the card. Under --studio-concurrent a training leg shares it and both allocates and frees inside the window, which is how one run read the delta as -182.0 MiB while the server held 2.6 GB, so accepting a rise there would pass a CPU-served run on somebody else's memory. Both fallbacks to the total now refuse when anything was on the card before the launch. Decided on evidence rather than configuration: the payload is not told which half it is, but a pid already present says so, and an empty card keeps the fallback the [N/A] parts need. Six mutations, all killed. Two tests from last round asserted the mixed listing case with a co-tenant present, which is exactly the scenario that must not pass; they now use an owned card, and the shared variants are their own cases. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: quote a rebased override path that needs it Resolve-WoaOverrideLine stripped the quotes off an -r/-c/-f target and wrote the absolute path back bare. These options take one file argument, so a space truncated it and the dependency pass failed opening a path cut at its first space. Two ways in, and only one is about quoting. A caller who quoted the value had the quotes discarded. A caller who had no reason to quote a plain relative name gets a space anyway when it rebases onto a directory that has one, which no amount of preserving would have caught. The rebased value is quoted whenever it contains whitespace, so both are covered, and setup.ps1's parity copy is kept identical by a test. The first version of this had the option token disappear: -match on the rebased value replaces , so reading the groups afterwards returned a bare path where -c <file> had been. The file already carries a comment warning about exactly that. The groups are copied out first now, and a test asserts the option survives. Four mutations, all killed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: four review fixes - The marker recorded the WoA chain, which never consults the generic UNSLOTH_TORCH_INDEX_URL pin the torch steps actually prefer. A run pinned elsewhere installed from the pin and then recorded an NVIDIA channel it had not used, and the next fresh shell went back to it. The pin decides what is recorded now, and an unrecordable one clears the marker. - _find_links_wheel_versions read PIP_FIND_LINKS as well as UV_FIND_LINKS. uv does not consume it, so a wheel hosted only there was counted as available, dropped off the skip list, and then invisible to the uv resolve that followed. pip cannot be the resolver on this path either, since pip_install refuses the fallback once the overrides are in force. - A transitive blocker had no direct row to satisfy, so any hosted version counted. tensorboard 2.21.0 requires grpcio>=1.74.0 and librosa 0.11.0 requires numba>=0.51.0: a wheelhouse holding an older one lifted the skip, and the optional package's own metadata then failed the whole extras pass. The floors are recorded with the release they were read from, and a test fails if extras.txt moves off that pin. llvmlite has no entry deliberately: nothing states a floor for it. - The PyPI pyarrow route cleared native mode on a wheel it then forgot, so no exact override was written. With only pyarrow>=21.0.0 in force, uv takes the newest release, and one that ships an sdist for this interpreter builds Arrow from source. The matched wheel is pinned now, and the probe clears the name on entry so a re-probe cannot inherit it. Six mutations, all killed. The first attempt at the second fix keyed on USE_UV, which is False until the bootstrap runs, so it read the wrong variable everywhere else; the list is unconditional now, which is correct because uv is the only resolver that reaches it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: open the downloaded pyarrow wheel before keeping native mode A mirror can serve a truncated body with a 200, and Invoke-WebRequest reports that as success. This is the mandatory wheel: the exact pyarrow== override is written from it, so an unreadable download kept the native route and then failed uv on that override, after x64 had been given up. The optional remote wheels below already opened what they kept; this one was the last route that did not. It throws into the existing catch, so the failure falls back to x64 rather than continuing, and the bad file is removed first -- left in the managed directory it would be read by the resolver on every later run, and by _find_links_wheel_versions as proof the package is available. Three mutations, all killed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record the effective torch index in the manifest, not just the marker install_python_stack.py writes UNSLOTH_WOA_SELECTED_TORCH_INDEX into the manifest as woa_torch_index, and the read chain in setup.ps1 prefers the manifest over the marker. Exporting the WoA chain while saving the pin left the manifest shadowing a marker that had already been corrected, so the next fresh shell went back to the index the run had not installed from. Resolve the value once from Get-PinnedTorchIndexUrl, falling back to the WoA chain, and write both records from it. The torchaudio and prerelease flags install.ps1 leaves in the shell describe the index it probed, so drop them when the resolved index differs from that one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop the generated requirement snapshots that reached the branch _filter_requirements writes its skip-filtered copy beside the source so relative -r/-c includes still resolve, and pip_install unlinks them in a finally. A test that calls the helper directly, or an install killed mid-run, leaves them in the checkout, where a git add -A picks them up. Nine were committed that way. They are a snapshot of a file already in the tree, so a stale one reads as a second, silently divergent copy of the pins. Remove them, ignore the pattern, and add two tests: one that no tracked file matches the generated shape, one that the ignore rule covers it. The shape is matched exactly rather than by a leading dot, so a real hidden requirements file such as .constraints.txt stays in scope. * Persist an explicit pin, install hosted optionals, scope the GPU samples Three separate holes. setup.ps1 ran the index persistence block only when the WoA chain had something in it. A venv installed through a credentialed mirror has nothing to recover, so a later run pinned at an NVIDIA channel installed from the pin and recorded neither record, and the next fresh shell fell back to the driver-derived index that publishes no win_arm64 CUDA wheel. Either record now opens the block. install.ps1 reported that a hosted optional wheel was keeping its package, but only declined to emit the removal override. hf_transfer and xformers are excluded on win_arm64 by marker, so no requirement survived for the wheel to satisfy, and torchcodec's only requirement line was filtered out here unconditionally. Make that filter conditional on a hosted wheel, and install the distributions the metadata cannot ask for, --no-deps and best effort. The Kaggle GPU harness sampled every physical card while the payload was pinned to one, so an unrelated process on a hidden card could supply the memory that carried a CPU-served run past the assertion. Both samplers now pass CUDA_VISIBLE_DEVICES to nvidia-smi -i. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the probed CUDA trio, and validate the wheelhouse xformers uv documents --index-strategy unsafe-best-match as selecting the best version from the combined candidate set of every index, and the PyPI extra index is there because NVIDIA's channel publishes only torch, torchvision and torchaudio. So torch>=2.4 takes PyPI's stable win_arm64 CPU wheel the moment it is one release ahead of the selected channel, and the native GPU path is replaced by a CPU build that imports perfectly. Probe torchvision as well as torch and torchaudio, keep all three versions, and install them pinned exactly, local +cu tag included. An unreadable version keeps the floor it had before. The optional wheelhouse installs added last commit took the bare name, so any tag-compatible wheel landed: below the declared xformers floor, or built for another torch and CUDA pair, whose extension then fails to load while the step reports success. Select the newest hosted version that clears the floor, install that version, and for xformers compare cpp_lib.json against the resident torch, removing it on a mismatch the way the post-repair resync already does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the repair path too, pair the companion wheels, fix two availability rules The forced-repair path in setup.ps1 installs the trio with the same unsafe-best-match and public PyPI extra index as the install path, so open-ended specs there had the defect the install path was just fixed for. Re-probe the effective index at repair time and pin what it publishes; a repair can happen in a fresh shell long after the install, so a remembered version would be stale. The probe is a parity copy of install.ps1's, compared body for body by a test. Pinning both torch and torchvision exactly made the pair mandatory, and NVIDIA's nightly channel publishes each project on its own schedule while nightly torchvision metadata pins its exact torch. Select the companions as builds of the torch that was chosen, matching on the dev stamp and the local tag, and fall back to the floor when the index carries no companion from that build. Test-WoaVersionAtLeast read .dev as a pre-release of the release, so 0.0.22.post7.dev0 came out at or above 0.0.22.post7. PEP 440 hangs .devN off whatever precedes it. Track it separately and as an ordinal. The win_arm64 skip list was decided from the local wheelhouse alone, so a native CPython 3.14 host dropped librosa even though llvmlite and numba publish cp314 win_arm64 wheels. Record what the public index resolves for this interpreter. Also removes a file named - from the root: a workspace simulation ran uv pip compile -o -, which writes a file rather than to stdout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the Windows on ARM comments Post-convergence comment pass over the three files that carry the added prose: collapse the multi-line rationales to the fact each one is there to state, and delete the ones that restate the line below them. 782 added comment lines to 569. Comments, docstrings and whitespace only, verified with comment_tools. * Stop the override file from undoing the CUDA pins, and two probe fixes uv's --overrides replace a requirement's version even when it is named directly on the command line: verified on 0.10.7, an override of packaging>=20 beat a CLI packaging==24.0. The generated file carries torch>=2.4 and torchvision>=0.19, so it discarded the exact CUDA pins the probe selected and best-match then took PyPI's newer CPU wheel. Build the value for that one command without the trio, passing every other file through, and restore it in a finally so the later unsloth resolve still gets the drop list. nvidia-smi failing to answer is not evidence the GPU is gone. A transient failure during a direct update dropped windows-arm64-cuda from the expected kinds, deleted the working llama.cpp tree and installed the CPU bundle. Accept a win_arm64 venv on a persisted WoA CUDA index as evidence too; only NVIDIA's own channels are ever persisted, so a cpu pin cannot claim it. The public-index wheel table is only availability if the resolve looks at PyPI. Offline, or pointed at an exclusive corporate index, unblocking librosa dropped the skip and then failed the extras pass on an unavailable numba chain. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Take a wheel from PyPI when PyPI builds it, not from the wheelhouse The Windows on ARM wheelhouse exists for distributions PyPI has no win_arm64 build of. regex is not one of them: PyPI has published win_arm64 regex since 2025.7.29, and the wheelhouse copy is the same version. The staging directory is first in UV_FIND_LINKS, so ours won the tie and the venv got our binary of a wheel the project releases itself. Nothing needed it there either: regex is in no skip list, no drop list and no constraint, so it unblocked nothing. Staging now asks PyPI first and skips the copy when PyPI publishes that project at or above the version being staged. Version-aware in both directions, because the reverse is a downgrade: an upstream release BEHIND the wheelhouse leaves ours in place. The probe is PyPI-only on purpose. Test-WoaWheelAvailable falls back to the wheelhouse, and a wheel is by definition present in the wheelhouse it is being staged from, so reusing it would answer $true every time and the guard would never fire. Its PyPI leg became Test-WoaPyPIWheel and it now calls that, so there is still one place that knows how to read the simple index. Tag matching is relaxed for this caller alone. hf_transfer ships cp38-abi3 and sqlite_vec ships py3-none, both of which import on cp313; the exact-tag test would call them foreign and go on shipping ours the day upstream publishes one. Free-threaded venvs keep the exact test, which is what it was written for. A wheel this venv could not have used is never judged: a cp312 wheel is not made redundant by a cp313 wheel. pyarrow keeps its own path. Get-WoaPyarrowSource already probes PyPI before the wheelhouse and reports which one it used, which is why the generic loop skips it. Two comments were wrong and are corrected. pyarrow has never published a win_arm64 wheel at any version, so the constraints floor is the floor of the wheels we stage rather than of anything upstream ships, and Get-WoaPyarrowSource checks an explicitly supplied UNSLOTH_PYARROW_WHEEL before PyPI, not after. * Fix uninstall.ps1 Split-Path calls that throw on Windows PowerShell 5.1 Split-Path puts -LiteralPath in its own parameter set, which carries only -Resolve and -Credential. -Parent belongs to the -Path set, so naming both resolves no parameter set and the call throws AmbiguousParameterSet. It throws at runtime rather than at parse time, which is how eight of these shipped in scripts/uninstall.ps1. Two of them sat outside a try/catch, so a real uninstall printed a red error and left the derived path null: the legacy sibling build at <parent>\stable-diffusion.cpp was never removed, and the empty path then tripped the deny list into "refusing to remove unsafe path:" with nothing after the colon. The other six were swallowed by their surrounding try/catch and failed silently. That left _RootFromConf returning nothing, so studio.conf-based root discovery never worked, and it disabled the parent-of-USERPROFILE rule in _IsUnsafeRoot, so C:\Users read as a safe recursive-delete target. Drop -Parent rather than switching to -Path. -LiteralPath on its own already returns the parent, and -Path globs, so an install root containing [ ] would be read as a wildcard. Verified on this box under Windows PowerShell 5.1.28000.2704 against a real env-mode install: the root, its studio.db, auth, outputs and exports all go, an Unsloth-owned sibling stable-diffusion.cpp now goes with them, and an unowned checkout at the same path is still kept. The new test is static because CI has no Windows PowerShell 5.1 to run the scripts under. It covers install.ps1 and studio/setup.ps1 too, which are already clean. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Give the uninstaller's remove retries a long enough budget Uninstalling a Studio that had actually been used left the whole install tree behind. The stop pass kills the server, but torch inductor's compile workers keep handles on the .py files they wrote under <root>\TORCHINDUCTOR_CACHE_DIR for several seconds after that. Those are plain data handles on files the worker never loaded as a module, so neither the ExecutablePath pass nor the loaded-module pass in _StopProcessesLockingRoots has a process to kill, and the only thing left to do is wait. _RemovePath waited 700ms three times, so it gave up after about 2.1s and reported 'could not remove', leaving the tree and studio.db on disk with a summary telling the user to delete them by hand. Replace the flat sleep with an escalating backoff that tops out near 20s, which covers the teardown with room to spare and still bounds the run. * Say that a Windows on ARM desktop build needs clang Building studio/src-tauri for aarch64-pc-windows-msvc with only MSVC BuildTools installed fails with ring@0.17.14: failed to find tool "clang": program not found because ring and aws-lc-sys assemble their aarch64 sources through cc-rs, which calls clang rather than cl.exe on that target. The error names the crate rather than the missing toolchain, and the prerequisite was written down nowhere: not in the workflows, not in CONTRIBUTING.md, not in the manifest. Recorded in build.rs, which is the file whose job this is and the one a reader reaches for when the build fails. No behaviour change. Not a hypothetical: the desktop app was built and launched on a Windows ARM64 box for this branch, and this was the only thing standing in the way. It builds and runs natively there once clang is present, PE machine 0xAA64, window up and the frontend rendering off tauri.localhost. * Fix the two CI reds: a dead import and a pid-file race Import hoist lint: run_studio_gpu.py still imported count_listed_pids after the listing helpers grew up around it. nvidia_compute_apps_listing now returns the pids themselves via listed_pids, because a MIXED listing needs names rather than a count, so the count import had no caller left and only survived in a comment. Dropped both. count_listed_pids itself stays: gpu_assert still exports it, its tests still cover it, and listed_pids documents itself against it. Tauri rust tests: stop_update_kills_descendants_after_the_group_leader_exits panicked with ParseIntError { kind: Empty }. The test child runs `echo $! > "$1"`, and the redirection creates that file before writing to it, so waiting on is_file() can win the race and read an empty string. It is a real flake rather than anything about this branch, and it does not reproduce reliably: the same test passed on main minutes earlier. It surfaced here only because the Tauri workflow is path filtered on studio/src-tauri and the clang note is the first change this branch makes there, so this was its first run. Wait for a pid that parses instead of for a path that exists, over the same kind of budget. A child that never writes one now fails with a sentence rather than an unwrap panic. * Close three holes in the PyPI-first wheelhouse guard Review of the guard I added in 7fe37c6 turned up three ways it could do harm, one of them worse than the duplication it was written to fix. The bad one: skipping a wheel made it stop counting as available. $WoaWheelNames is rebuilt by scanning the staging directory, and a $WoaDropCandidates name that is missing from it gets emitted as `name ; platform_machine == "AMD64"`, which EXCLUDES the package on ARM64. So the day PyPI published a win_arm64 hf_transfer or brotli, the guard would have turned "installed from our wheelhouse" into "not installed at all" -- the opposite of preferring upstream, firing on exactly the event the guard exists for. A skipped wheel is now recorded in $script:WoaPyPIProvided with the version PyPI matched, and that is merged into $WoaWheelNames, so availability reads the same whether the wheel came from the wheelhouse or from PyPI. The xformers floor still applies to the PyPI version, so a release below it keeps the drop. abi3 was treated as universally compatible. It is forward compatible from the version it was built against, so a cp314-abi3 wheel does not import on cp313, and calling ours redundant against one would have left the package uninstallable. Test-WoaWheelTagsUsable now applies the same minimum rule the staging scan already used for abi3 and py3N, and free-threaded venvs keep the exact-tag test, which is what it was written for. PyPI publishing a wheel is only availability if the resolve will look at PyPI. UV_INDEX_URL and UV_DEFAULT_INDEX REPLACE the default index, and offline flags remove it, so on a corporate mirror the guard was dropping our wheel and leaving it obtainable from nowhere. Test-WoaResolveReachesPyPI mirrors _public_pypi_is_reachable in studio/install_python_stack.py, which reasons about the same five variables for the same reason. An unreachable pypi.org already failed safe; reachable but unused did not. The new tests execute the helpers under pwsh instead of reading install.ps1 as text, because two of these three were wired correctly and answered wrongly, which is precisely what a source-shape test cannot see. Hermetic: the live PyPI leg is exercised on hardware, not in CI. * Don't recursively delete ~/.unsloth/studio unless it is ours The custom-root branch refuses a path that carries none of the four ownership sentinels ("refusing to remove non-Unsloth path"), and then the default root a few lines down is removed with no check at all. So a bare run, the documented one-liner with no UNSLOTH_STUDIO_HOME set, recursively deletes whatever happens to sit at ~/.unsloth/studio, and the empty-dir prune takes ~/.unsloth with it. That path is not far-fetched to own. On a machine where Studio was only ever installed in env mode, ~/.unsloth/studio is an ordinary place for someone to keep notes or a checkout, and nothing warns them that running the uninstaller for their custom install will also clear it out. Hit it here for real: a run against a disposable env-mode install removed an unrelated directory that held only a log file, and reported nothing unusual while doing it. Gate it on _IsStudioRoot, the same test the custom root gets. An interrupted install that lost every sentinel is now left on disk with a message instead of removed, which is the failure direction that does not destroy data. Both scripts had it; uninstall.sh line 588 was the same bare call. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pair stable companions by release line, honour uv config files, and clean up the override swap Stable torch, torchvision and torchaudio releases all carry an empty dev stamp, so the CUDA local tag alone paired a companion from any release the index still served and the exact-pin install then asked for a pair that does not exist. Test-WoaWheelPairsWithTorch now applies the release mapping for stable builds: torchvision 0.(M+15) pairs with torch 2.M and torchaudio agrees on major.minor. Nightlies still pair by dev stamp. The setup.ps1 parity copy is identical. Only environment variables were consulted before assuming a resolve reaches public PyPI. uv also discovers uv.toml and pyproject [tool.uv] from the current directory and its parents, then the user and system files, so a no-index or an exclusive default-index set there still answered "PyPI". Both the PowerShell probe (Test-WoaResolveReachesPyPI, now also gating the pyarrow source probe) and the Python one (_public_pypi_is_reachable) read that configuration, honour UV_NO_CONFIG and UV_CONFIG_FILE, let an index in the environment decide alone, and resolve doubt to "not PyPI". The filtered torch-step override was written with GetTempFileName(), which follows %TEMP% and so could produce a quoted, uv-rejected path, and the copies were never deleted. New-WoaTorchStepOverrideValue now writes under the already validated WoA directory, passes every path through Get-UvSafePath, and reports the copies so the caller removes them in finally. setup.ps1 never performed this swap around its own CUDA trio install, so the restored overrides.txt floors undid its exact pins too; it now swaps the same way. On the manifest fast path the dependency block is skipped, so the llama.cpp NVIDIA-evidence check read an unset index and a transient nvidia-smi failure deleted a working CUDA bundle. The check now reads the pin, manifest or marker itself when the pass did not run. sqlite-vec joins the explicit ARM64 optional install map: its requirement is marker-excluded on win_arm64 everywhere the torch path applies, so a staged wheel was ignored. The Kaggle harness stops the Studio server before the CLI baseline regardless of skip_ui, so its own llama-server is not counted as a co-tenant. Tests cover each change and the free-threaded probe test inlines the helpers Test-WoaWheelAvailable now routes through. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Decide "public PyPI" by host, not by substring An exclusive index whose URL merely contains the name, such as https://pypi.org.corp.example/simple or .../api/pypi/pypi.org/simple, passed the substring test in both the PowerShell and the Python reachability probes, so the pyarrow route preferred a wheel the resolve could never fetch. Both now parse the URL and require the host to be exactly pypi.org. TestPyPI, which does not carry these packages, is therefore not public PyPI either. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rank uv's [pip] index settings above the top-level ones, and drop a redundant wheel's managed copy uv pip gives [pip].no-index and [pip].index-url precedence over their top-level twins, and an [[index]] entry with default = true beats [pip].index-url (verified on uv 0.10.7 with dry-run resolves). Both readers assigned whichever value came first in the file, so a uv.toml that set the top-level key ahead of the [pip] one could report public PyPI as reachable while the real resolve went to an exclusive index. Each value is now collected and ranked at the end, in both install.ps1 and install_python_stack.py. A wheelhouse wheel that PyPI now publishes was skipped at staging time but a copy already in the managed directory stayed there, and that directory is first in UV_FIND_LINKS, so it still won the tie: the offline-cache mode points the wheelhouse at that very directory, and a URL wheelhouse leaves the copy an earlier install staged. The managed copy is removed in both modes; an external wheelhouse file is never touched. test_woa_wheelhouse_guard_behaviour inlines every helper the reachability probe now calls, so a missing one no longer answers True from inside a non-terminating error. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Run the two uninstaller tests that no workflow ran cross-platform-parity-ci already triggered on uninstall*.ps1 and scripts/uninstall*.ps1, so editing the uninstaller queued a run that then asserted nothing about it. test_uninstall_prebuilt_parity.ps1 and test_uninstall_reparse_stop_roots.ps1 were referenced by no workflow at all. Both are hermetic -- they parse scripts/uninstall.ps1 with the AST and execute only the helper under test -- so they cost a few seconds on a row that is already running. Worth being precise about what this does not buy, because the gap is wider than the wiring. Neither test executes the uninstaller BODY, and neither does any other: every uninstaller test we have asserts source shape. A runtime parameter-binding fault is therefore invisible to all of them, which is exactly how `Split-Path -LiteralPath $p -Parent` reached users. It binds to no parameter set and throws on every uninstall, on 5.1 and on 7 alike, but it parses, so the board stayed green. The reparse test also could not run unprivileged. It created its home with New-Item -ItemType SymbolicLink, which needs elevation or Developer Mode, so it died on NewItemSymbolicLinkElevationRequired before the first subtree check. It now runs over both reparse kinds. A junction needs no privilege and is the shape a user is likelier to have -- mklink /J is what gets reached for when moving a studio home to another volume -- and _ManagedPathsUnderReparseTargets reads nothing but .Target, which both kinds supply rooted. A kind that cannot be created is reported as SKIP rather than passed over, and a final check fails the run if neither kind was exercised, so an environment that can make no links cannot report a clean pass having asserted nothing. * Read a uv config the way TOML is actually written Test-WoaResolveReachesPyPI decides whether PyPI is in the resolve at all, and a wrong "yes" is the expensive answer: it lets the redundancy guard delete the wheelhouse copy of a wheel nothing can then fetch, and makes Get-WoaPyarrowSource answer "pypi" and skip a usable local pyarrow, which gates the whole native path. The subset parser behind it answered "yes" for six shapes of config that had replaced or disabled PyPI. All six failed in that direction. Held against tomllib over 41 fixtures, the parser and a correct reading now agree everywhere; before this they disagreed on exactly these six: no-index = true# offline lab a comment needs no space in front index-url = "https://corp/simple"#corp same, on the value side "index-url" = "https://corp/simple" a quoted key is the same key pip.no-index = true a dotted key is [pip]'s no-index pip.index-url = "https://corp/simple" same [tool.uv] + pip.no-index = true same, in pyproject.toml The comment cut was `(^|\s)#`, which misses a `#` with nothing before it. That cannot be widened to a bare `#`, though, because the same line may carry one inside a string: `index-url = "https://host/simple#frag"` is a fragment, not a comment, and the old pattern got that right only by accident. Both halves need a scan that knows where the quotes are, so Remove-WoaTomlComment does that, and Split-WoaTomlKey does the matching job for keys -- unquoting them and splitting on dots, so `pip.no-index` is routed into the [pip] scope instead of being compared, whole, against the literal `no-index` and never matching. The two new helpers are added to the behaviour test's HELPERS tuple in the same commit. Leaving them out would have been the bug that tuple exists to prevent: a helper the prelude does not extract is a command-not-found inside Test-WoaResolveReachesPyPI, which aborts the statement, skips the branch, and answers True. Coverage was the reason these survived. The PowerShell side had no config-file fixtures at all -- test_pypi_counts_only_when_the_resolve_would_reach_it drives environment variables only -- while the Python twin has had file fixtures and a real tomllib all along. Sixteen cases now cover both directions of the scan, the [pip] precedence both ways, a lookalike host, and the shapes that must keep answering True so this cannot be mistaken for a blanket "not reachable". * Put the uv user config where uv would look for it on Windows The user-file row of test_a_configured_exclusive_source_is_not_pypi failed on Windows and always would have. The fixture deletes APPDATA and then points only XDG_CONFIG_HOME at a directory, but _uv_config_files reads %APPDATA%\uv\uv.toml on Windows and $XDG_CONFIG_HOME/uv/uv.toml elsewhere, so on Windows the file was written somewhere uv never looks, nothing was discovered, and the assertion that a user-level no-index takes PyPI out of the resolve could not hold. It stayed green because tests/python runs on the Linux row, where XDG is the right answer. Both names now point at one directory, so the case is real on every platform. test_project_outranks_user_for_a_scalar needed the same treatment for a sharper reason: it wrote the user file to the old xdg path, which after this change is not a config location anywhere, so it would have passed on the project file alone while asserting nothing about precedence. It now writes to the same place and first asserts the file is one _uv_config_files discovers, because a precedence test whose lower-ranked file was never found is a test that cannot fail. * Don't strand an old install behind the new ownership gate bcc0ffd stopped the uninstaller recursively deleting ~/.unsloth/studio unless one of install.ps1's sentinels says the directory is ours. That was right, and the list it used was too short. On Windows share\studio.conf is never written -- only install.sh writes it -- so the three sentinels that actually decide a Windows root all postdate the bin\ shim dir and the unsloth_studio rename. An install from before those has neither: the venv lived at <root>\.venv, the venv's Scripts dir went on PATH in place of a shim dir, and .unsloth-studio-owned did not exist yet. install.ps1 still migrates exactly that layout, at "found legacy Unsloth environment", so it is not hypothetical. Uninstalling one refused, printed the user's own install as a non-Unsloth path, and left the tree and studio.db on disk. tests/sh/test_uninstall_arg_guard.sh has been asserting this all along -- its fixture is a pre-marker unsloth_studio venv and "no arguments removes the install" fails at eda3801. So accept two more things Unsloth put there: the owner marker inside the legacy .venv, and either venv dir carrying the unsloth console script pip generates for the CLI. That is the property the gate is testing. A bare .venv, a venv merely named unsloth_studio and a hand-made studio directory still have none of them and are still refused. (cherry picked from commit 05c6cdf9a23273bb7a5e9fe4900b59dbe6cda3a1) * Recognise the abi3 pyarrow upstream is going to publish pyarrow is the mandatory gate for the whole native path: staging turns the chosen wheel into an exact `pyarrow==` override, so whichever file we select is what the resolve gets. Today that is our own 24.0.0.dev260, a pre-release, while upstream stable is 25.0.1 -- acceptable only because PyPI publishes no win_arm64 pyarrow at all yet. It will. apache/arrow#48539 adds the wheel and is held behind apache/arrow#50398, whose plan is an abi3 floor of 3.11, so the first one on PyPI will be named pyarrow-<v>-cp311-abi3-win_arm64.whl. Every pyarrow candidate went through Test-WoaPyarrowWheelUsable, which gated on Test-WoaWheelTags -- exact PEP 425 fields -- and an exact test answers false for cp311-abi3 on every interpreter we build. The day upstream shipped, we would have gone on selecting our own dev build and pinning it, and nothing anywhere would have said so. Silent, permanent, and pointing the wrong way for a PR whose whole aim is that a new user on this hardware installs from upstream wheels. Test-WoaWheelTagsUsable is the predicate that already answers this correctly, and it keeps both properties exactness was buying: an abi3 wheel built against a NEWER interpreter than the venv is still refused, because abi3 reaches forward from its build version and not backward, and abi3 is refused outright on free-threaded builds, which have no stable ABI (CPython #111506). One function changed, because all six call sites already funnel through it -- the supplied wheel, the PyPI scan, the wheelhouse directory and index, and the two in staging. test_every_pyarrow_candidate_goes_through_the_floor counts them and still reads 7. Four cases cover the shape upstream will ship and the two directions it must not loosen, and a fifth asserts the free-threaded refusal on its own. The preludes that lift Test-WoaPyarrowWheelUsable out of install.ps1 gained Test-WoaWheelTagsUsable in the same commit, for the reason that keeps recurring in this file: a helper the prelude does not lift is a command-not-found, which aborts the statement rather than answering, and the caller then reports success. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows on ARM: make the purge and merge tests speak the host's path separator These tests are gated on pwsh being present, not on the platform, so on a Windows box with PowerShell installed they run and 11 of them fail. None of it is the product: the purge builds its owned prefix with Join-Path, which uses the host separator, and the tests hand it a hardcoded POSIX home, so "C:\...\unsloth\woa" was compared against ".../unsloth/woa" and never matched. In a real run both sides come from the same Join-Path, so the purge is correct either way. Build the fake home and the entries the way the host does. The same cases now cover the separator the machine actually uses, and on POSIX the values are byte-identical to before. The two merge tests failed for a second reason: their fakepython stub is a shebang script, which Windows cannot execute, so the merge got no pins back and the failure was an empty path rather than anything about rebasing. Write a .cmd stub there instead; PowerShell's & runs it and the stub ignores -c regardless. 473 passed, 0 skipped on win-arm64 with PowerShell 5.1. (cherry picked from commit 8dc13a7692c325408a9cb56eb94b612e71f45a2f) * Test the Windows on ARM install path on Windows on ARM Everything this repo knows about win_arm64 has been tested only on x64 runners: ubuntu-latest and windows-latest. That is the wrong machine for all of it, and it is why several real bugs sat behind a green board. Three of them could not have failed anywhere else. The purge builds its owned prefix with Join-Path, which uses the HOST separator, so a test hardcoding a POSIX home compared "C:\...\woa" against ".../woa" and never matched -- eleven tests passed on Linux and failed on Windows for a reason that had nothing to do with the purge. The uv config parser, which exists because Windows PowerShell 5.1 ships no TOML reader, read six shapes of config wrongly, every one of them answering "PyPI is reachable" for a config that had taken PyPI out of the resolve. And the uninstaller's ownership gate refused genuinely old installs, leaving the user's tree and studio.db on disk while calling their own install a non-Unsloth path. windows-11-arm has been free for public repositories since 2025-08-07, and this leg is cheap because none of this needs a GPU, CUDA or a compiler: it is arithmetic over wheel tags, path handling and PowerShell parsing. A full install or a Tauri build needs much more and is deliberately not attempted here. Two rows, because the shells are not interchangeable and each is what something real uses. pwsh 7 is what CI has always run. Windows PowerShell 5.1 is what the CLI launches setup.ps1 with, and it is the only shell on a stock Windows box, so a developer on this hardware runs 5.1 whether or not anyone planned for it. cross-platform-parity-ci already keeps a 5.1 row for #8335 on the same reasoning. The suites locate their shell with shutil.which("pwsh"), which 5.1 does not provide, so that row puts a copy of powershell.exe on PATH under that name -- without it the row would SKIP everything and report green having run nothing. A copy rather than a .cmd shim, because CreateProcess will not launch a .cmd without a shell, and Python is what starts them. Both architectures are asserted rather than assumed. A leg that silently ran on x64 would be worse than no leg, since it would report green for precisely the platform it does not cover, and setup-python needs an explicit architecture: arm64 because the image ships an x64 Python that runs under emulation and reports win-amd64 from sysconfig -- half of these assertions are about wheel tags, so an emulated interpreter would quietly test the wrong platform. A third check pins that the shell resolved for the row is really the major version that row is for. Dry run of this exact selection on win_arm64 under PowerShell 5.1: 671 passed, 0 skipped. One note recorded in the workflow for later: GitHub moves the windows-11-arm image to Visual Studio 2026 between 2026-09-21 and 2026-09-30. This job invokes no compiler so it cannot be affected, but a leg that builds studio/src-tauri would be, and would also need clang on PATH for ring and aws-lc-sys. * Only ask for a junction where junctions exist My own regression, caught by the parity job the same commit wired these tests into: parity (ubuntu-latest) went red on FAIL Junction : a linked root yields the sd.cpp tree under its physical target FAIL Junction : ... and the venv under it A junction is a Windows-only reparse type, and New-Item -ItemType Junction does not throw on Linux pwsh. It quietly produces a plain directory with no .Target, so the SKIP guard -- which only catches a failure -- never fired, and the row ran and failed every positive assertion. The negative ones passed for the wrong reason: _ManagedPathsUnderReparseTargets returns nothing for a path with no target, which is also what "the bare physical target is NOT in scope" wants. Choose the kinds by platform instead of by catching an exception that never comes, and treat $IsWindows being undefined as Windows, since that is Windows PowerShell 5.1 and 5.1 runs nowhere else. Also verify the link after creating it. Created is not the same as usable: the helper reads nothing but .Target, so a link without one cannot exercise anything, and reporting that as a failure blames the helper for the filesystem's answer. A kind that produces no target is now a SKIP, and the existing check that at least one kind ran still fails a run that exercised nothing. Windows unprivileged: Junction all pass, SymbolicLink SKIP (needs elevation), overall pass. Linux keeps the SymbolicLink row, which passed on that job already. * Assert the uninstaller's ownership gate on POSIX too The gate that decides whether ~/.unsloth/studio is recursively deleted exists in both uninstallers, and when it regressed it regressed in both. Verified on real Linux (aarch64, bash 5.2.21): against scripts/uninstall.sh before the fix, three layouts Unsloth itself created were refused -- FAIL: legacy .venv carrying the owner marker (got foreign, want own) FAIL: pre-marker unsloth_studio venv (got foreign, want own) FAIL: pre-marker legacy .venv (got foreign, want own) -- and against the current one all thirteen checks pass. So this was never a Windows problem: a Linux or macOS user with an install predating the unsloth_studio rename was told their own install was a non-Unsloth path and left with the tree and studio.db on disk. There was no POSIX test for it. tests/studio/test_uninstall_legacy_layout_gate.ps1 covers the Windows side; this is its twin, following the extract-via-sed pattern of test_uninstall_shared_icon.sh so the uninstaller body, which kills processes and deletes trees, is never executed. Both directions are asserted, because a gate is only as good as what it refuses. The negative cases include a venv that merely carries an unrelated console script, so accepting bin/unsloth cannot quietly become "any venv with anything in it", and a bare .venv, which is the case the gate was added for and which stays refused because it is genuinely indistinguishable from a user's own project venv. Wired into the parity workflow's Linux row. Worth noting how it went unnoticed: the shell uninstaller tests are named individually in workflows and four of them are named in none, while consolidated-tests-ci excludes tests/sh outright. A test nobody runs is indistinguishable from a test that passes. * Cut the comment volume in the Windows on ARM installer * Cut the comment volume in the studio setup script * Cut the comment volume in the python installers and the CI workflows * Cut the comment volume in the uninstaller and its tests * Cut the comment volume in the tests, requirements and tauri sources * Second comment reduction sweep across the Windows on ARM changes * Third comment reduction sweep over the Windows on ARM changes * Rebase editable and bare local-directory override lines too New-UnslothTorchOverridesFile used to write its merge beside the caller's UV_OVERRIDE file when there was exactly one, so relative references in it kept resolving. The merge moved to %TEMP% with per-line rebasing, which also fixes the multi-directory case that was always broken, but Resolve-WoaOverrideLine did not know -e/--editable or a bare ./pkg directory requirement, so those two forms started pointing at nothing. Reached every Windows host with UV_OVERRIDE set, not only Windows on ARM. * Fold 64 copies of the PowerShell invocation into one helper Every test that runs a lifted PowerShell function spelled out the same five-line subprocess.run and the same `assert done.returncode == 0, done.stderr`, 64 times. _ps and _ps_ok replace them, and the timeout stays a parameter because five sites need 180 and one needs 60. 369 lines out of the file, no test added or removed: 502 collected before and after. Mechanically substituted, then checked the tests still catch what they are for, by breaking install.ps1 three ways and comparing which tests fail. The failing node ID sets are identical before and after (27 for a Split-WoaTomlKey that never splits, 8 for a Test-WoaWheelTagsUsable that always says yes), so the tests still execute PowerShell rather than passing vacuously. * Test the default wheelhouse URL, which nothing tested Found by mutation: changing the default to a working but wrong host left all 1492 tests passing. It is load bearing twice over. It is the only source for the pyarrow that gates the native path, so a typo silently sends every Windows on ARM host back to the emulated x64 stack with no error anywhere, and it is fetched over the network, so a wrong host is a wrong download. Pins the literal, that it is an https resolve/main URL under our own org rather than a repo page that would serve HTML the staging code saves as a .whl, and runs the assignment for eight override shapes: unset, empty, a mirror, one and several trailing slashes, surrounding whitespace, and a local directory whose trailing backslash TrimEnd('/') must not touch. The same mutation now fails 3 tests instead of 0. * Keep windows-vulkan in the ARM64 kind lists main just widened main added windows-vulkan to every branch of the llama.cpp kind check, because any x64 Windows host can land on a Vulkan bundle and a guard without it deletes a working install on every setup run. That reasoning holds for the ARM64 arms this branch adds, so it goes in $_nvidiaKinds rather than being appended at the one call site: the opted-out arm needs it as much as the CUDA one. Test expectations and the source-slice anchors move with it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Move the uninstaller fixes out to their own PR The ownership gate, the ~/.unsloth/studio guard, the Split-Path call that throws and the remove-retry budget are not about Windows on ARM, and two of them change behaviour on Linux and macOS. Reviewing them as a footnote to a Windows installer feature was the wrong shape. They are now #10471, against main, with the parity CI wiring that runs them. These files go back to main exactly. Also drops this workflow's references to test_uninstall_legacy_layout_gate.ps1, which now lives in that PR; the runner loop already skips a test file it cannot find, so it can be listed again once #10471 lands. * Move the Kaggle GPU harness fix out to its own PR An all-[N/A] compute-apps listing means nvidia-smi cannot attribute memory, not that the card is idle, which is how WDDM and unified-memory parts report. That is a GPU attribution fix in the test harness and touches no installer code, so it reviews better on its own: #10472. These files go back to main exactly. * Read the two installer scripts once, not 117 times 62 tests re-read install.ps1 and 55 re-read setup.ps1, each on its own line, and none of them mutate what they get back. Two module constants replace all of it, and _ps_function serves from them rather than opening the file again on each of its 60 calls. The wheelhouse-tag tests get the same treatment for sys.version_info and the cp tag derived from it, restated in nearly every test. 130 lines out, no test added or removed: 1502 collected before and after. Substituted on NAME tokens through tokenize, never on raw text, so an occurrence of "text" inside a PowerShell string literal could not be caught by it. Verified by mutation, not by the suite going green: breaking install.ps1 three ways fails exactly the same node IDs as before, 3 for the default wheelhouse URL, 27 for a Split-WoaTomlKey that never splits, 8 for a Test-WoaWheelTagsUsable that always says yes. Formatted with scripts/run_ruff_format.py and confirmed idempotent. * Lower the win_arm64 floors to the first release that actually has a wheel Three floors were set one release too high, and each one excludes a wheel that exists. scikit-learn is the one that bites. 1.8.0 publishes six win_arm64 wheels including cp313-cp313t; 1.9.0 publishes five and dropped the free-threaded one. So >=1.9.0 leaves a free-threaded 3.13 with no binary candidate and an sdist to compile, which is what the ARM64 rows exist to avoid. The comment two lines up already said "from 1.8 only" and contradicted its own pin. av first ships win_arm64 at 17.0.0 (cp311-abi3, plus a cp314t), not 17.1. No release has a 3.13t wheel, so that half of the comment now says so. bitsandbytes is a comment fix only: the wheel starts at 0.50.0, and the pin was already below both. tests/studio/install/test_diffusers_pin.py pins all three against the index rather than against an audit, and rejects a second ARM64 row for the same project so a future edit cannot leave two floors disagreeing. It fails on each of the three old floors. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Translate every spelling of --prerelease for the pip fallback, not one Remove-UvOnlyResolverFlags handled --prerelease two ways, and both were wrong away from the single spelling the two call sites use. The space-separated form dropped the flag and its value and emitted nothing, so a pip fallback given `--prerelease allow` resolved without prereleases and could not see the torch build the run had just asked for. uv accepts that form everywhere it accepts the = form. The = form emitted --pre for any value. Of uv's five, only allow means what pip's --pre means: disallow is pip's default and if-necessary, explicit and if-necessary-or-explicit have no pip flag at all. So --prerelease=disallow became --pre and inverted the caller's intent. Both are latent today, since install.ps1 and setup.ps1 both pass --prerelease=allow, which is the one case the old code got right. The helper takes a caller-supplied argument list, so the grammar it accepts should be uv's. The test walks ten argument lists through the real function: both spellings of both flags, four --prerelease values, a bare value that must not be swallowed by a flag it does not belong to, and the two flags combined. Four fail on the old implementation. * Stop the llama.cpp gate deleting what the selector just installed on ARM64 setup.ps1 deletes an existing llama.cpp tree whose recorded install_kind is not in a set it computes for the host, then reinstalls via install_llama_prebuilt.py. Nothing made those two agree, and where they do not the disagreement is permanent: every update deletes the tree and refetches the identical bundle, forever, on a machine that is working. Walking the product of what either side branches on finds two disagreements on main, both only reachable from Windows ARM64, which is why they lasted: arm64 + nvidia selector installs windows-arm64, gate expected windows-cuda arm64 + rocm selector installs windows-arm64, gate expected windows-rocm The first is every Windows ARM64 machine with an NVIDIA GPU, and this branch already fixed it by widening the gate to the ARM64 kinds. The second is fixed here. No ROCm bundle exists for Windows ARM64 at all: upstream ships hip-radeon-x64 and we publish none, so the selector falls through to the ARM64 CPU bundle while the gate went on expecting windows-rocm or windows-hip. The two ROCm kinds stay in the set for the day one is published. tests/studio/install/test_llama_kind_gate_matches_selector.py asserts the only relationship that matters between the two sets, over the twelve host combinations, rather than the text of either. Offline: direct_upstream_release_plan takes a release dict, and the gate's own block is lifted out of setup.ps1 with its inputs replaced. It fails 3 checks against the gate before this commit. While confirming this, the x64-fallback churn loop I had noted earlier turned out not to exist. An emulated x64 Python on an ARM64 box reports AMD64, so that venv is an x64 host to the selector, it installs windows-cuda, and the gate expects windows-cuda. All four x64 rows agree, before and after. * Fold the repeated script and source boilerplate in the WoA persistence tests Four whole installer and stack files were still being re-read per test; they are read once now. The 62 copies of the join-a-list-of-lines idiom and the 48 run-then-take-the-last-line pairs become _script and _ps_last, and the three copies of the parity normaliser become one _ps_copies helper. * Give the wheelhouse tag tests one cache fixture and one read of each source The find-links listing was cleared by hand in eleven try/finally blocks and a dozen bookend pairs; an autouse fixture now empties it before and after every test, and the clears that sit between two states inside a test stay. install.ps1, install_python_stack.py and extras.txt are read once, and the wheelhouse builder becomes a fixture rather than a helper taking three fixtures per call. * Read each source file once in the marker and guard behaviour tests install_python_stack.py was exec'd twice in the marker tests and install.ps1 was re-read and re-scanned for ten helper bodies on every one of the guard tests' thirty-five pwsh calls. * Use the hoisted source constants directly rather than aliasing them Sixteen tests opened with a one line rebinding of a module constant to a shorter local name; the constant is now named at the point of use. * Let the prerelease gate test tell producing the flag from comparing to it Remove-UvOnlyResolverFlags now names --prerelease=allow to translate it into pip's --pre. That is not a call site adding the flag, it runs after the gate has already allowed it, so the array-shape rule the gate test enforces does not apply to it. * Verify the Windows ARM64 CUDA bundle against GitHub's asset digest This was the one archive the installer would install unchecked. Every other prebuilt goes through apply_approved_hashes, which drops an attempt our own checksum manifest does not cover. The ARM64 CUDA branch could not: we publish no windows-arm64-cuda artifact yet, so nothing covers upstream's zip, and refusing outright would mean no CUDA llama.cpp on this hardware at all. So it logged that it was installing without a hash, and did. GitHub's release API now reports a per-asset digest. Checked against ggml-org/llama.cpp b10853: 27 of 27 assets carry one, including llama-b10853-bin-win-cuda-13.4-arm64.zip and the cudart archive it pairs with. That is weaker than our manifest, which we compute ourselves rather than read from the same host that serves the bytes, and the comment says so. It still pins the download to what the API listed, which is enough for this branch to stop being the exception. An asset GitHub states no digest for is now refused and the install falls through to the ARM64 CPU bundle, so no path installs an unverified archive. Three things kept deliberately narrow: * A separate github_release_asset_digests rather than a second return value from github_release_assets. That one is mocked in a dozen suites and used on every platform; this is read on one branch. * Any failure fetching the digests is an empty mapping, which that branch reads as refuse. A rate limit costs the CUDA bundle, not the install, and cannot be mistaken for "no digest needed". * release_asset_digests requires the sha256: prefix rather than stripping it for convenience. normalize_sha256_digest accepts a bare 64-hex string, and an unprefixed digest is one whose algorithm GitHub did not state. tests/studio/install/test_arm64_cuda_digest_verification.py covers the digest parser over seven rejected shapes, the drop and unpair rules, the fail-closed fetcher, and asserts against the source that every return in the resolver is hash gated. That last one fails on the branch before this commit, on the literal `return upstream_arm64_cuda`. 143 failures before and after across the four llama selection suites, identical sets, so nothing here changes any other platform's routing. * Bound the early GPU probes, gate the wheel on the driver, and tighten the resolver and index checks install.ps1 - Test-WoaNvidiaPresent and the new Get-WoaDriverCudaVersion run nvidia-smi through the bounded runner. - A CUDA wheel whose major exceeds the driver's falls back to the x64 stack with a driver-update message. - The av probe is gated on the resolver reaching PyPI; public PyPI supplied as an extra index (env or uv config) keeps the probe in play. - A managed wheel is redundant only when PyPI has the exact version, and the managed scan lifts win_arm64 or any wheels only. - On an equal numeric release a final build outranks a .dev one; later stamps win among dev builds. - The exact CUDA trio pin ignores UV_EXCLUDE_NEWER for that one install, since the index carries no upload dates. - A pyarrow staging failure after the venv exists is a hard stop with the opt-out named. - Rebased file: references become file URIs, so spaces are encoded. studio/setup.ps1 - The merged override file is recorded, removed at the end of the run and on failure, and any stale copy is removed first. - WinArm64NoAudio follows the effective-index probe; retention pairs torchaudio with the new torch by both predicates. - Same tie-break, file URI and exclude-newer handling as the installer. studio/install_python_stack.py, install_manifest.py, install_llama_prebuilt.py - Extra indexes that are public PyPI count; the resident xformers check runs after a failed refresh. - The NVIDIA index is persisted with scheme and host normalised. - Legacy bundle ordering synthesises names for the arch being resolved and passes it through. .github/workflows/windows-arm64-ci.yml - The ARM job fails when a required test is reported as skipped. Tests cover each change, including a real Exit-SetupFailure run under UNSLOTH_TAURI_MODE. * Make a paired torchvision part of the native gate, and read a supplied wheel's name from its archive install.ps1 - An index qualifies for the native path only when it publishes a torchvision paired with its torch; otherwise the next candidate is tried, and with none pairing the x64 stack is kept with the reason logged. The vision pin is the version the gate accepted. - UNSLOTH_PYARROW_WHEEL saved under another name (.bin, no extension) has its wheel name read from the archive before the project and tag checks, as staging already does. studio/setup.ps1 - When the effective index pairs no torchvision with its newest torch, the fresh-shell probe keeps the installed torch and torchvision pair instead of pinning a torch whose torchvision would resolve against another build. Tests cover the gate order, the x64 fallback, the kept pair, and archives named .bin or without an extension. * Rank prereleases in PEP 440 order, probe a reselected interpreter, and put the caller's resolver variables back install.ps1 - Within one numeric release the wheel pick follows PEP 440 order in full: dev below a below b below rc below the final build, later numbers winning within a kind. Same in setup.ps1's copy. - After the re-probe flips native mode, the reselected interpreter is probed before it is taken; one with no stack of its own is declined and the accepted interpreter keeps its answer. - UV_OVERRIDE, UV_FIND_LINKS and PIP_FIND_LINKS are snapshotted before the first assignment and restored in the script-level finally, so an `irm | iex` session does not keep resolving with Studio's overrides and wheelhouse. studio/backend/plugins/data-designer-unstructured-seed - The plugin admits pandas 3, which constraints.txt installs on Windows on ARM; it only reads and writes parquet. Tests cover the ordering, the reselection, the restore, and the plugin range against the constraints. * Format the three files this branch owns with the ruff the hooks actually pin .pre-commit-config.yaml pins ruff v0.15.18. I had been running scripts/run_ruff_format.py against a local ruff 0.6.9, and the two disagree about where an assert's message goes: 0.6.9 assert ( cond ), "message" 0.15.18 assert cond, ( "message" ) So every local format pass wrote the older shape and pre-commit.ci rewrote it on push. This puts the three files that are ours at the pinned formatter's fixed point instead. Verified rather than assumed: with 0.15.18, running the wrapper twice leaves every file byte-identical, so the wrapper is idempotent and it was only the version that was wrong. Four more files still differ under 0.15.18 and are left alone, because they differ identically on origin/main and so are not this branch's drift: studio/prebuilt_core.py, tests/studio/install/test_cuda_repair.py, tests/studio/install/test_rdna1_unsupported_message_8529.py and tests/studio/install/test_windows_torch_flavor_invariant.py. 855 tests pass across the three. * Let the dependency index follow the resolver policy, widen the ARM CI filters, and survive an upstream listing failure install.ps1 and studio/setup.ps1 - torch's shared dependencies resolve from the index the caller's resolver policy names (environment, then uv config), from public PyPI when it names none, and from the find-links wheelhouse alone under no-index. Invoke-InstallCommand clears the inherited index settings whenever --default-index is passed, so the policy is restated on the command line. setup.ps1 carries the readers as parity copies. .github/workflows/windows-arm64-ci.yml - The path filters name pyproject.toml and the whole requirements tree the ARM tests read. studio/install_llama_prebuilt.py - A failure listing the upstream release assets on a Windows ARM64 NVIDIA host costs the CUDA bundle only; the planner falls through to the published ARM64 CPU bundle. Tests cover the policy readers in both scripts, the filters, and the fall-through. * Judge index reachability and the dependency index for the resolver that runs the pass uv reads UV_* and its configuration files and ignores PIP_*; pip reads PIP_* alone. Reading both sets together reported public PyPI reachable from a PIP_EXTRA_INDEX_URL that uv, the resolver in use, never consults, and then dropped the librosa and tensorboard skip on wheels the resolve could not fetch. - studio/install_python_stack.py: the probe branches on USE_UV; the uv path also honours UV_NO_INDEX. - install.ps1: the probe reads uv's policy only, since this script resolves with uv; Get-WoaDependencyIndexArgs takes the resolver and reads that resolver's variables, uv's configuration files for uv alone. - studio/setup.ps1: the parity copy, with the resolver it will use passed in. Tests cover the cross-resolver case in all three places and the pip fallback's own policy. * Keep markers out of rebased file references, clean the merged overrides on every exit, and stop on an unwritable wheel directory install.ps1 and studio/setup.ps1 - A PEP 508 marker on a `pkg @ file:...` override is kept aside while the path is rebased and re-attached after the URI. - setup.ps1 traps a terminating error after the resolver restore, removes the merged override file, and rethrows. install.ps1 - A wheel directory that cannot be created after the ARM64 venv exists is a stop with the reason, not a stand-down into an index with no win_arm64 wheel. The directory is verified after New-Item, which can return without creating it when a file is in the way. - The trio command names the staged wheelhouse with --find-links, since the inherited UV_FIND_LINKS is cleared for a pinned index; UV_NO_INDEX yields for that one command and is restored after it. - The two probe warnings that print an index URL redact its credentials. studio/backend/requirements/no-torch-runtime.txt - sqlite-vec carries the negative Windows ARM64 marker, as it does in studio.txt. Tests cover each change; the marker table claims the new row. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the five Windows-on-ARM tests actually run on Windows All five assert Windows-on-ARM behaviour and none of them could execute their own fake on Windows, so they failed on the one platform they are about while passing on Linux. windows-arm64-ci.yml runs this file on a win-arm64 runner under both shells and asserts sysconfig.get_platform() first, so these were red, not skipped. Test-WoaNvidiaPresent. The fake was an extensionless file containing #!/bin/sh. Windows has no shebang handling and PATHEXT covers no extensionless name, so the probe found nothing at all. Now written as nvidia-smi.cmd on Windows, which is the idiom the rest of this suite already uses, and the sh script elsewhere. The hung-probe case shared the same fake and so was a false pass: it asserted the call returns, and a probe that finds no executable returns immediately. It now actually blocks, via ping rather than timeout, since timeout /t needs a console. The installed-pair probe. The block runs <VenvDir>/Scripts/python.exe by that exact name, and the fake was a shell script called python.exe, which Windows will not execute because it is not a PE. All three cases therefore asserted the floor rather than the branch they were written for. Replaced with a real venv carrying real .dist-info metadata, so importlib.metadata answers the way it would after a real install and the interpreter answers the tag query itself. Confirmed it now bites: breaking the installed-pair read fails test_the_installed_pair_is_kept, which it could not do before. The file: URL row. Its expectation was built by pasting a rebased path after "file://". A rebased Windows path is C:\opt\corp\ov\dist\a.whl, and "file://" plus that is not a file URL: the authority would be C: and the separators are wrong. The rewriter emits file:///C:/opt/corp/ov/dist/a.whl, which is correct, so the code was right and the expectation was not. Split into its own test that builds the URL with pathlib as_uri instead of concatenating. 26 pass here on native Windows ARM64 under PowerShell 5.1. * Add soxr to librosa's unblock set: it has no win_arm64 wheel either librosa is on the win_arm64 skip list and comes off it once its blockers are installable. The blocker set was llvmlite and numba, but librosa 0.11.0, which is what extras.txt pins, requires soxr>=0.3.2 as well, and soxr has published no win_arm64 wheel in any release up to 1.1.0. Verified against the PyPI JSON API for the pinned version rather than for latest, since only the pinned release's metadata is the one that applies. So hosting cp313 numba and llvmlite would have un-skipped librosa, and the extras pass would then have gone looking for soxr and found only an sdist. Building that needs the toolchain the whole skip list exists to avoid. Nobody has hit it because numba and llvmlite publish win_arm64 for cp314 only, so librosa has never actually un-skipped on a cp313 host, which is what kept it latent. soxr also gets a floor entry, read from librosa 0.11.0's own metadata, matching the provenance convention the other two floors already follow. openai-whisper is left alone deliberately: its metadata asks for numba and tiktoken and does not mention soxr. Upstream is dofuuz/python-soxr PR 36, "Add Windows ARM64 wheels", still open. The three existing tests that drove librosa off the skip list now stub the index answer for soxr, so they stay about what they were written for: the interpreter tag deciding availability, the empty-wheelhouse early return, and losing the index offline. The new truth gets its own test, which fails when soxr is taken back out of the tuple. 165 pass. * Name the wheelhouse on the setup.ps1 trio command, read pip's config files, evict a stale xformers without a hosted wheel for PR #10282 setup.ps1: Fast-Install clears UV_FIND_LINKS and PIP_FIND_LINKS beside the other inherited index settings whenever --index-url is given and leaves UV_NO_INDEX alone, so a native CUDA update under --no-index saw neither the staged wheelhouse nor the CUDA index. $WinArm64IndexArgs now carries --find-links for the wheelhouse when the directory exists, and UV_NO_INDEX yields for the trio command through the existing cutoff save and restore, as install.ps1 already does. install_python_stack.py: the pip path of the PyPI reachability check read PIP_* only. It now consults `pip config list` when the environment is silent: an `[install]` or `[global]` no-index or exclusive index-url set in pip's files means PyPI is not in play, a PyPI extra-index-url keeps it in play, and a pip config that cannot be read keeps the skip. :env: rows are skipped since the variables were already read. install_python_stack.py: the resident xformers compatibility check ran only after a hosted install attempt, so a copy an earlier run installed against another torch stayed resident whenever the wheelhouse stopped offering a usable wheel. The check is now a helper that also runs when nothing usable is hosted. Tests: setup.ps1 twin of the no-index trio class (find-links on the command line, UV_NO_INDEX yield and restore, off-ARM64 untouched), the pip fallback rows carry --find-links through the pip translation, a pip configuration class with stubbed and real `pip config list` output, and the wheelhouse optional rows for eviction without a hosted wheel. Twelve mutants on the three changes are killed. * Search every supported nvidia-smi location for the driver version Test-WoaNvidiaPresent searched PATH, System32 and the NVSMI folder, but Get-WoaDriverCudaVersion searched PATH and System32 only. On a host that carries nvidia-smi.exe under %ProgramFiles%\NVIDIA Corporation\NVSMI alone, the presence probe reported a GPU and the version probe returned nothing, so Initialize-WoaNativeCudaTorch skipped its CUDA-major compatibility guard and could select a cu13x ARM64 wheel against a driver that only advertises CUDA 12. Factor the candidate list into Get-WoaNvidiaSmiPath and have both probes call it, so the two cannot disagree. Same two locations the main GPU detection already searches. * Inject Get-WoaNvidiaSmiPath into the scripts that paste the probes' real bodies The pwsh tests compose real function bodies into a bare script. Routing both nvidia-smi probes through the shared lookup gave those bodies a call the compositions did not inject, leaving it unresolved. One of the two would have failed loudly. The other passes for the wrong reason: PowerShell writes the unresolved-command error and continues with $null, so Get-WoaDriverCudaVersion returns $null before it ever calls nvidia-smi and the '[]' assertion still holds, with the timeout the test exists to bound never exercised. Both are marked requires_pwsh, so on a host without pwsh they skip and the suite is green either way. Add a guard that needs no pwsh: it fails if a composition injects a body calling Get-WoaNvidiaSmiPath without also injecting the helper. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read uv's inline index spelling, and stop guessing when it cannot be read Read-WoaUvTomlIndexKeys returned $null on sight of an `index` key, so `index = [{ url = "...", default = true }]` -- valid, documented uv config -- read as Unreadable. Get-WoaDependencyIndexArgs never looked at Unreadable and fell through to its https://pypi.org/simple default, silently replacing whatever mirror the file configured. Test-WoaResolveReachesPyPI had honoured the same flag all along, so the two consumers of one signal disagreed, and the unsafe one built the install command. Parse the inline form, conservatively: a single-line, brace-balanced array of flat inline tables only. A multi-line array, a nested value, a bare entry beside a table, a missing url, a non-boolean default or an explicit index all still return $null and stay Unreadable, because guessing is the only outcome worse than not parsing. Then make the consumers agree, per call site, since they genuinely differ. With no env default and an unreadable policy, name no dependency index rather than substituting PyPI. install.ps1's trio step calls uv directly, so uv reads the file itself and the empty answer is strictly better than today: no substitution and no failure. setup.ps1's runs under Fast-Install with --index-url, which sets UV_NO_CONFIG and scrubs UV_*, so uv cannot read it and an empty answer would leave the trio index as the only source; that one stops with a message naming the file, the spelling, and both ways out. Get-WoaUvConfigIndexPolicy now records which file it could not read so the message can name it. Blast radius, traced before writing: a previously-$null read now returns a value, which moves Test-WoaResolveReachesPyPI and therefore wheel pruning. A mirror URL still gives false and still disables pruning, so that case is unmoved. A genuine pypi.org URL now gives true and enables it, which is correct and was previously suppressed by a parse failure rather than by a decision. Verified against pwsh 7.4.6 rather than skipped: the inline reader is exercised on five accepted shapes and six ambiguous ones. Running them for real caught a bare string entry beside a table being accepted, because the scanner entered string mode before testing depth. Full tests/studio/install plus the wheelhouse tags suite gives an identical failure set to HEAD, 173 either side, none new. * Apply the pinned formatter's kwarg spacing to the new inline-index cases The ruff-format-with-kwargs hook pins ruff==0.6.9, which wraps the new parametrize tuples one element per line. Formatting only; the 733 tests in the two files are unchanged and still pass, and a second run of the hook is a no-op. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Default the Windows on ARM wheelhouse to the published one for PR #10282 The default named unsloth/windows-arm64-wheels, which does not exist, so no host took the native path without UNSLOTH_WOA_WHEELHOUSE set. The wheelhouse now lives under danielhanchen/unsloth-blackwell-docker: pyarrow 25.0.1 for cp311 through cp314, built from the Arrow release tag with apache/arrow#48539's build script on a windows-11-arm runner, and sqlite-vec 0.1.9 repacked with an ARM64 DLL built from its release tag. Every wheel was smoke tested on the ARM64 runner, every native image inside is ARM64, and install.ps1 ran its native path against the published URL end to end (pyarrow 25.0.1, torch 2.14.0+cu134, sqlite_vec 0.1.9 all importing in the win-arm64 venv). The test that pins the URL follows. * Share the PowerShell test harness and parametrize the Windows on ARM persistence tests for PR #10282 * Shorten the comments the Windows on ARM installer added for PR #10282 * Parametrize and share fixtures across the Windows on ARM wheelhouse tests for PR #10282 * Keep explicit uv indexes out of the dependency index, prune wheels an earlier wheelhouse left for PR #10282 Explicit indexes: uv serves an index marked explicit = true only to packages pinned to it via [tool.uv.sources], so it is neither the default nor an extra for the trio's shared dependencies. The block form [[index]] reader in install.ps1 and setup.ps1 read url and default only and flushed such an entry as a general extra, which Get-WoaDependencyIndexArgs then passed as --extra-index-url to the native torch resolve; the inline table reader refused the key outright, which turned the documented PyTorch pattern into an unreadable policy. Both readers and the Python reader now skip an explicit entry; explicit together with default = true also removes PyPI as the default, which is not modelled, so that combination stays doubt. Stale wheels: the managed woa\wheels directory persists across runs, so after UNSLOTH_WOA_WHEELHOUSE changed a wheel the earlier wheelhouse staged stayed in it, the hosted scan read it as current and UV_FIND_LINKS installed it from a source no longer configured. Staging now reconciles the directory against the wheelhouse listing it just read, keeping the listed wheels and the selected pyarrow wheel; an unreadable listing (offline) keeps the staged copies, and the directory used as its own wheelhouse lists everything it holds. Tests: block-form and inline rows on both scripts plus the Python reader, the dependency args never naming an explicit index, and the reconciliation across a local wheelhouse, a URL wheelhouse, an offline listing and the self-sourced directory. Ten mutants on the change are killed. * Shorten the comments the explicit-index and wheelhouse reconciliation changes added for PR #10282 * Hand PyPI-provided wheels to the skip list, install a hosted torchcodec, order xformers after the final torch repair, and key the pip refusal to the generated overrides for PR #10282 * Keep folded caller overrides in a per-run file rather than the persistent WoA overrides for PR #10282 * Backtrack to the newest paired torch and keep inline comments when rebasing override lines for PR #10282 * Form the rebase test's expected paths per host and let the ARM64 CI print its pytest output for PR #10282 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add a signed Windows on ARM wheelhouse workflow that builds, Authenticode signs and publishes the pyarrow and sqlite-vec win_arm64 wheels for PR #10282 * Publish the Windows on ARM wheelhouse under the Windows-ARM64 release tag for PR #10282 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>
This commit is contained in:
parent
f621d12a21
commit
b535eca250
40 changed files with 15085 additions and 239 deletions
44
.github/scripts/repack_sqlite_vec.py
vendored
Normal file
44
.github/scripts/repack_sqlite_vec.py
vendored
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
"""Repack the PyPI sqlite-vec win_amd64 wheel with an ARM64 vec0.dll under the win_arm64 tag.
|
||||
|
||||
The wheel is pure Python plus one DLL, generated upstream by sqlite-dist: replacing the DLL,
|
||||
the WHEEL tag and RECORD is the whole port. Usage: repack_sqlite_vec.py <amd64 wheel> <vec0.dll> <out dir>
|
||||
"""
|
||||
import base64, hashlib, struct, sys, zipfile
|
||||
from pathlib import Path
|
||||
|
||||
src, dll, out = Path(sys.argv[1]), Path(sys.argv[2]), Path(sys.argv[3])
|
||||
assert src.name.endswith("-py3-none-win_amd64.whl"), src.name
|
||||
dst = out / src.name.replace("-win_amd64.whl", "-win_arm64.whl")
|
||||
|
||||
data = dll.read_bytes()
|
||||
e_lfanew = struct.unpack_from("<I", data, 0x3C)[0]
|
||||
assert data[e_lfanew : e_lfanew + 4] == b"PE\0\0", "not a PE file"
|
||||
machine = struct.unpack_from("<H", data, e_lfanew + 4)[0]
|
||||
assert machine == 0xAA64, f"vec0.dll machine is {machine:#x}, not ARM64 (0xaa64)"
|
||||
|
||||
def record_line(name: str, blob: bytes) -> str:
|
||||
digest = base64.urlsafe_b64encode(hashlib.sha256(blob).digest()).rstrip(b"=").decode()
|
||||
return f"{name},sha256={digest},{len(blob)}"
|
||||
|
||||
out.mkdir(parents = True, exist_ok = True)
|
||||
with zipfile.ZipFile(src) as zin, zipfile.ZipFile(dst, "w", zipfile.ZIP_DEFLATED) as zout:
|
||||
names = zin.namelist()
|
||||
record_name = next(n for n in names if n.endswith(".dist-info/RECORD"))
|
||||
lines = []
|
||||
for name in names:
|
||||
if name == record_name:
|
||||
continue
|
||||
blob = zin.read(name)
|
||||
if name.endswith("/vec0.dll"):
|
||||
blob = data
|
||||
elif name.endswith(".dist-info/WHEEL"):
|
||||
text = blob.decode()
|
||||
assert "Tag: py3-none-win_amd64" in text, text
|
||||
blob = text.replace("Tag: py3-none-win_amd64", "Tag: py3-none-win_arm64").encode()
|
||||
zout.writestr(name, blob)
|
||||
lines.append(record_line(name, blob))
|
||||
lines.append(f"{record_name},,")
|
||||
zout.writestr(record_name, "\n".join(lines) + "\n")
|
||||
print(dst.name, dst.stat().st_size, "bytes")
|
||||
205
.github/scripts/sign_wheel_binaries.py
vendored
Normal file
205
.github/scripts/sign_wheel_binaries.py
vendored
Normal file
|
|
@ -0,0 +1,205 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""Authenticode-sign the native binaries inside a wheel and repack it.
|
||||
|
||||
A .whl cannot itself be Authenticode signed: it is a zip, and Windows has no subject
|
||||
interface package for one. What CAN be signed are the PE images inside it -- the .pyd
|
||||
extension modules and the .dll files they link -- which is what Windows actually loads and
|
||||
what SmartScreen and an enterprise WDAC policy judge. So this unpacks the wheel, signs each
|
||||
PE we built, and packs it back under the same filename.
|
||||
|
||||
RECORD is regenerated, because it has to be: signing appends a certificate table, so every
|
||||
signed member's sha256 and length change, and a wheel whose RECORD disagrees with its
|
||||
contents is one `pip check`, `uv pip install --strict` or reproducibility audit away from
|
||||
being called corrupt.
|
||||
|
||||
Files that already carry a certificate are left alone. delvewheel vendors Microsoft's own
|
||||
redistributables (msvcp140.dll and friends) into pyarrow, already signed by Microsoft;
|
||||
re-signing those would replace their signature with ours and claim authorship of somebody
|
||||
else's binary.
|
||||
|
||||
The signing itself is delegated, never reimplemented: --signer names the command, and CI
|
||||
passes studio/src-tauri/windows/sign-with-trusted-signing.ps1, the same helper the desktop
|
||||
release uses, so the endpoint, the identity and the Azure retry policy have one definition.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import hashlib
|
||||
import shutil
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
#: IMAGE_FILE_MACHINE_ARM64. The wheels this signs are win_arm64.
|
||||
MACHINE_ARM64 = 0xAA64
|
||||
#: Data directory index 4 is the Certificate Table -- a non-zero size means "already signed".
|
||||
CERTIFICATE_TABLE_INDEX = 4
|
||||
|
||||
|
||||
def _pe_header_offset(blob: bytes) -> "int | None":
|
||||
"""Offset of the PE signature, or None when this is not a PE image at all."""
|
||||
if len(blob) < 0x40 or blob[:2] != b"MZ":
|
||||
return None
|
||||
try:
|
||||
offset = struct.unpack_from("<I", blob, 0x3C)[0]
|
||||
except struct.error:
|
||||
return None
|
||||
if offset <= 0 or offset + 24 > len(blob) or blob[offset : offset + 4] != b"PE\0\0":
|
||||
return None
|
||||
return offset
|
||||
|
||||
|
||||
def pe_machine(blob: bytes) -> "int | None":
|
||||
"""The COFF machine type, or None when the bytes are not a PE image."""
|
||||
offset = _pe_header_offset(blob)
|
||||
if offset is None:
|
||||
return None
|
||||
return struct.unpack_from("<H", blob, offset + 4)[0]
|
||||
|
||||
|
||||
def is_signed(blob: bytes) -> bool:
|
||||
"""Does this PE image already carry a certificate table?
|
||||
|
||||
Read from the optional header's data directory rather than by running a tool, so the
|
||||
answer does not depend on a Windows host or on the file being unpacked first.
|
||||
"""
|
||||
offset = _pe_header_offset(blob)
|
||||
if offset is None:
|
||||
return False
|
||||
magic_at = offset + 24
|
||||
if magic_at + 2 > len(blob):
|
||||
return False
|
||||
magic = struct.unpack_from("<H", blob, magic_at)[0]
|
||||
# PE32+ has 8 more bytes of fixed fields before the data directory than PE32 does.
|
||||
directory_at = magic_at + (112 if magic == 0x20B else 96) + CERTIFICATE_TABLE_INDEX * 8
|
||||
if directory_at + 8 > len(blob):
|
||||
return False
|
||||
_, size = struct.unpack_from("<II", blob, directory_at)
|
||||
return size > 0
|
||||
|
||||
|
||||
def record_line(name: str, blob: bytes) -> str:
|
||||
"""One RECORD row: urlsafe base64 sha256 with the padding stripped, per PEP 376."""
|
||||
digest = base64.urlsafe_b64encode(hashlib.sha256(blob).digest()).rstrip(b"=").decode()
|
||||
return f"{name},sha256={digest},{len(blob)}"
|
||||
|
||||
|
||||
def sign_file(path: Path, signer: "list[str]") -> None:
|
||||
"""Run the signing command over one file, failing the build if it cannot sign."""
|
||||
result = subprocess.run([*signer, str(path)], text = True)
|
||||
if result.returncode != 0:
|
||||
raise SystemExit(f"signing failed for {path.name} (exit {result.returncode})")
|
||||
|
||||
|
||||
def sign_wheel(
|
||||
wheel: Path,
|
||||
out_dir: Path,
|
||||
signer: "list[str]",
|
||||
*,
|
||||
require_machine: "int | None" = MACHINE_ARM64,
|
||||
) -> "tuple[Path, list[str], list[str]]":
|
||||
"""Sign every unsigned PE inside `wheel` and write the result into `out_dir`.
|
||||
|
||||
Returns (written wheel, names signed, names skipped as already signed).
|
||||
"""
|
||||
out_dir.mkdir(parents = True, exist_ok = True)
|
||||
destination = out_dir / wheel.name
|
||||
signed: list[str] = []
|
||||
skipped: list[str] = []
|
||||
|
||||
with tempfile.TemporaryDirectory() as scratch:
|
||||
work = Path(scratch)
|
||||
with zipfile.ZipFile(wheel) as archive:
|
||||
names = archive.namelist()
|
||||
payload = {name: archive.read(name) for name in names}
|
||||
|
||||
for name in names:
|
||||
blob = payload[name]
|
||||
machine = pe_machine(blob)
|
||||
if machine is None:
|
||||
continue
|
||||
if require_machine is not None and machine != require_machine:
|
||||
raise SystemExit(
|
||||
f"{wheel.name}: {name} is machine {machine:#x}, not {require_machine:#x}"
|
||||
)
|
||||
if is_signed(blob):
|
||||
skipped.append(name)
|
||||
continue
|
||||
# Signed on disk under its own name: signtool and trusted-signing-cli both work
|
||||
# in place, and the suffix decides which subject interface package is used.
|
||||
staged = work / Path(name).name
|
||||
staged.write_bytes(blob)
|
||||
sign_file(staged, signer)
|
||||
resigned = staged.read_bytes()
|
||||
if not is_signed(resigned):
|
||||
raise SystemExit(f"{wheel.name}: {name} has no certificate after signing")
|
||||
payload[name] = resigned
|
||||
signed.append(name)
|
||||
staged.unlink()
|
||||
|
||||
if not signed and not skipped:
|
||||
raise SystemExit(f"{wheel.name}: no PE images found -- nothing to sign")
|
||||
|
||||
record_name = next((n for n in names if n.endswith(".dist-info/RECORD")), None)
|
||||
if record_name is None:
|
||||
raise SystemExit(f"{wheel.name}: no .dist-info/RECORD")
|
||||
|
||||
# Member order preserved: the installer does not care, but a diff between the signed
|
||||
# and unsigned wheel should show the signatures and nothing else.
|
||||
with zipfile.ZipFile(destination, "w", zipfile.ZIP_DEFLATED) as out:
|
||||
lines = []
|
||||
for name in names:
|
||||
if name == record_name:
|
||||
continue
|
||||
out.writestr(name, payload[name])
|
||||
lines.append(record_line(name, payload[name]))
|
||||
lines.append(f"{record_name},,")
|
||||
out.writestr(record_name, "\n".join(lines) + "\n")
|
||||
|
||||
return destination, signed, skipped
|
||||
|
||||
|
||||
def main(argv: "list[str] | None" = None) -> int:
|
||||
parser = argparse.ArgumentParser(description = __doc__)
|
||||
parser.add_argument("wheels", nargs = "+", type = Path)
|
||||
parser.add_argument("--out", required = True, type = Path)
|
||||
parser.add_argument(
|
||||
"--signer",
|
||||
required = True,
|
||||
help = "signing command; the file to sign is appended as the last argument",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--expect-machine",
|
||||
default = "arm64",
|
||||
choices = ("arm64", "any"),
|
||||
help = "refuse a wheel carrying a PE for another architecture (default: arm64)",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
import shlex
|
||||
|
||||
signer = shlex.split(args.signer, posix = False)
|
||||
require = MACHINE_ARM64 if args.expect_machine == "arm64" else None
|
||||
failures = 0
|
||||
for wheel in args.wheels:
|
||||
written, signed, skipped = sign_wheel(wheel, args.out, signer, require_machine = require)
|
||||
print(f"{written.name}: signed {len(signed)}, already signed {len(skipped)}")
|
||||
for name in signed:
|
||||
print(f" signed {name}")
|
||||
for name in skipped:
|
||||
print(f" kept {name} (already carries a certificate)")
|
||||
if not signed:
|
||||
print(f"::warning::{wheel.name} had nothing to sign")
|
||||
failures += 1
|
||||
return 1 if failures and len(args.wheels) == failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
167
.github/workflows/windows-arm64-ci.yml
vendored
Normal file
167
.github/workflows/windows-arm64-ci.yml
vendored
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
# The native Windows on ARM install path, run on Windows on ARM: wheel-tag arithmetic, path handling and
|
||||
# PowerShell parsing, none of which fails the same way on x64. No GPU, CUDA or compiler, so a full install or a Tauri build is out of scope (ring and aws-lc-sys need clang for their aarch64 sources).
|
||||
name: Windows ARM64 CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'install.ps1'
|
||||
- 'scripts/uninstall.ps1'
|
||||
- 'scripts/uninstall.sh'
|
||||
- 'studio/install_python_stack.py'
|
||||
- 'studio/setup.ps1'
|
||||
- 'pyproject.toml'
|
||||
- 'studio/backend/requirements/**'
|
||||
- 'tests/studio/install/test_woa_wheelhouse_prefers_pypi.py'
|
||||
- 'tests/studio/install/test_woa_wheelhouse_guard_behaviour.py'
|
||||
- 'tests/studio/install/test_woa_torch_index_persistence.py'
|
||||
- 'tests/studio/install/test_diffusers_pin.py'
|
||||
- 'tests/python/test_windows_arm64_wheelhouse_tags.py'
|
||||
- 'tests/studio/test_uninstall_arg_guard.ps1'
|
||||
- 'tests/studio/test_uninstall_dual_install_icon.ps1'
|
||||
- 'tests/studio/test_uninstall_prebuilt_parity.ps1'
|
||||
- 'tests/studio/test_uninstall_reparse_stop_roots.ps1'
|
||||
- '.github/workflows/windows-arm64-ci.yml'
|
||||
pull_request:
|
||||
paths:
|
||||
- 'install.ps1'
|
||||
- 'scripts/uninstall.ps1'
|
||||
- 'scripts/uninstall.sh'
|
||||
- 'studio/install_python_stack.py'
|
||||
- 'studio/setup.ps1'
|
||||
- 'pyproject.toml'
|
||||
- 'studio/backend/requirements/**'
|
||||
- 'tests/studio/install/test_woa_wheelhouse_prefers_pypi.py'
|
||||
- 'tests/studio/install/test_woa_wheelhouse_guard_behaviour.py'
|
||||
- 'tests/studio/install/test_woa_torch_index_persistence.py'
|
||||
- 'tests/studio/install/test_diffusers_pin.py'
|
||||
- 'tests/python/test_windows_arm64_wheelhouse_tags.py'
|
||||
- 'tests/studio/test_uninstall_arg_guard.ps1'
|
||||
- 'tests/studio/test_uninstall_dual_install_icon.ps1'
|
||||
- 'tests/studio/test_uninstall_prebuilt_parity.ps1'
|
||||
- 'tests/studio/test_uninstall_reparse_stop_roots.ps1'
|
||||
- '.github/workflows/windows-arm64-ci.yml'
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || '' }}
|
||||
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
woa:
|
||||
# Free for public repositories since 2025-08-07; a fork on a private plan fails to start.
|
||||
runs-on: windows-11-arm
|
||||
timeout-minutes: 40
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
# Both shells: pwsh 7 is what CI has always run, and Windows PowerShell 5.1 is the only shell on a stock Windows box, which the CLI launches setup.ps1 with (see #8335).
|
||||
powershell: [pwsh-7, windows-powershell-5.1]
|
||||
name: woa (${{ matrix.powershell }})
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: The runner really is ARM64
|
||||
shell: pwsh
|
||||
run: |
|
||||
$arch = [System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture
|
||||
$proc = [System.Runtime.InteropServices.RuntimeInformation]::ProcessArchitecture
|
||||
# Machine scope: an x64-emulated process reports AMD64 in its own environment block.
|
||||
$machine = [Environment]::GetEnvironmentVariable(
|
||||
"PROCESSOR_ARCHITECTURE", [EnvironmentVariableTarget]::Machine)
|
||||
Write-Host "OSArchitecture=$arch ProcessArchitecture=$proc machine=$machine"
|
||||
if ("$arch" -ne "Arm64") {
|
||||
Write-Host "::error::this job is not running on ARM64 (OSArchitecture=$arch)"
|
||||
exit 1
|
||||
}
|
||||
|
||||
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
# Required: the image ships an emulated x64 Python whose sysconfig says win-amd64.
|
||||
architecture: 'arm64'
|
||||
|
||||
- name: The interpreter really is ARM64
|
||||
shell: pwsh
|
||||
run: |
|
||||
python -c "import platform, sysconfig, sys; print(sys.version); print('machine', platform.machine()); print('platform', sysconfig.get_platform())"
|
||||
python -c "import sysconfig, sys; p = sysconfig.get_platform(); sys.exit(0 if p == 'win-arm64' else (print(f'::error::sysconfig reports {p}, expected win-arm64') or 1))"
|
||||
|
||||
- run: python -m pip install -U pip pytest packaging
|
||||
|
||||
# 5.1 has no `pwsh` on PATH and the suites locate their shell with shutil.which("pwsh"), so without this they SKIP and report green. A copy, because CreateProcess will not run a .cmd.
|
||||
- name: Put Windows PowerShell 5.1 on PATH as pwsh
|
||||
if: matrix.powershell == 'windows-powershell-5.1'
|
||||
shell: pwsh
|
||||
run: |
|
||||
$dir = Join-Path $env:RUNNER_TEMP "wps51"
|
||||
New-Item -ItemType Directory -Path $dir -Force | Out-Null
|
||||
$src = Join-Path $env:SystemRoot "System32\WindowsPowerShell\v1.0\powershell.exe"
|
||||
if (-not (Test-Path -LiteralPath $src)) {
|
||||
Write-Host "::error::Windows PowerShell 5.1 was not found at $src"
|
||||
exit 1
|
||||
}
|
||||
Copy-Item -LiteralPath $src -Destination (Join-Path $dir "pwsh.exe") -Force
|
||||
"$dir" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
|
||||
|
||||
- name: Which PowerShell the suites will use
|
||||
shell: pwsh
|
||||
run: |
|
||||
$exe = (Get-Command pwsh).Source
|
||||
$ver = & $exe -NoProfile -Command '$PSVersionTable.PSVersion.ToString()'
|
||||
Write-Host "pwsh resolves to $exe (version $ver)"
|
||||
$wanted = if ("${{ matrix.powershell }}" -eq "pwsh-7") { "7" } else { "5" }
|
||||
if (-not $ver.StartsWith($wanted)) {
|
||||
Write-Host "::error::expected PowerShell $wanted.x for this row, got $ver"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# A skip is a failure here: these files exist for this row, and -rs only REPORTS skipped cases, so the summary line is read back and any skipped count fails the step.
|
||||
- name: WoA installer and wheelhouse tests
|
||||
shell: pwsh
|
||||
env:
|
||||
UNSLOTH_NO_TORCH: '1'
|
||||
run: |
|
||||
& python -m pytest `
|
||||
tests/studio/install/test_woa_wheelhouse_prefers_pypi.py `
|
||||
tests/studio/install/test_woa_wheelhouse_guard_behaviour.py `
|
||||
tests/studio/install/test_woa_torch_index_persistence.py `
|
||||
tests/studio/install/test_diffusers_pin.py `
|
||||
tests/python/test_windows_arm64_wheelhouse_tags.py `
|
||||
-q -rs 2>&1 | Tee-Object -Variable pytestOut
|
||||
$rc = $LASTEXITCODE
|
||||
$summary = ($pytestOut | Where-Object { $_ -match '\d+ (passed|failed|error)' } | Select-Object -Last 1)
|
||||
if ($rc -ne 0) { exit $rc }
|
||||
if ($summary -match '(\d+) skipped') {
|
||||
Write-Host "::error::$($Matches[1]) test(s) skipped on the row they exist for: $summary"
|
||||
exit 1
|
||||
}
|
||||
|
||||
- name: Uninstaller tests
|
||||
shell: pwsh
|
||||
run: |
|
||||
$failed = @()
|
||||
foreach ($t in @(
|
||||
"test_uninstall_arg_guard",
|
||||
"test_uninstall_dual_install_icon",
|
||||
"test_uninstall_prebuilt_parity",
|
||||
"test_uninstall_reparse_stop_roots"
|
||||
)) {
|
||||
$path = "tests/studio/$t.ps1"
|
||||
if (-not (Test-Path -LiteralPath $path)) { continue }
|
||||
Write-Host "::group::$t"
|
||||
& (Get-Command pwsh).Source -NoProfile -File $path
|
||||
$code = $LASTEXITCODE
|
||||
Write-Host "::endgroup::"
|
||||
if ($code -ne 0) { $failed += "$t (exit $code)" }
|
||||
}
|
||||
if ($failed.Count -gt 0) {
|
||||
Write-Host "::error::failing uninstaller tests: $($failed -join ', ')"
|
||||
exit 1
|
||||
}
|
||||
634
.github/workflows/woa-wheelhouse.yml
vendored
Normal file
634
.github/workflows/woa-wheelhouse.yml
vendored
Normal file
|
|
@ -0,0 +1,634 @@
|
|||
# The Windows on ARM wheelhouse: build, SIGN and publish the two win_arm64 wheels the
|
||||
# native ARM64 installer cannot get anywhere else.
|
||||
#
|
||||
# pyarrow -- mandatory. `datasets` imports it, and PyPI has published no win_arm64 wheel
|
||||
# in any release (apache/arrow#48539 adds the CI support and is still open).
|
||||
# Its sdist needs Arrow C++, vcpkg and the MSVC ARM64 toolchain, which is
|
||||
# ~45 min on this runner and flatly out of reach on a user's machine.
|
||||
# sqlite-vec -- optional (RAG). PyPI ships one py3-none-win_amd64 wheel: pure Python plus a
|
||||
# single x64 vec0.dll. The port is that DLL rebuilt for ARM64.
|
||||
#
|
||||
# Signed, because these are the only binaries in the stack we produce ourselves. Everything
|
||||
# else on that install comes from PyPI or from NVIDIA. An unsigned .pyd we host and tell the
|
||||
# installer to fetch is the weakest link in the chain, and Authenticode is what Windows,
|
||||
# SmartScreen and a WDAC policy actually check. The procedure is release-desktop.yml's,
|
||||
# unchanged: the digest-pinned trusted-signing-cli, sign-with-trusted-signing.ps1, and the
|
||||
# release-signing environment gating the Azure secrets.
|
||||
#
|
||||
# A .whl cannot be Authenticode signed (a zip has no subject interface package), so what is
|
||||
# signed is every PE inside it; sign_wheel_binaries.py repacks and regenerates RECORD.
|
||||
#
|
||||
# Dispatch only. This holds signing credentials and writes a release, so it must never be
|
||||
# reachable from a pull request, and `publish: false` is the default: a run then builds,
|
||||
# signs and verifies, and leaves the wheels as artifacts to inspect.
|
||||
name: Windows on ARM wheelhouse
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
arrow_tag:
|
||||
description: "apache/arrow tag to build pyarrow from"
|
||||
default: "apache-arrow-25.0.1"
|
||||
sqlite_vec_tag:
|
||||
description: "asg017/sqlite-vec tag"
|
||||
default: "v0.1.9"
|
||||
release_tag:
|
||||
description: "release tag to publish under"
|
||||
default: "Windows-ARM64"
|
||||
publish:
|
||||
description: "upload to the release (otherwise artifacts only)"
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: woa-wheelhouse-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
||||
env:
|
||||
# Windows ARM64 support in Arrow's wheel build is CI-only and still unmerged, so it is
|
||||
# overlaid onto the release tag. Both halves move together or not at all.
|
||||
ARROW_WOA_REPO: MugundanMCW/arrow
|
||||
ARROW_WOA_REF: Pyarrow_WoA
|
||||
|
||||
jobs:
|
||||
build-pyarrow:
|
||||
name: pyarrow win_arm64 / cp${{ matrix.python }}
|
||||
runs-on: windows-11-arm
|
||||
timeout-minutes: 300
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
python: ['3.11', '3.12', '3.13', '3.14']
|
||||
env:
|
||||
CMAKE_BUILD_TYPE: release
|
||||
ARCH: ARM64
|
||||
steps:
|
||||
- name: Harden runner (audit)
|
||||
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
repository: apache/arrow
|
||||
ref: ${{ inputs.arrow_tag }}
|
||||
path: arrow
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
repository: ${{ env.ARROW_WOA_REPO }}
|
||||
ref: ${{ env.ARROW_WOA_REF }}
|
||||
path: arrow-woa
|
||||
sparse-checkout: |
|
||||
ci/scripts
|
||||
ci/vcpkg
|
||||
sparse-checkout-cone-mode: false
|
||||
|
||||
# Every edit below asserts on the text it replaces, so an upstream rename fails this
|
||||
# step rather than silently producing an x64 wheel with an arm64 filename.
|
||||
- name: Overlay the ARM64 build support onto the release tag
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
Copy-Item arrow-woa/ci/scripts/python_wheel_windows_build.bat arrow/ci/scripts/ -Force
|
||||
Copy-Item arrow-woa/ci/scripts/python_wheel_windows_test.bat arrow/ci/scripts/ -Force
|
||||
|
||||
$json = 'arrow/ci/vcpkg/vcpkg.json'
|
||||
$t = Get-Content $json -Raw
|
||||
$edits = @(
|
||||
@('"supports": "x64 | (arm64 & !windows)"', '"supports": "x64 | arm64"'),
|
||||
@(" `"xsimd`",", " {`n `"name`": `"xsimd`",`n `"platform`": `"!(arm64 & windows)`"`n },")
|
||||
)
|
||||
foreach ($e in $edits) {
|
||||
if (-not $t.Contains($e[0])) { throw "vcpkg.json no longer contains: $($e[0])" }
|
||||
$t = $t.Replace($e[0], $e[1])
|
||||
}
|
||||
Set-Content $json -Value $t -Encoding ascii -NoNewline
|
||||
python -c "import json; json.load(open('$json'))"
|
||||
|
||||
# The upstream script hardcodes VS 2022 Enterprise and its generator. This image
|
||||
# moves to VS 2026, so both come from the installed toolset instead.
|
||||
$vswhere = "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe"
|
||||
$vs = & $vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.ARM64 -property installationPath
|
||||
if (-not $vs) { throw 'no Visual Studio with the ARM64 C++ tools' }
|
||||
$major = [int]((& $vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.ARM64 -property installationVersion) -split '\.')[0]
|
||||
$generator = switch ($major) { 17 { 'Visual Studio 17 2022' } 18 { 'Visual Studio 18 2026' } default { throw "unknown Visual Studio major $major" } }
|
||||
$bat = 'arrow/ci/scripts/python_wheel_windows_build.bat'
|
||||
$b = Get-Content $bat -Raw
|
||||
$edits = @(
|
||||
@('set VCVARS_BAT=C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsarm64.bat', "set VCVARS_BAT=$vs\VC\Auxiliary\Build\vcvarsarm64.bat"),
|
||||
@('set CMAKE_GENERATOR=Visual Studio 17 2022', "set CMAKE_GENERATOR=$generator")
|
||||
)
|
||||
foreach ($e in $edits) {
|
||||
if (-not $b.Contains($e[0])) { throw "build.bat no longer contains: $($e[0])" }
|
||||
$b = $b.Replace($e[0], $e[1])
|
||||
}
|
||||
Set-Content $bat -Value $b -Encoding ascii
|
||||
if (-not (Test-Path "$vs\VC\Auxiliary\Build\vcvarsarm64.bat")) { throw "no vcvarsarm64.bat under $vs" }
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ matrix.python }}
|
||||
architecture: arm64
|
||||
|
||||
- name: Python build dependencies
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
python -c "import sys, sysconfig; print(sys.version, sysconfig.get_platform())"
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install -r arrow/python/requirements-wheel-build.txt
|
||||
"PYTHON_CMD=$env:pythonLocation\python.exe" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
|
||||
|
||||
- name: Version from the C++ tree
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
$cm = Get-Content arrow/cpp/CMakeLists.txt -Raw
|
||||
if ($cm -notmatch 'set\(ARROW_VERSION "([0-9.]+)(-SNAPSHOT)?"\)') { throw 'no ARROW_VERSION' }
|
||||
$v = $Matches[1]; if ($Matches[2]) { $v = "$v.dev${{ github.run_number }}" }
|
||||
Write-Host "pyarrow version: $v"
|
||||
"SETUPTOOLS_SCM_PRETEND_VERSION=$v" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
|
||||
|
||||
# Trimmed to what `datasets` reaches: this removes gRPC and the cloud SDKs from the
|
||||
# vcpkg build entirely, which is most of the wall clock and most of the attack surface.
|
||||
- name: Trim the build to what datasets needs
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
$bat = 'arrow/ci/scripts/python_wheel_windows_build.bat'
|
||||
$t = Get-Content $bat -Raw
|
||||
foreach ($flag in 'ARROW_AZURE','ARROW_FLIGHT','ARROW_GCS','ARROW_HDFS','ARROW_ORC','ARROW_S3','ARROW_SUBSTRAIT','ARROW_TENSORFLOW','ARROW_WITH_OPENTELEMETRY') {
|
||||
$before = $t
|
||||
$t = $t -replace "set $flag=ON", "set $flag=OFF"
|
||||
if ($t -eq $before) { throw "did not find 'set $flag=ON'" }
|
||||
}
|
||||
Set-Content $bat -Value $t -Encoding ascii
|
||||
Select-String -Path $bat -Pattern '^set ARROW_' | ForEach-Object { $_.Line }
|
||||
|
||||
- name: Cache vcpkg packages
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: C:\vcpkg\installed
|
||||
key: vcpkg-arm64-static-md-trimmed-${{ inputs.arrow_tag }}-${{ hashFiles('arrow/ci/vcpkg/vcpkg.json') }}
|
||||
restore-keys: |
|
||||
vcpkg-arm64-static-md-trimmed-${{ inputs.arrow_tag }}-
|
||||
vcpkg-arm64-static-md-trimmed-
|
||||
|
||||
# Retried: the ports fetch source tarballs from archive.apache.org and friends, and one
|
||||
# timed-out download fails the whole step. Packages already built survive an attempt.
|
||||
- name: vcpkg dependencies (base + json + parquet)
|
||||
shell: cmd
|
||||
run: |
|
||||
cd arrow\ci\vcpkg
|
||||
set attempt=0
|
||||
:again
|
||||
set /a attempt+=1
|
||||
C:\vcpkg\vcpkg.exe install ^
|
||||
--triplet arm64-windows-static-md ^
|
||||
--x-install-root=C:\vcpkg\installed ^
|
||||
--x-manifest-root=. ^
|
||||
--x-feature=json ^
|
||||
--x-feature=parquet && exit /B 0
|
||||
if %attempt% GEQ 3 exit /B 1
|
||||
echo vcpkg install failed, attempt %attempt% of 3; retrying in 60s
|
||||
timeout /t 60 /nobreak >nul
|
||||
goto again
|
||||
|
||||
- name: Build Arrow C++ and the pyarrow wheel
|
||||
shell: cmd
|
||||
run: |
|
||||
call "arrow\ci\scripts\python_wheel_windows_build.bat"
|
||||
|
||||
# A win_arm64 filename says nothing about what was linked in, so every native image is
|
||||
# opened and its COFF machine checked before anything downstream trusts the wheel.
|
||||
- name: Verify the wheel is really ARM64
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
$whl = Get-ChildItem arrow/python/repaired_wheels/*.whl | Select-Object -First 1
|
||||
Write-Host "wheel: $($whl.Name) ($([math]::Round($whl.Length/1MB)) MB)"
|
||||
if ($whl.Name -notmatch '-win_arm64\.whl$') { throw "not a win_arm64 wheel: $($whl.Name)" }
|
||||
python -c @"
|
||||
import struct, sys, zipfile
|
||||
z = zipfile.ZipFile(sys.argv[1]); bad = []; n = 0
|
||||
for name in z.namelist():
|
||||
if not name.lower().endswith(('.pyd', '.dll')): continue
|
||||
d = z.read(name); off = struct.unpack_from('<I', d, 0x3C)[0]
|
||||
machine = struct.unpack_from('<H', d, off + 4)[0]; n += 1
|
||||
if machine != 0xAA64: bad.append((name, hex(machine)))
|
||||
print(n, 'native images, all ARM64' if not bad else bad)
|
||||
sys.exit(1 if bad or n == 0 else 0)
|
||||
"@ $whl.FullName
|
||||
if ($LASTEXITCODE -ne 0) { exit 1 }
|
||||
|
||||
- name: Smoke test in a fresh venv
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
$whl = Get-ChildItem arrow/python/repaired_wheels/*.whl | Select-Object -First 1
|
||||
python -m venv smoke
|
||||
smoke\Scripts\python.exe -m pip install --quiet $whl.FullName numpy
|
||||
smoke\Scripts\python.exe -c @"
|
||||
import pyarrow as pa, pyarrow.parquet as pq, pyarrow.dataset as ds, pyarrow.compute as pc, pyarrow.csv, pyarrow.json, sysconfig
|
||||
print('pyarrow', pa.__version__, sysconfig.get_platform())
|
||||
t = pa.table({'a': [1, 2, 3], 'b': ['x', 'y', 'z']})
|
||||
pq.write_table(t, 'smoke.parquet')
|
||||
back = pq.read_table('smoke.parquet')
|
||||
assert back.equals(t), back
|
||||
print('sum a =', pc.sum(back['a']).as_py())
|
||||
print('dataset rows =', ds.dataset('smoke.parquet').count_rows())
|
||||
print('compression:', [c for c in ('snappy','zstd','lz4','brotli','gzip','bz2') if pa.Codec.is_available(c)])
|
||||
"@
|
||||
if ($LASTEXITCODE -ne 0) { exit 1 }
|
||||
|
||||
# The reason the wheel exists at all, so it is a gate and not a nice-to-have.
|
||||
- name: Does datasets work on it
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
smoke\Scripts\python.exe -m pip install --quiet datasets
|
||||
smoke\Scripts\python.exe -c @"
|
||||
from datasets import Dataset
|
||||
d = Dataset.from_dict({'text': ['hello', 'world'], 'n': [1, 2]})
|
||||
d = d.map(lambda r: {'len': len(r['text'])})
|
||||
d.to_parquet('ds.parquet')
|
||||
print('datasets round-trip ok:', Dataset.from_parquet('ds.parquet').to_dict())
|
||||
"@
|
||||
if ($LASTEXITCODE -ne 0) { exit 1 }
|
||||
|
||||
- uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: unsigned-pyarrow-cp${{ matrix.python }}
|
||||
path: arrow/python/repaired_wheels/*.whl
|
||||
if-no-files-found: error
|
||||
|
||||
build-sqlite-vec:
|
||||
name: sqlite-vec win_arm64
|
||||
runs-on: windows-11-arm
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
- name: Harden runner (audit)
|
||||
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
repository: asg017/sqlite-vec
|
||||
ref: ${{ inputs.sqlite_vec_tag }}
|
||||
path: sv
|
||||
fetch-depth: 0
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
path: tools
|
||||
sparse-checkout: |
|
||||
.github/scripts
|
||||
sparse-checkout-cone-mode: false
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.13'
|
||||
architecture: arm64
|
||||
|
||||
# scripts/vendor.sh without the unzip dependency. The amalgamation URL is read from
|
||||
# that script rather than hardcoded here, so the tag decides which sqlite is vendored.
|
||||
- name: Vendor the sqlite amalgamation
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
$vendor = Get-Content sv/scripts/vendor.sh -Raw
|
||||
if ($vendor -notmatch '(https://www\.sqlite\.org/\d{4}/sqlite-amalgamation-\d+\.zip)') {
|
||||
throw "could not read the amalgamation URL out of scripts/vendor.sh"
|
||||
}
|
||||
$url = $Matches[1]
|
||||
Write-Host "amalgamation: $url"
|
||||
Invoke-WebRequest -Uri $url -OutFile amalgamation.zip -UseBasicParsing -MaximumRetryCount 3 -RetryIntervalSec 5
|
||||
Expand-Archive amalgamation.zip -DestinationPath amalgamation
|
||||
New-Item -ItemType Directory -Force -Path sv/vendor | Out-Null
|
||||
Get-ChildItem amalgamation/*/* | Move-Item -Destination sv/vendor
|
||||
Get-ChildItem sv/vendor
|
||||
|
||||
- name: Generate sqlite-vec.h from its template
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
Set-Location sv
|
||||
$version = (Get-Content VERSION -Raw).Trim()
|
||||
$parts = $version.Split('.')
|
||||
$source = (git log -n 1 --pretty=format:%H -- VERSION)
|
||||
$date = (git log -n 1 --pretty=format:%cI -- VERSION)
|
||||
$h = Get-Content sqlite-vec.h.tmpl -Raw
|
||||
$h = $h.Replace('${VERSION}', $version).Replace('${DATE}', $date).Replace('${SOURCE}', $source)
|
||||
$h = $h.Replace('${VERSION_MAJOR}', $parts[0]).Replace('${VERSION_MINOR}', $parts[1]).Replace('${VERSION_PATCH}', ($parts[2] -split '-')[0])
|
||||
if ($h -match '\$\{') { throw "unreplaced placeholder in sqlite-vec.h" }
|
||||
Set-Content sqlite-vec.h -Value $h -Encoding ascii
|
||||
"SQLITE_VEC_VERSION=$version" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
|
||||
|
||||
- name: Locate the ARM64 MSVC environment
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
$vswhere = "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe"
|
||||
$vs = & $vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.ARM64 -property installationPath
|
||||
if (-not $vs) { $vs = & $vswhere -latest -products * -property installationPath }
|
||||
$bat = Join-Path $vs 'VC\Auxiliary\Build\vcvarsarm64.bat'
|
||||
if (-not (Test-Path $bat)) { throw "no vcvarsarm64.bat under $vs" }
|
||||
Write-Host "vcvars: $bat"
|
||||
"VCVARS_BAT=$bat" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
|
||||
|
||||
# The upstream release workflow's own cl line, run natively for arm64.
|
||||
- name: Compile vec0.dll for ARM64
|
||||
shell: cmd
|
||||
run: |
|
||||
call "%VCVARS_BAT%" || exit /B 1
|
||||
cd sv
|
||||
mkdir dist
|
||||
cl.exe /nologo /W4 /O2 /LD /Ivendor sqlite-vec.c /Fe:dist\vec0.dll || exit /B 1
|
||||
dir dist
|
||||
|
||||
# The PyPI wheel is pure Python plus that one DLL, generated by sqlite-dist, so the
|
||||
# port is a DLL swap, a tag rewrite and a regenerated RECORD -- not a rebuild.
|
||||
- name: Repack the PyPI wheel with the ARM64 DLL
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
python -m pip download --quiet --no-deps --only-binary=:all: --platform win_amd64 --python-version 3.13 -d amd64 "sqlite-vec==$env:SQLITE_VEC_VERSION"
|
||||
$src = Get-ChildItem amd64/*.whl | Select-Object -First 1
|
||||
python tools/.github/scripts/repack_sqlite_vec.py $src.FullName sv/dist/vec0.dll wheel
|
||||
Get-ChildItem wheel
|
||||
|
||||
- name: Load it through sqlite3 and run a vec0 query
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
$whl = Get-ChildItem wheel/*.whl | Select-Object -First 1
|
||||
python -m venv smoke
|
||||
smoke\Scripts\python.exe -m pip install --quiet $whl.FullName
|
||||
smoke\Scripts\python.exe -c @"
|
||||
import sqlite3, sqlite_vec, sysconfig
|
||||
print('platform', sysconfig.get_platform(), 'sqlite', sqlite3.sqlite_version)
|
||||
db = sqlite3.connect(':memory:')
|
||||
db.enable_load_extension(True)
|
||||
sqlite_vec.load(db)
|
||||
db.enable_load_extension(False)
|
||||
print('vec_version', db.execute('select vec_version()').fetchone()[0])
|
||||
db.execute('create virtual table v using vec0(embedding float[4])')
|
||||
for rid, vec in [(1, [1, 0, 0, 0]), (2, [0, 1, 0, 0]), (3, [0.9, 0.1, 0, 0])]:
|
||||
db.execute('insert into v(rowid, embedding) values (?, ?)', (rid, sqlite_vec.serialize_float32(vec)))
|
||||
got = db.execute('select rowid, distance from v where embedding match ? order by distance limit 2', (sqlite_vec.serialize_float32([1, 0, 0, 0]),)).fetchall()
|
||||
print('knn', got)
|
||||
assert [r[0] for r in got] == [1, 3], got
|
||||
"@
|
||||
if ($LASTEXITCODE -ne 0) { exit 1 }
|
||||
|
||||
- uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: unsigned-sqlite-vec
|
||||
path: wheel/*.whl
|
||||
if-no-files-found: error
|
||||
|
||||
sign:
|
||||
name: Sign the native binaries
|
||||
needs: [build-pyarrow, build-sqlite-vec]
|
||||
# x64, deliberately. The wheels are ARM64 but Authenticode does not care what the host
|
||||
# is, and trusted-signing-cli is an x64 binary whose digest is pinned for windows-latest
|
||||
# in release-desktop.yml. Signing here runs the same verified bytes rather than the same
|
||||
# bytes under emulation.
|
||||
runs-on: windows-latest
|
||||
timeout-minutes: 45
|
||||
# The Azure Trusted Signing secrets are read by this job and no other. Naming the
|
||||
# environment is what makes them gateable; add required reviewers in repo settings to
|
||||
# close the gap, exactly as release-desktop.yml documents.
|
||||
environment: release-signing
|
||||
steps:
|
||||
- name: Harden runner (audit)
|
||||
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: unsigned-*
|
||||
merge-multiple: true
|
||||
path: unsigned
|
||||
|
||||
- name: List what was built
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
$wheels = Get-ChildItem unsigned/*.whl
|
||||
$wheels | ForEach-Object { "{0,-52} {1,8:N0} KB" -f $_.Name, ($_.Length / 1KB) }
|
||||
# Four pyarrow (cp311-cp314) and one sqlite-vec. A short matrix leg would otherwise
|
||||
# publish a wheelhouse missing an interpreter, which reads as "no wheel" downstream.
|
||||
if ($wheels.Count -ne 5) { throw "expected 5 wheels, got $($wheels.Count)" }
|
||||
|
||||
# Same pin as release-desktop.yml. The URL is reproducibility, the digest is the gate:
|
||||
# a release asset can be replaced after upload, and a replaced one must fail the run
|
||||
# rather than sign with an unknown binary. Bump both together.
|
||||
- name: Install trusted-signing-cli
|
||||
shell: pwsh
|
||||
env:
|
||||
TRUSTED_SIGNING_CLI_URL: "https://github.com/Levminer/artifact-signing-cli/releases/download/0.10.0/trusted-signing-cli.exe"
|
||||
TRUSTED_SIGNING_CLI_SHA256: "8c9d750ca582891a7433925dcf302d5d3761fbed757b46c4bf5b417562dccb0e"
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
$dir = Join-Path $env:RUNNER_TEMP "trusted-signing-cli"
|
||||
New-Item -ItemType Directory -Force -Path $dir | Out-Null
|
||||
$dest = Join-Path $dir "trusted-signing-cli.exe"
|
||||
Invoke-WebRequest -Uri $env:TRUSTED_SIGNING_CLI_URL -OutFile $dest -MaximumRetryCount 3 -RetryIntervalSec 5
|
||||
$actual = (Get-FileHash -Path $dest -Algorithm SHA256).Hash.ToLower()
|
||||
$expected = $env:TRUSTED_SIGNING_CLI_SHA256.ToLower()
|
||||
if ($actual -ne $expected) {
|
||||
Remove-Item -Force $dest
|
||||
Write-Output "::error::trusted-signing-cli digest mismatch, refusing to sign with an unverified binary. Expected $expected, got $actual."
|
||||
exit 1
|
||||
}
|
||||
Write-Output "verified trusted-signing-cli sha256=$actual"
|
||||
$dir | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
|
||||
|
||||
- name: Verify trusted-signing-cli
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
$cli = Get-Command trusted-signing-cli -ErrorAction SilentlyContinue
|
||||
if (-not $cli) {
|
||||
Write-Output "::error::trusted-signing-cli is not on PATH. Check the install step above."
|
||||
exit 1
|
||||
}
|
||||
# The helper invokes it by bare name, so whatever PATH resolves is what signs. Only
|
||||
# the digest-verified copy may do that.
|
||||
$verified = [IO.Path]::GetFullPath((Join-Path (Join-Path $env:RUNNER_TEMP "trusted-signing-cli") "trusted-signing-cli.exe"))
|
||||
if ([IO.Path]::GetFullPath($cli.Source) -ne $verified) {
|
||||
Write-Output "::error::trusted-signing-cli resolved to $($cli.Source), not the verified $verified."
|
||||
exit 1
|
||||
}
|
||||
try { & $cli.Source --version }
|
||||
catch {
|
||||
$detail = ($_.Exception.Message -replace '\s+', ' ').Trim()
|
||||
Write-Output "::error::trusted-signing-cli could not be started. Underlying error: $detail"
|
||||
exit 1
|
||||
}
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-Output "::error::trusted-signing-cli is on PATH but exited $LASTEXITCODE."
|
||||
exit 1
|
||||
}
|
||||
|
||||
- name: Sign every native binary inside the wheels
|
||||
shell: pwsh
|
||||
env:
|
||||
AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}
|
||||
AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }}
|
||||
AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
|
||||
AZURE_TRUSTED_SIGNING_ACCOUNT_NAME: ${{ secrets.AZURE_TRUSTED_SIGNING_ACCOUNT_NAME }}
|
||||
AZURE_CERTIFICATE_PROFILE_NAME: ${{ secrets.AZURE_CERTIFICATE_PROFILE_NAME }}
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
python .github/scripts/sign_wheel_binaries.py (Get-ChildItem unsigned/*.whl | ForEach-Object { $_.FullName }) `
|
||||
--out signed `
|
||||
--signer "pwsh -NoProfile -File studio/src-tauri/windows/sign-with-trusted-signing.ps1 -Path"
|
||||
if ($LASTEXITCODE -ne 0) { Write-Output "::error::signing failed"; exit 1 }
|
||||
|
||||
# Independent of the signing tool's own report: unpack what will actually be published
|
||||
# and ask Windows whether each signature validates on this machine.
|
||||
- name: Verify the published wheels validate
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
$bad = 0; $ok = 0; $ms = 0
|
||||
foreach ($whl in Get-ChildItem signed/*.whl) {
|
||||
$dir = Join-Path $env:RUNNER_TEMP ("verify-" + $whl.BaseName)
|
||||
Remove-Item -Recurse -Force $dir -ErrorAction SilentlyContinue
|
||||
Expand-Archive -Path $whl.FullName -DestinationPath $dir
|
||||
foreach ($pe in Get-ChildItem $dir -Recurse -Include *.pyd, *.dll) {
|
||||
$sig = Get-AuthenticodeSignature $pe.FullName
|
||||
if ($sig.Status -ne 'Valid') {
|
||||
Write-Output "::error::$($whl.Name) :: $($pe.Name) is $($sig.Status)"
|
||||
$bad++
|
||||
continue
|
||||
}
|
||||
if ($sig.SignerCertificate.Subject -match 'Unsloth') { $ok++ } else { $ms++ }
|
||||
}
|
||||
}
|
||||
Write-Output "signed by us: $ok, upstream-signed (Microsoft redistributables): $ms, invalid: $bad"
|
||||
if ($bad -gt 0) { exit 1 }
|
||||
if ($ok -eq 0) { Write-Output "::error::nothing carries our signature"; exit 1 }
|
||||
|
||||
# RECORD is rewritten by the signing step, so this proves the wheel is still installable
|
||||
# and that its own manifest agrees with its contents.
|
||||
- name: Install each wheel and check its RECORD
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
python -c @"
|
||||
import base64, hashlib, pathlib, sys, zipfile
|
||||
bad = 0
|
||||
for whl in sorted(pathlib.Path('signed').glob('*.whl')):
|
||||
z = zipfile.ZipFile(whl)
|
||||
record = next(n for n in z.namelist() if n.endswith('.dist-info/RECORD'))
|
||||
rows = {}
|
||||
for line in z.read(record).decode().splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
name, digest, _, = (line.split(',') + ['', ''])[:3]
|
||||
rows[name] = digest
|
||||
for name in z.namelist():
|
||||
if name == record:
|
||||
continue
|
||||
want = rows.get(name)
|
||||
if not want:
|
||||
print('MISSING from RECORD:', whl.name, name); bad += 1; continue
|
||||
got = 'sha256=' + base64.urlsafe_b64encode(hashlib.sha256(z.read(name)).digest()).rstrip(b'=').decode()
|
||||
if got != want:
|
||||
print('DIGEST MISMATCH:', whl.name, name); bad += 1
|
||||
print(whl.name, 'RECORD ok,', len(rows), 'entries')
|
||||
sys.exit(1 if bad else 0)
|
||||
"@
|
||||
if ($LASTEXITCODE -ne 0) { exit 1 }
|
||||
|
||||
- name: Build index.txt and SHA256SUMS
|
||||
shell: pwsh
|
||||
run: |
|
||||
$ErrorActionPreference = "Stop"
|
||||
$wheels = Get-ChildItem signed/*.whl | Sort-Object Name
|
||||
# Flat find-links listing: install.ps1 reads this, then fetches each name beside it.
|
||||
($wheels | ForEach-Object { $_.Name }) -join "`n" | Set-Content signed/index.txt -Encoding ascii
|
||||
($wheels | ForEach-Object { "{0} {1}" -f (Get-FileHash $_.FullName -Algorithm SHA256).Hash.ToLower(), $_.Name }) -join "`n" |
|
||||
Set-Content signed/SHA256SUMS -Encoding ascii
|
||||
Get-Content signed/index.txt
|
||||
Get-Content signed/SHA256SUMS
|
||||
|
||||
- uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: woa-wheelhouse-signed
|
||||
path: |
|
||||
signed/*.whl
|
||||
signed/index.txt
|
||||
signed/SHA256SUMS
|
||||
if-no-files-found: error
|
||||
|
||||
publish:
|
||||
name: Publish the wheelhouse
|
||||
needs: sign
|
||||
if: inputs.publish
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: write
|
||||
steps:
|
||||
- name: Harden runner (audit)
|
||||
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: woa-wheelhouse-signed
|
||||
path: wheelhouse
|
||||
|
||||
# A release, not a branch: GitHub reports a per-asset `digest` through the release API,
|
||||
# which is the same verification the installer already applies to the llama.cpp bundles.
|
||||
- name: Create or update the release
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
TAG: ${{ inputs.release_tag }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if ! gh release view "$TAG" --repo "$GITHUB_REPOSITORY" >/dev/null 2>&1; then
|
||||
gh release create "$TAG" \
|
||||
--repo "$GITHUB_REPOSITORY" \
|
||||
--title "Windows on ARM wheelhouse" \
|
||||
--notes "Signed win_arm64 wheels for the native Windows on ARM installer: pyarrow (cp311-cp314) and sqlite-vec. Neither publishes a win_arm64 wheel on PyPI. The native binaries inside each wheel are Authenticode signed through Azure Trusted Signing; SHA256SUMS lists the digest of every wheel as published."
|
||||
fi
|
||||
# --clobber: this tag is a rolling wheelhouse, and a rebuilt wheel for the same
|
||||
# interpreter is meant to replace its predecessor rather than fail the upload.
|
||||
gh release upload "$TAG" --repo "$GITHUB_REPOSITORY" --clobber \
|
||||
wheelhouse/*.whl wheelhouse/index.txt wheelhouse/SHA256SUMS
|
||||
|
||||
- name: Report the wheelhouse URL
|
||||
shell: bash
|
||||
env:
|
||||
TAG: ${{ inputs.release_tag }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
url="https://github.com/$GITHUB_REPOSITORY/releases/download/$TAG"
|
||||
{
|
||||
echo "### Windows on ARM wheelhouse published"
|
||||
echo
|
||||
echo 'Point the installer at it with:'
|
||||
echo
|
||||
echo '```powershell'
|
||||
echo "\$env:UNSLOTH_WOA_WHEELHOUSE = \"$url\""
|
||||
echo '```'
|
||||
echo
|
||||
echo '| wheel | sha256 |'
|
||||
echo '| --- | --- |'
|
||||
while read -r digest name; do echo "| $name | \`$digest\` |"; done < wheelhouse/SHA256SUMS
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
4
.gitignore
vendored
4
.gitignore
vendored
|
|
@ -255,6 +255,10 @@ Speech_12dB_s16.flac
|
|||
# so a test run or an Unsloth start rewrites it and dirties the checkout.
|
||||
.studio-test-root/
|
||||
|
||||
# _filter_requirements() in studio/install_python_stack.py writes its skip-filtered copy beside
|
||||
# the source file so relative -r/-c includes still resolve. A killed install leaves them behind.
|
||||
studio/backend/requirements/.*-filtered-*.txt
|
||||
|
||||
# launch.py --outdir scratch from a local run
|
||||
/out/
|
||||
|
||||
|
|
|
|||
1806
install.ps1
1806
install.ps1
File diff suppressed because it is too large
Load diff
|
|
@ -149,7 +149,9 @@ studio = [
|
|||
"pydantic==2.13.4",
|
||||
"packaging==26.3",
|
||||
"matplotlib==3.10.9",
|
||||
"pandas==2.3.3",
|
||||
# win_arm64 pandas wheels start at 3.0; the 3.11 floor is pandas 3's own requires-python.
|
||||
"pandas==2.3.3 ; sys_platform != 'win32' or platform_machine != 'ARM64' or python_version < '3.11'",
|
||||
"pandas>=3.0,<4 ; sys_platform == 'win32' and platform_machine == 'ARM64' and python_version >= '3.11'",
|
||||
"nest_asyncio==1.6.0",
|
||||
"datasets==4.3.0",
|
||||
"pyjwt==2.13.0",
|
||||
|
|
@ -161,15 +163,21 @@ studio = [
|
|||
"diceware==1.0.1",
|
||||
"ddgs==9.14.4 ; python_version >= '3.10'",
|
||||
"ddgs==9.8.0 ; python_version < '3.10'",
|
||||
"cryptography>=42.0.0",
|
||||
"cryptography>=42.0.0; sys_platform != 'win32' or platform_machine != 'ARM64'",
|
||||
# Windows on ARM: 46.0.7 publishes no win_arm64 wheel and the sdist needs Rust.
|
||||
"cryptography>=42.0.0,<=46.0.3; sys_platform == 'win32' and platform_machine == 'ARM64'",
|
||||
"boto3>=1.34.0",
|
||||
"httpx>=0.27.0",
|
||||
"fastmcp>=3.0.2",
|
||||
"gguf==0.19.0 ; python_version >= '3.10'",
|
||||
"gguf==0.18.0 ; python_version < '3.10'",
|
||||
"av>=12.3.0",
|
||||
"sqlite-vec==0.1.9",
|
||||
"pymupdf==1.27.2.3",
|
||||
# No win_arm64 wheel and no sdist either, so it cannot be floored or built. Dropped rather
|
||||
# than pinned: the RAG store already treats a missing sqlite_vec as "RAG unavailable".
|
||||
"sqlite-vec==0.1.9 ; sys_platform != 'win32' or platform_machine != 'ARM64'",
|
||||
# Mirrors studio.txt: pymupdf publishes a win_arm64 wheel from 1.28 only.
|
||||
"pymupdf==1.27.2.3 ; sys_platform != 'win32' or platform_machine != 'ARM64'",
|
||||
"pymupdf>=1.28.2 ; sys_platform == 'win32' and platform_machine == 'ARM64'",
|
||||
"pymupdf4llm==0.3.4",
|
||||
"python-docx==1.2.0",
|
||||
"markdown-it-py>=3.0",
|
||||
|
|
@ -177,7 +185,9 @@ studio = [
|
|||
|
||||
triton = [
|
||||
"triton>=3.0.0 ; ('linux' in sys_platform)",
|
||||
# triton-windows has win_arm64 wheels from 3.8.0.post28, so a floor works here, not a marker.
|
||||
"triton-windows ; (sys_platform == 'win32') and (platform_machine == 'AMD64' or platform_machine == 'x86_64')",
|
||||
"triton-windows>=3.8.0.post28 ; (sys_platform == 'win32') and platform_machine == 'ARM64'",
|
||||
]
|
||||
|
||||
huggingfacenotorch = [
|
||||
|
|
@ -194,7 +204,9 @@ huggingfacenotorch = [
|
|||
"accelerate>=0.34.1",
|
||||
"peft>=0.18.0,!=0.11.0",
|
||||
"huggingface_hub>=0.34.0",
|
||||
"hf_transfer",
|
||||
# A Rust extension with no win_arm64 wheel. huggingface_hub imports it only under
|
||||
# HF_HUB_ENABLE_HF_TRANSFER=1, and hf_xet ships win_arm64 and covers fast downloads.
|
||||
"hf_transfer ; sys_platform != 'win32' or platform_machine != 'ARM64'",
|
||||
# 0.39 is the floor the diffusion stack already assumes: Krea2Pipeline (the Krea 2
|
||||
# workflow raises without it), the cache_context child-registry behavior, and the
|
||||
# Flux2 / Z-Image pipelines. An unversioned entry let an upgrade keep an older
|
||||
|
|
@ -287,8 +299,11 @@ huggingface = [
|
|||
# handles instead.
|
||||
windows = [
|
||||
"unsloth[huggingface]",
|
||||
# bitsandbytes ships a win_arm64 wheel from 0.50.0, so this row covers it.
|
||||
"bitsandbytes>=0.45.5,!=0.46.0,!=0.48.0 ; (sys_platform == 'win32')",
|
||||
"xformers>=0.0.22.post7 ; (sys_platform == 'win32')",
|
||||
# No win_arm64 wheel and no sdist that builds without CUDA's Windows-ARM toolchain, so this
|
||||
# row would fail the whole extra. Torch's SDPA is the fallback.
|
||||
"xformers>=0.0.22.post7 ; (sys_platform == 'win32') and platform_machine != 'ARM64'",
|
||||
]
|
||||
base = [
|
||||
"unsloth[huggingface]",
|
||||
|
|
|
|||
|
|
@ -12,7 +12,8 @@ description = "Local Data Designer unstructured seed reader plugin"
|
|||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"data-designer-engine>=0.5.4,<0.6",
|
||||
"pandas>=2,<3",
|
||||
# <4: Windows on ARM installs pandas 3 (constraints.txt), and this plugin only reads and writes parquet.
|
||||
"pandas>=2,<4",
|
||||
"pymupdf>=1.24.0",
|
||||
"pymupdf4llm>=0.0.17",
|
||||
"mammoth>=1.8.0",
|
||||
|
|
|
|||
|
|
@ -2,7 +2,9 @@
|
|||
flatbuffers==25.12.19
|
||||
# Also needed by sentence_transformers (installed with --no-deps in extras-no-deps.txt);
|
||||
# librosa pulls it in too, but is skipped in no-torch mode.
|
||||
scikit-learn==1.7.1
|
||||
# scikit-learn publishes win_arm64 wheels from 1.8 only; 1.7.1 would source-build there.
|
||||
scikit-learn==1.7.1; sys_platform != "win32" or platform_machine != "ARM64"
|
||||
scikit-learn>=1.8.0; sys_platform == "win32" and platform_machine == "ARM64"
|
||||
|
||||
# Additional extras
|
||||
jiwer==4.0.0 # WER/CER metrics for vision OCR save-merge benchmarks
|
||||
|
|
@ -18,7 +20,9 @@ openai-whisper==20250625
|
|||
# av<16 because 16+ builds its macOS arm64 wheels against macosx_14_0 and so has
|
||||
# no installable wheel on macOS 13. Pinning past the cap makes that leg
|
||||
# unsatisfiable. Lift both together.
|
||||
av==15.1.0
|
||||
# See constraints.txt: win_arm64 av wheels start at 17.0.0.
|
||||
av==15.1.0; sys_platform != "win32" or platform_machine != "ARM64"
|
||||
av>=17.0.0; sys_platform == "win32" and platform_machine == "ARM64"
|
||||
uroman==1.3.1.1 # 4.0 MB - used for Outetts.
|
||||
# 19.9 MB - used for Outetts. No release ships a macOS cp314 wheel (0.996.12 added
|
||||
# cp314 for Linux and win_amd64 only), so a 3.14 macOS host has no binary candidate
|
||||
|
|
|
|||
|
|
@ -48,7 +48,8 @@ accelerate>=0.34.1
|
|||
peft>=0.18.0,!=0.11.0
|
||||
huggingface_hub>=1.23.0,<2.0; python_version >= "3.10"
|
||||
huggingface_hub==0.36.2; python_version < "3.10"
|
||||
hf_transfer==0.1.9
|
||||
# No win_arm64 wheel and this file installs --no-deps, so pip would need cargo plus MSVC.
|
||||
hf_transfer==0.1.9; sys_platform != "win32" or platform_machine != "ARM64"
|
||||
# diffusers: see diffusers-pin.txt, installed by its own step (this file is
|
||||
# installed --no-deps, and the pin has to apply on the torch path too).
|
||||
|
||||
|
|
@ -96,8 +97,12 @@ pillow==11.3.0; python_version < "3.10"
|
|||
|
||||
# RAG store + document parsing, mirroring studio.txt. Pinned here because
|
||||
# this file installs --no-deps; without them Unsloth runs with RAG disabled.
|
||||
sqlite-vec==0.1.9
|
||||
pymupdf==1.27.2.3
|
||||
# sqlite-vec publishes win_amd64 only and no sdist; the RAG store treats a missing sqlite_vec as
|
||||
# "RAG unavailable". Same marker as studio.txt: this file is installed directly on an update.
|
||||
sqlite-vec==0.1.9; sys_platform != "win32" or platform_machine != "ARM64"
|
||||
# pymupdf publishes a win_arm64 wheel from 1.28 only.
|
||||
pymupdf==1.27.2.3; sys_platform != "win32" or platform_machine != "ARM64"
|
||||
pymupdf>=1.28.2; sys_platform == "win32" and platform_machine == "ARM64"
|
||||
# 0.3.x keeps pymupdf-layout (which pulls onnxruntime) an optional extra; the
|
||||
# lockstep 1.27.x line makes it a hard dep we do not need for to_markdown().
|
||||
pymupdf4llm==0.3.4
|
||||
|
|
|
|||
|
|
@ -13,7 +13,10 @@ huggingface-hub==0.36.2; python_version < "3.10"
|
|||
|
||||
# Unsloth stack
|
||||
datasets==4.3.0
|
||||
pyarrow==23.0.1
|
||||
# pyarrow has never published a win_arm64 wheel (apache/arrow#48539). The floor below is the
|
||||
# floor of the wheels install.ps1 stages, not of anything upstream ships.
|
||||
pyarrow==23.0.1; sys_platform != "win32" or platform_machine != "ARM64"
|
||||
pyarrow>=21.0.0; sys_platform == "win32" and platform_machine == "ARM64"
|
||||
|
||||
# FastMCP compat
|
||||
fastmcp>=3.0.2
|
||||
|
|
@ -34,7 +37,9 @@ anyio<4.14.0
|
|||
# guards the call. install_python_stack.py repairs an install that is already past the cap.
|
||||
accelerate<1.15.0; sys_platform == "win32"
|
||||
|
||||
pandas==2.3.3
|
||||
# pandas publishes win_arm64 wheels from 3.0 only; 2.3.x would source-build there.
|
||||
pandas==2.3.3; sys_platform != "win32" or platform_machine != "ARM64" or python_version < "3.11"
|
||||
pandas>=3.0,<4; sys_platform == "win32" and platform_machine == "ARM64" and python_version >= "3.11"
|
||||
|
||||
# av (PyAV) 16+ builds its macOS arm64 wheels against macosx_14_0, so on macOS 13 none
|
||||
# are installable and the resolver falls back to a source build, which needs FFmpeg
|
||||
|
|
@ -45,10 +50,16 @@ pandas==2.3.3
|
|||
# The remaining sdist-only macOS defaults are pure Python, hence allowlisted in
|
||||
# .github/scripts/clean-machine-assert.sh instead; cryptography below is the one
|
||||
# other package that would compile.
|
||||
av<16
|
||||
av<16; sys_platform != "win32" or platform_machine != "ARM64"
|
||||
# The opposite case: av publishes win_arm64 wheels from 17.0.0 only (cp311-abi3,
|
||||
# plus a cp314t; there is no 3.13t wheel on any release).
|
||||
av>=17.0.0; sys_platform == "win32" and platform_machine == "ARM64"
|
||||
|
||||
# cryptography 49.0.0 dropped the macosx_10_9_universal2 wheel for arm64-only, so
|
||||
# x86_64 macOS has no wheel and builds the sdist, needing Rust plus a working
|
||||
# linker. 48.0.1 is the newest release with a universal2 wheel. Lift when
|
||||
# cryptography ships an x86_64-capable macOS wheel again.
|
||||
cryptography<49; sys_platform == "darwin" and platform_machine == "x86_64"
|
||||
|
||||
# 46.0.3 is the newest cryptography with a win_arm64 wheel; raise the cap when upstream resumes.
|
||||
cryptography<=46.0.3; sys_platform == "win32" and platform_machine == "ARM64"
|
||||
|
|
|
|||
|
|
@ -12,8 +12,10 @@ packaging==26.3
|
|||
matplotlib==3.10.9
|
||||
# Held at 2.3.3 rather than the 3.0.x line: single-env/constraints.txt pins
|
||||
# pandas==2.3.3, and 2.3.3 is also the newest release that still installs on
|
||||
# 3.9. Bump both together.
|
||||
pandas==2.3.3
|
||||
# 3.9. Bump both together. Windows on ARM is the exception: win_arm64 pandas wheels
|
||||
# start at 3.0, and the 3.11 floor there is pandas 3's own requires-python.
|
||||
pandas==2.3.3; sys_platform != "win32" or platform_machine != "ARM64" or python_version < "3.11"
|
||||
pandas>=3.0,<4; sys_platform == "win32" and platform_machine == "ARM64" and python_version >= "3.11"
|
||||
nest_asyncio==1.6.0
|
||||
datasets==4.3.0
|
||||
pyjwt==2.13.0
|
||||
|
|
@ -37,7 +39,10 @@ structlog>=24.1.0
|
|||
diceware==1.0.1
|
||||
ddgs==9.14.4; python_version >= "3.10"
|
||||
ddgs==9.8.0; python_version < "3.10"
|
||||
cryptography>=42.0.0
|
||||
cryptography>=42.0.0; sys_platform != "win32" or platform_machine != "ARM64"
|
||||
# Windows on ARM: 46.0.7 has no win_arm64 wheel and its sdist needs Rust. Matches
|
||||
# constraints.txt, which a plain pip install of the extra never reaches.
|
||||
cryptography>=42.0.0,<=46.0.3; sys_platform == "win32" and platform_machine == "ARM64"
|
||||
boto3>=1.34.0 # optional: S3 dataset loading
|
||||
httpx>=0.27.0
|
||||
fastmcp>=3.0.2
|
||||
|
|
@ -53,8 +58,12 @@ gguf==0.18.0; python_version < "3.10"
|
|||
av>=12.3.0
|
||||
# RAG (knowledge bases, hybrid retrieval). sentence-transformers lives in
|
||||
# extras-no-deps.txt; these add the lexical+dense store and document parsing.
|
||||
sqlite-vec==0.1.9
|
||||
pymupdf==1.27.2.3
|
||||
# sqlite-vec publishes win_amd64 only and no sdist, so there is nothing to floor to. The RAG
|
||||
# store treats a missing sqlite_vec as "RAG unavailable".
|
||||
sqlite-vec==0.1.9; sys_platform != "win32" or platform_machine != "ARM64"
|
||||
# pymupdf publishes a win_arm64 wheel from 1.28 only.
|
||||
pymupdf==1.27.2.3; sys_platform != "win32" or platform_machine != "ARM64"
|
||||
pymupdf>=1.28.2; sys_platform == "win32" and platform_machine == "ARM64"
|
||||
# 0.3.x keeps pymupdf-layout (which pulls onnxruntime) an optional extra; the
|
||||
# lockstep 1.27.x line makes it a hard dep we do not need for to_markdown().
|
||||
pymupdf4llm==0.3.4
|
||||
|
|
|
|||
|
|
@ -13,6 +13,8 @@ INSTALL_KIND_BACKENDS: dict[str, str] = {
|
|||
"linux-cuda": "cuda",
|
||||
"linux-arm64-cuda": "cuda",
|
||||
"windows-cuda": "cuda",
|
||||
# Distinct from windows-cuda: the arm64 bundle ships its own ARM64 cudart/cublas.
|
||||
"windows-arm64-cuda": "cuda",
|
||||
"linux-rocm": "rocm",
|
||||
"windows-hip": "rocm",
|
||||
"windows-rocm": "rocm",
|
||||
|
|
|
|||
|
|
@ -776,10 +776,52 @@ def checkout_friendly_ref(ref_kind: str | None, ref: str | None) -> str | None:
|
|||
return ref
|
||||
|
||||
|
||||
def windows_cuda_upstream_asset_names(llama_tag: str, runtime: str) -> list[str]:
|
||||
# Upstream names these llama-<tag>-bin-win-cuda-<runtime>-<arch>.zip, arch in {x64, arm64}.
|
||||
WINDOWS_CUDA_ARCHS = ("x64", "arm64")
|
||||
_WINDOWS_CUDA_INSTALL_KINDS = ("windows-cuda", "windows-arm64-cuda")
|
||||
|
||||
|
||||
def _upstream_arm64_cuda_allowed() -> bool:
|
||||
"""May a Windows ARM64 NVIDIA host install a CUDA llama.cpp bundle at all? Defaults
|
||||
to yes, because the only other answer is a CPU-only build on a CUDA machine.
|
||||
UNSLOTH_LLAMA_ARM64_CUDA=0/false/no/off opts out.
|
||||
|
||||
This governs every ARM64 CUDA bundle, published or upstream. It started narrower --
|
||||
only the unverified upstream bundle, since that is the one with a provenance question
|
||||
-- but what the flag promises the user is a CPU bundle, and scoping it to the
|
||||
unverified case meant the escape hatch would switch itself off the day the fork
|
||||
published an approved windows-arm64-cuda artifact. A setting that silently stops
|
||||
working on someone else's release is worse than one that is merely blunt.
|
||||
"""
|
||||
raw = (os.environ.get("UNSLOTH_LLAMA_ARM64_CUDA") or "").strip().lower()
|
||||
return raw not in {"0", "false", "no", "off"}
|
||||
|
||||
|
||||
def windows_cuda_arch_for_host(host: HostInfo) -> str:
|
||||
"""CUDA prebuilt arch token for a Windows host: arm64 on Windows on ARM, else x64.
|
||||
|
||||
``is_arm64`` is also true for macOS arm64 and Linux aarch64, so the Windows test
|
||||
is part of the question rather than the caller's job. Every caller today is already
|
||||
inside a Windows branch; this keeps the answer right if one ever is not.
|
||||
"""
|
||||
return "arm64" if (host.is_windows and host.is_arm64) else "x64"
|
||||
|
||||
|
||||
def windows_cuda_install_kind_for_arch(arch: str) -> str:
|
||||
"""install_kind for a Windows CUDA prebuilt of `arch`. Kept separate from the x64
|
||||
kind so the runtime-overlay, health-check and marker tables can tell the two apart
|
||||
(their payloads differ: the arm64 bundle carries no cuBLASLt-for-x64 DLL set)."""
|
||||
return "windows-arm64-cuda" if arch == "arm64" else "windows-cuda"
|
||||
|
||||
|
||||
def windows_cuda_upstream_asset_names(
|
||||
llama_tag: str,
|
||||
runtime: str,
|
||||
arch: str = "x64",
|
||||
) -> list[str]:
|
||||
return [
|
||||
f"llama-{llama_tag}-bin-win-cuda-{runtime}-x64.zip",
|
||||
f"cudart-llama-bin-win-cuda-{runtime}-x64.zip",
|
||||
f"llama-{llama_tag}-bin-win-cuda-{runtime}-{arch}.zip",
|
||||
f"cudart-llama-bin-win-cuda-{runtime}-{arch}.zip",
|
||||
]
|
||||
|
||||
|
||||
|
|
@ -787,29 +829,35 @@ def windows_cuda_asset_aliases(
|
|||
asset_name: str, *, compatibility_tag: str | None = None
|
||||
) -> list[str]:
|
||||
aliases: list[str] = []
|
||||
_arch_re = "|".join(WINDOWS_CUDA_ARCHS)
|
||||
legacy_match = re.fullmatch(
|
||||
r"llama-(?P<tag>[^/]+)-bin-win-cuda-(?P<runtime>\d+\.\d+)-x64\.zip",
|
||||
rf"llama-(?P<tag>[^/]+)-bin-win-cuda-(?P<runtime>\d+\.\d+)-(?P<arch>{_arch_re})\.zip",
|
||||
asset_name,
|
||||
)
|
||||
if legacy_match:
|
||||
runtime = legacy_match.group("runtime")
|
||||
aliases.append(f"cudart-llama-bin-win-cuda-{runtime}-x64.zip")
|
||||
arch = legacy_match.group("arch")
|
||||
aliases.append(f"cudart-llama-bin-win-cuda-{runtime}-{arch}.zip")
|
||||
if compatibility_tag:
|
||||
aliases.append(f"llama-{compatibility_tag}-bin-win-cuda-{runtime}-x64.zip")
|
||||
aliases.append(f"llama-{compatibility_tag}-bin-win-cuda-{runtime}-{arch}.zip")
|
||||
return aliases
|
||||
|
||||
current_match = re.fullmatch(
|
||||
r"cudart-llama-bin-win-cuda-(?P<runtime>\d+\.\d+)-x64\.zip",
|
||||
rf"cudart-llama-bin-win-cuda-(?P<runtime>\d+\.\d+)-(?P<arch>{_arch_re})\.zip",
|
||||
asset_name,
|
||||
)
|
||||
if current_match and compatibility_tag:
|
||||
runtime = current_match.group("runtime")
|
||||
aliases.append(f"llama-{compatibility_tag}-bin-win-cuda-{runtime}-x64.zip")
|
||||
arch = current_match.group("arch")
|
||||
aliases.append(f"llama-{compatibility_tag}-bin-win-cuda-{runtime}-{arch}.zip")
|
||||
return aliases
|
||||
|
||||
|
||||
def _published_windows_cuda_runtime(
|
||||
upstream_assets: dict[str, str], major: int, driver: tuple[int, int] | None
|
||||
upstream_assets: dict[str, str],
|
||||
major: int,
|
||||
driver: tuple[int, int] | None,
|
||||
arch: str = "x64",
|
||||
) -> str | None:
|
||||
"""Highest cuda-<major>.<minor> published upstream that `driver` can run by
|
||||
default CUDA compatibility, i.e. (major, minor) <= driver. None if nothing
|
||||
|
|
@ -820,7 +868,7 @@ def _published_windows_cuda_runtime(
|
|||
return None
|
||||
best: int | None = None
|
||||
for name in upstream_assets:
|
||||
m = re.search(r"-bin-win-cuda-(\d+)\.(\d+)-x64\.zip$", name)
|
||||
m = re.search(rf"-bin-win-cuda-(\d+)\.(\d+)-{re.escape(arch)}\.zip$", name)
|
||||
if m and int(m.group(1)) == major:
|
||||
minor = int(m.group(2))
|
||||
if (major, minor) <= driver and (best is None or minor > best):
|
||||
|
|
@ -1151,6 +1199,20 @@ def direct_upstream_release_plan(
|
|||
)
|
||||
)
|
||||
elif host.is_windows and host.is_arm64:
|
||||
# Upstream ships -arm64 CUDA bundles for these parts; try them first, CPU bundle still last.
|
||||
if host.has_usable_nvidia and _upstream_arm64_cuda_allowed():
|
||||
torch_preference = detect_torch_cuda_runtime_preference(host)
|
||||
attempts.extend(
|
||||
windows_cuda_attempts(
|
||||
host,
|
||||
release_tag,
|
||||
assets,
|
||||
torch_preference.runtime_line,
|
||||
torch_preference.selection_log,
|
||||
arch = "arm64",
|
||||
)
|
||||
)
|
||||
attempts[:] = _drop_blackwell_incapable_windows_cuda(host, attempts)
|
||||
# Upstream ggml-org/llama.cpp ships llama-bNNNN-bin-win-cpu-arm64.zip
|
||||
# (visible in the b9334 release manifest). Without this branch the
|
||||
# selector returned 0 attempts and the installer fell back to a
|
||||
|
|
@ -1376,6 +1438,22 @@ def detected_linux_runtime_lines() -> tuple[list[str], dict[str, list[str]]]:
|
|||
|
||||
|
||||
release_asset_map = _core.release_asset_map
|
||||
release_asset_digests = _core.release_asset_digests
|
||||
|
||||
|
||||
def github_release_asset_digests(repo: str, tag: str) -> dict[str, str]:
|
||||
"""asset name -> sha256, from the release GitHub already serves us.
|
||||
|
||||
Separate call rather than a second return value from github_release_assets: that one is
|
||||
mocked in a dozen suites and used on every platform, while this is read on one branch
|
||||
only. Any failure is an empty mapping, which that branch reads as "refuse", so a rate
|
||||
limit or an outage costs the CUDA bundle rather than installing it unchecked.
|
||||
"""
|
||||
try:
|
||||
return release_asset_digests(github_release(repo, tag))
|
||||
except Exception as exc:
|
||||
log(f"could not read release asset digests for {repo}@{tag}: {exc}")
|
||||
return {}
|
||||
|
||||
|
||||
def parse_published_artifact(raw: Any) -> PublishedLlamaArtifact | None:
|
||||
|
|
@ -3103,8 +3181,14 @@ def windows_cuda_attempts(
|
|||
upstream_assets: dict[str, str],
|
||||
preferred_runtime_line: str | None,
|
||||
selection_preamble: Iterable[str] = (),
|
||||
arch: str | None = None,
|
||||
) -> list[AssetChoice]:
|
||||
if arch is None:
|
||||
arch = windows_cuda_arch_for_host(host)
|
||||
install_kind = windows_cuda_install_kind_for_arch(arch)
|
||||
selection_log = list(selection_preamble)
|
||||
if arch != "x64":
|
||||
selection_log.append(f"windows_cuda_selection: arch={arch}")
|
||||
driver_runtime = pick_windows_cuda_runtime(host)
|
||||
detected_runtime_lines, runtime_dirs = detected_windows_runtime_lines()
|
||||
compatible_runtime_lines = compatible_windows_runtime_lines(host)
|
||||
|
|
@ -3189,7 +3273,9 @@ def windows_cuda_attempts(
|
|||
# Track whatever minor llama.cpp actually ships for this major
|
||||
# (cuda13 -> 13.1, 13.3, ...). Skip the line when the release has no
|
||||
# matching asset instead of guessing a now-missing name.
|
||||
runtime = _published_windows_cuda_runtime(upstream_assets, major, host.driver_cuda_version)
|
||||
runtime = _published_windows_cuda_runtime(
|
||||
upstream_assets, major, host.driver_cuda_version, arch
|
||||
)
|
||||
if runtime is None:
|
||||
selection_log.append(
|
||||
f"windows_cuda_selection: no driver-supported asset for {runtime_line}"
|
||||
|
|
@ -3197,7 +3283,7 @@ def windows_cuda_attempts(
|
|||
continue
|
||||
selected_name = None
|
||||
asset_url = None
|
||||
for candidate_name in windows_cuda_upstream_asset_names(llama_tag, runtime):
|
||||
for candidate_name in windows_cuda_upstream_asset_names(llama_tag, runtime, arch):
|
||||
asset_url = upstream_assets.get(candidate_name)
|
||||
if asset_url:
|
||||
selected_name = candidate_name
|
||||
|
|
@ -3205,7 +3291,7 @@ def windows_cuda_attempts(
|
|||
if not asset_url or not selected_name:
|
||||
selection_log.append(
|
||||
"windows_cuda_selection: skip missing assets "
|
||||
+ ",".join(windows_cuda_upstream_asset_names(llama_tag, runtime))
|
||||
+ ",".join(windows_cuda_upstream_asset_names(llama_tag, runtime, arch))
|
||||
)
|
||||
continue
|
||||
# Pair the cudart bundle when upstream ships it. Without this
|
||||
|
|
@ -3215,7 +3301,7 @@ def windows_cuda_attempts(
|
|||
runtime_archive_name: str | None = None
|
||||
runtime_archive_url: str | None = None
|
||||
if selected_name.startswith("llama-"):
|
||||
cudart_name = f"cudart-llama-bin-win-cuda-{runtime}-x64.zip"
|
||||
cudart_name = f"cudart-llama-bin-win-cuda-{runtime}-{arch}.zip"
|
||||
cudart_url = upstream_assets.get(cudart_name)
|
||||
if cudart_url and cudart_url != asset_url:
|
||||
runtime_archive_name = cudart_name
|
||||
|
|
@ -3239,7 +3325,7 @@ def windows_cuda_attempts(
|
|||
name = selected_name,
|
||||
url = asset_url,
|
||||
source_label = "upstream",
|
||||
install_kind = "windows-cuda",
|
||||
install_kind = install_kind,
|
||||
runtime_line = runtime_line,
|
||||
runtime_name = runtime_archive_name,
|
||||
runtime_url = runtime_archive_url,
|
||||
|
|
@ -3252,13 +3338,15 @@ def windows_cuda_attempts(
|
|||
def _windows_cuda_attempt_covers_blackwell(
|
||||
attempt: AssetChoice, min_toolkit: tuple[int, int] = _BLACKWELL_MIN_TOOLKIT
|
||||
) -> bool:
|
||||
"""True if a windows-cuda attempt is Blackwell-capable (app bundles via
|
||||
"""True if a windows CUDA attempt is Blackwell-capable (app bundles via
|
||||
declared SMs; legacy upstream bundles via toolkit minor >= min_toolkit:
|
||||
12.8 for the family, 12.9 for sm_103/sm_121)."""
|
||||
if attempt.install_kind != "windows-cuda":
|
||||
if attempt.install_kind not in _WINDOWS_CUDA_INSTALL_KINDS:
|
||||
return False
|
||||
# Legacy bundle: the toolkit minor binds (12.4 cannot offload Blackwell).
|
||||
m = re.search(r"-bin-win-cuda-(\d+)\.(\d+)-x64\.zip$", attempt.name)
|
||||
m = re.search(
|
||||
rf"-bin-win-cuda-(\d+)\.(\d+)-(?:{'|'.join(WINDOWS_CUDA_ARCHS)})\.zip$", attempt.name
|
||||
)
|
||||
if m is not None:
|
||||
return (int(m.group(1)), int(m.group(2))) >= min_toolkit
|
||||
# App-named bundles carry no minor and declare their SM coverage directly.
|
||||
|
|
@ -3272,17 +3360,19 @@ _blackwell_min_toolkit_for_host = _core.blackwell_min_toolkit_for_host
|
|||
def _drop_blackwell_incapable_windows_cuda(
|
||||
host: HostInfo, attempts: list[AssetChoice]
|
||||
) -> list[AssetChoice]:
|
||||
"""On a Blackwell host, drop windows-cuda attempts that can't offload
|
||||
"""On a Blackwell host, drop windows CUDA attempts that can't offload
|
||||
Blackwell (e.g. cuda-12.4): they load and validate but run a slow non-native
|
||||
path (RTX 5090: 7.1 vs 551.2 tok/s on cuda-13.3). Non-cuda attempts pass
|
||||
through so the host can still fall back to an honest CPU install."""
|
||||
through so the host can still fall back to an honest CPU install. Windows on ARM
|
||||
is covered too: every NVIDIA part in that class (GB10 / N1X) is Blackwell, and its
|
||||
only CUDA bundles are cuda-13.x, so nothing legitimate is filtered there."""
|
||||
if not _host_is_blackwell(host):
|
||||
return attempts
|
||||
min_toolkit = _blackwell_min_toolkit_for_host(host)
|
||||
return [
|
||||
attempt
|
||||
for attempt in attempts
|
||||
if attempt.install_kind != "windows-cuda"
|
||||
if attempt.install_kind not in _WINDOWS_CUDA_INSTALL_KINDS
|
||||
or _windows_cuda_attempt_covers_blackwell(attempt, min_toolkit)
|
||||
]
|
||||
|
||||
|
|
@ -3292,10 +3382,14 @@ def published_windows_cuda_attempts(
|
|||
release: PublishedReleaseBundle,
|
||||
preferred_runtime_line: str | None,
|
||||
selection_preamble: Iterable[str] = (),
|
||||
arch: str | None = None,
|
||||
) -> list[AssetChoice]:
|
||||
if arch is None:
|
||||
arch = windows_cuda_arch_for_host(host)
|
||||
install_kind = windows_cuda_install_kind_for_arch(arch)
|
||||
selection_log = list(release.selection_log) + list(selection_preamble)
|
||||
published_artifacts = [
|
||||
artifact for artifact in release.artifacts if artifact.install_kind == "windows-cuda"
|
||||
artifact for artifact in release.artifacts if artifact.install_kind == install_kind
|
||||
]
|
||||
artifacts_by_runtime: dict[str, list[PublishedLlamaArtifact]] = {}
|
||||
for artifact in published_artifacts:
|
||||
|
|
@ -3310,8 +3404,10 @@ def published_windows_cuda_attempts(
|
|||
# actually provides on the host, preferring the torch line. Routing them
|
||||
# through the synthetic-minor path wrongly dropped cuda13 on a 13.0 driver.
|
||||
legacy_minors: list[str] = []
|
||||
# Scoped to the arch being resolved: hardcoded x64 would mis-order the arm64 runtime lines.
|
||||
_legacy_pattern = rf"-bin-win-cuda-(\d+\.\d+)-{re.escape(arch)}\.zip$"
|
||||
for artifact in published_artifacts:
|
||||
m = re.search(r"-bin-win-cuda-(\d+\.\d+)-x64\.zip$", artifact.asset_name)
|
||||
m = re.search(_legacy_pattern, artifact.asset_name)
|
||||
if m:
|
||||
legacy_minors.append(m.group(1))
|
||||
if legacy_minors:
|
||||
|
|
@ -3321,11 +3417,12 @@ def published_windows_cuda_attempts(
|
|||
host,
|
||||
release.upstream_tag,
|
||||
{
|
||||
f"llama-{release.upstream_tag}-bin-win-cuda-{minor}-x64.zip": "published"
|
||||
f"llama-{release.upstream_tag}-bin-win-cuda-{minor}-{arch}.zip": "published"
|
||||
for minor in legacy_minors
|
||||
},
|
||||
preferred_runtime_line,
|
||||
selection_log,
|
||||
arch = arch,
|
||||
)
|
||||
if attempt.runtime_line
|
||||
]
|
||||
|
|
@ -3363,7 +3460,10 @@ def published_windows_cuda_attempts(
|
|||
asset_url = release.assets.get(artifact.asset_name)
|
||||
if not asset_url:
|
||||
continue
|
||||
am = re.search(r"-bin-win-cuda-(\d+)\.(\d+)-x64\.zip$", artifact.asset_name)
|
||||
# Two groups (major, minor), for THIS arch: the x64 spelling never matched an arm64 bundle.
|
||||
am = re.search(
|
||||
rf"-bin-win-cuda-(\d+)\.(\d+)-{re.escape(arch)}\.zip$", artifact.asset_name
|
||||
)
|
||||
# Legacy upstream-named bundles encode the minor; gate it against the
|
||||
# driver. app-named bundles carry no minor and are driver-gated at the
|
||||
# runtime-line level by windows_cuda_attempts above.
|
||||
|
|
@ -3410,7 +3510,7 @@ def published_windows_cuda_attempts(
|
|||
runtime_archive_url: str | None = None
|
||||
if am is not None and artifact.asset_name.startswith("llama-"):
|
||||
runtime = f"{am.group(1)}.{am.group(2)}"
|
||||
cudart_name = f"cudart-llama-bin-win-cuda-{runtime}-x64.zip"
|
||||
cudart_name = f"cudart-llama-bin-win-cuda-{runtime}-{arch}.zip"
|
||||
cudart_url = release.assets.get(cudart_name)
|
||||
if cudart_url and cudart_url != asset_url:
|
||||
runtime_archive_name = cudart_name
|
||||
|
|
@ -3430,7 +3530,7 @@ def published_windows_cuda_attempts(
|
|||
name = artifact.asset_name,
|
||||
url = asset_url,
|
||||
source_label = "published",
|
||||
install_kind = "windows-cuda",
|
||||
install_kind = install_kind,
|
||||
runtime_line = runtime_line,
|
||||
runtime_name = runtime_archive_name,
|
||||
runtime_url = runtime_archive_url,
|
||||
|
|
@ -3446,7 +3546,10 @@ def published_windows_cuda_attempts(
|
|||
|
||||
|
||||
def resolve_windows_cuda_choices(
|
||||
host: HostInfo, llama_tag: str, upstream_assets: dict[str, str]
|
||||
host: HostInfo,
|
||||
llama_tag: str,
|
||||
upstream_assets: dict[str, str],
|
||||
arch: str | None = None,
|
||||
) -> list[AssetChoice]:
|
||||
torch_preference = detect_torch_cuda_runtime_preference(host)
|
||||
attempts = windows_cuda_attempts(
|
||||
|
|
@ -3455,6 +3558,7 @@ def resolve_windows_cuda_choices(
|
|||
upstream_assets,
|
||||
torch_preference.runtime_line,
|
||||
torch_preference.selection_log,
|
||||
arch = arch,
|
||||
)
|
||||
return attempts
|
||||
|
||||
|
|
@ -3811,6 +3915,28 @@ def resolve_upstream_asset_choice(host: HostInfo, llama_tag: str) -> AssetChoice
|
|||
install_kind = "windows-cpu",
|
||||
)
|
||||
|
||||
if host.is_windows and host.is_arm64:
|
||||
# CUDA first on an NVIDIA host, then the CPU bundle; without this branch every Windows ARM64 host fell through to a source build.
|
||||
if host.has_usable_nvidia and _upstream_arm64_cuda_allowed():
|
||||
attempts = _drop_blackwell_incapable_windows_cuda(
|
||||
host,
|
||||
resolve_windows_cuda_choices(host, llama_tag, upstream_assets, arch = "arm64"),
|
||||
)
|
||||
if attempts:
|
||||
return attempts[0]
|
||||
log("no compatible Windows ARM64 CUDA asset was found -- falling back to CPU")
|
||||
upstream_name = f"llama-{llama_tag}-bin-win-cpu-arm64.zip"
|
||||
if upstream_name not in upstream_assets:
|
||||
raise PrebuiltFallback("upstream Windows ARM64 CPU asset was not found")
|
||||
return AssetChoice(
|
||||
repo = UPSTREAM_REPO,
|
||||
tag = llama_tag,
|
||||
name = upstream_name,
|
||||
url = upstream_assets[upstream_name],
|
||||
source_label = "upstream",
|
||||
install_kind = "windows-arm64",
|
||||
)
|
||||
|
||||
if host.is_macos and host.is_arm64:
|
||||
upstream_name = f"llama-{llama_tag}-bin-macos-arm64.tar.gz"
|
||||
if upstream_name not in upstream_assets:
|
||||
|
|
@ -3918,7 +4044,72 @@ def resolve_release_asset_choice(
|
|||
else:
|
||||
published_choice = published_asset_choice_for_kind(release, "windows-cpu")
|
||||
elif host.is_windows and host.is_arm64:
|
||||
# Windows arm64 has no GPU prebuilt, so it always takes the CPU bundle.
|
||||
# Prefer a CUDA bundle, as x64 does: the published windows-arm64-cuda artifact first, then
|
||||
# upstream's -arm64 zip, both hash-gated. The opt-out gates the WHOLE branch.
|
||||
if host.has_usable_nvidia and _upstream_arm64_cuda_allowed():
|
||||
torch_preference = detect_torch_cuda_runtime_preference(host)
|
||||
published_arm64_cuda = _drop_blackwell_incapable_windows_cuda(
|
||||
host,
|
||||
published_windows_cuda_attempts(
|
||||
host,
|
||||
release,
|
||||
torch_preference.runtime_line,
|
||||
torch_preference.selection_log,
|
||||
arch = "arm64",
|
||||
),
|
||||
)
|
||||
if published_arm64_cuda:
|
||||
try:
|
||||
return apply_approved_hashes(published_arm64_cuda, checksums)
|
||||
except PrebuiltFallback as exc:
|
||||
log(
|
||||
"published Windows ARM64 CUDA assets ignored for install planning: "
|
||||
f"{release.repo}@{release.release_tag} ({exc})"
|
||||
)
|
||||
# Same rule as the digest fetch below: a rate limit, an outage or a malformed payload from the
|
||||
# release API costs the CUDA bundle, never the install; the ARM64 CPU bundle is still there.
|
||||
try:
|
||||
upstream_arm64_assets = github_release_assets(UPSTREAM_REPO, llama_tag)
|
||||
except Exception as exc:
|
||||
log(
|
||||
f"could not list the upstream {UPSTREAM_REPO}@{llama_tag} release assets "
|
||||
f"({exc}); falling through to the ARM64 CPU bundle."
|
||||
)
|
||||
upstream_arm64_assets = {}
|
||||
upstream_arm64_cuda = _drop_blackwell_incapable_windows_cuda(
|
||||
host,
|
||||
resolve_windows_cuda_choices(
|
||||
host,
|
||||
llama_tag,
|
||||
upstream_arm64_assets,
|
||||
arch = "arm64",
|
||||
),
|
||||
)
|
||||
if upstream_arm64_cuda:
|
||||
try:
|
||||
return apply_approved_hashes(upstream_arm64_cuda, checksums)
|
||||
except PrebuiltFallback as exc:
|
||||
# The fork publishes no windows-arm64-cuda bundle yet, so take upstream ggml-org's, the same release this fork
|
||||
# repackages. Verified against GitHub's per-asset digest; an asset it states no digest for is refused, not installed.
|
||||
verified = _apply_release_digests(
|
||||
upstream_arm64_cuda,
|
||||
github_release_asset_digests(UPSTREAM_REPO, llama_tag),
|
||||
)
|
||||
if verified:
|
||||
log(
|
||||
"no approved checksum covers a Windows ARM64 CUDA bundle "
|
||||
f"({exc}); installing the upstream {UPSTREAM_REPO} bundle "
|
||||
f"{verified[0].name}, verified against the release asset digest "
|
||||
"GitHub reports. Set UNSLOTH_LLAMA_ARM64_CUDA=0 to use the CPU "
|
||||
"bundle instead."
|
||||
)
|
||||
return verified
|
||||
log(
|
||||
"no approved checksum covers a Windows ARM64 CUDA bundle "
|
||||
f"({exc}), and GitHub reports no asset digest for the upstream "
|
||||
f"{UPSTREAM_REPO} bundle either; refusing to install it unverified "
|
||||
"and falling through to the ARM64 CPU bundle."
|
||||
)
|
||||
published_choice = published_asset_choice_for_kind(release, "windows-arm64")
|
||||
elif host.is_macos and host.is_arm64:
|
||||
published_choice = published_asset_choice_for_kind(release, "macos-arm64")
|
||||
|
|
@ -4283,8 +4474,10 @@ def paired_runtime_dll_patterns(choice: AssetChoice) -> list[str]:
|
|||
into the install. Used for the second copy_globs pass in
|
||||
install_from_archives, narrower than runtime_patterns_for_choice so
|
||||
the runtime archive cannot overwrite main-archive payload like
|
||||
llama-server.exe. Only Windows CUDA has paired runtimes today."""
|
||||
if choice.install_kind == "windows-cuda":
|
||||
llama-server.exe. Only Windows CUDA has paired runtimes today (x64 and arm64
|
||||
alike: upstream ships a cudart-...-arm64.zip beside the arm64 binary bundle,
|
||||
carrying the same DLL names built for ARM64)."""
|
||||
if choice.install_kind in _WINDOWS_CUDA_INSTALL_KINDS:
|
||||
return ["cudart64_*.dll", "cublas64_*.dll", "cublasLt64_*.dll"]
|
||||
return []
|
||||
|
||||
|
|
@ -4325,6 +4518,7 @@ def runtime_patterns_for_install_kind(
|
|||
if install_kind in {
|
||||
"windows-cpu",
|
||||
"windows-cuda",
|
||||
"windows-arm64-cuda",
|
||||
"windows-hip",
|
||||
"windows-vulkan",
|
||||
"windows-rocm",
|
||||
|
|
@ -6073,6 +6267,7 @@ def validate_server(
|
|||
"linux-rocm",
|
||||
"linux-vulkan",
|
||||
"windows-cuda",
|
||||
"windows-arm64-cuda",
|
||||
"windows-hip",
|
||||
"windows-vulkan",
|
||||
"windows-rocm",
|
||||
|
|
@ -6405,6 +6600,34 @@ def collect_system_report(host: HostInfo, choice: AssetChoice | None, install_di
|
|||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _apply_release_digests(
|
||||
attempts: Iterable[AssetChoice], digests: dict[str, str]
|
||||
) -> list[AssetChoice]:
|
||||
"""apply_approved_hashes against GitHub's per-asset digests instead of our manifest.
|
||||
|
||||
Same two rules, because they are the ones that matter: an attempt with no digest is
|
||||
dropped rather than installed unverified, and a paired runtime archive with no digest
|
||||
unpairs rather than riding along unchecked. Returns [] when nothing survives, which the
|
||||
one caller reads as "refuse and fall through", not as "install anyway".
|
||||
"""
|
||||
kept: list[AssetChoice] = []
|
||||
for attempt in attempts:
|
||||
digest = digests.get(attempt.name)
|
||||
if digest is None:
|
||||
continue
|
||||
attempt.expected_sha256 = digest
|
||||
if attempt.runtime_name and attempt.runtime_url:
|
||||
runtime_digest = digests.get(attempt.runtime_name)
|
||||
if runtime_digest is None:
|
||||
attempt.runtime_name = None
|
||||
attempt.runtime_url = None
|
||||
attempt.runtime_sha256 = None
|
||||
else:
|
||||
attempt.runtime_sha256 = runtime_digest
|
||||
kept.append(attempt)
|
||||
return kept
|
||||
|
||||
|
||||
def apply_approved_hashes(
|
||||
attempts: Iterable[AssetChoice], checksums: ApprovedReleaseChecksums
|
||||
) -> list[AssetChoice]:
|
||||
|
|
@ -7367,7 +7590,7 @@ def runtime_payload_health_groups(
|
|||
return groups
|
||||
if install_kind in {"windows-cpu", "windows-arm64"}:
|
||||
return _windows_shared_groups(source_label, tag)
|
||||
if install_kind == "windows-cuda":
|
||||
if install_kind in {"windows-cuda", "windows-arm64-cuda"}:
|
||||
groups = _windows_shared_groups(source_label, tag) + [["ggml-cuda.dll"]]
|
||||
# Require the complete cudart trio only when it was paired with this install.
|
||||
if runtime_name:
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ import re
|
|||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlsplit
|
||||
from typing import Dict, List, Optional, Sequence, Tuple
|
||||
|
||||
MANIFEST_NAME = "unsloth_install_manifest.json"
|
||||
|
|
@ -635,6 +636,9 @@ PROTECTED_MANIFEST_KEYS: Tuple[str, ...] = (
|
|||
"no_torch",
|
||||
"expected_torch_tag",
|
||||
"expected_torch_tag_pinned",
|
||||
# Validated on write (NVIDIA's own https channel, no userinfo/query/fragment) and read back
|
||||
# as an index to install from, so caller-composed evidence must not be able to name it.
|
||||
"woa_torch_index",
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -646,6 +650,7 @@ def write_manifest(
|
|||
no_torch: Optional[bool] = None,
|
||||
expected_torch_tag: Optional[str] = None,
|
||||
expected_torch_tag_pinned: Optional[bool] = None,
|
||||
woa_torch_index: Optional[str] = None,
|
||||
extra: Optional[Dict[str, object]] = None,
|
||||
) -> Optional[Path]:
|
||||
"""Record a completed install. Never raises: no manifest reads as incomplete,
|
||||
|
|
@ -679,6 +684,25 @@ def write_manifest(
|
|||
# eGPU with no repair offered. Absent means unknown, as with every other additive key.
|
||||
if expected_torch_tag_pinned is not None:
|
||||
payload["expected_torch_tag_pinned"] = bool(expected_torch_tag_pinned)
|
||||
# Windows on ARM has no CUDA wheels on download.pytorch.org, so a fresh shell cannot re-derive.
|
||||
# Only NVIDIA's own channels, with no userinfo, query or fragment: a mirror is not persisted.
|
||||
if woa_torch_index:
|
||||
candidate = str(woa_torch_index).strip()
|
||||
# urlsplit raises on a malformed authority, which would break the never-raises contract. Scheme and host compare case-insensitively (RFC 3986); the path keeps its case.
|
||||
try:
|
||||
parsed = urlsplit(candidate)
|
||||
_woa_ok = (
|
||||
parsed.scheme.lower() == "https"
|
||||
and parsed.hostname == "pypi.nvidia.com"
|
||||
and parsed.netloc.lower() == parsed.hostname
|
||||
and not parsed.query
|
||||
and not parsed.fragment
|
||||
)
|
||||
except ValueError:
|
||||
_woa_ok = False
|
||||
# netloc, not hostname: hostname strips ":443", so a value with a port is refused. Equality drops userinfo with it.
|
||||
if _woa_ok:
|
||||
payload["woa_torch_index"] = "https://pypi.nvidia.com" + parsed.path.rstrip("/")
|
||||
# The dependency pass's own evidence (`pass_inputs`, `step_results`, `pip_check_ok`, ...):
|
||||
# additive, never authoritative, every consumer re-verifies on disk. None is dropped so "absent
|
||||
# means unknown" holds, and nothing may shadow a field verify_install reads.
|
||||
|
|
|
|||
|
|
@ -83,15 +83,37 @@ _PLATFORM_HAS_TORCHCODEC_WHEEL = (
|
|||
PLATFORM_LACKS_TORCHCODEC_WHEEL = not _PLATFORM_HAS_TORCHCODEC_WHEEL
|
||||
|
||||
|
||||
def _machine_arch_from_registry() -> str:
|
||||
"""The machine-scope PROCESSOR_ARCHITECTURE, which an emulated process cannot
|
||||
misreport. Every per-process signal follows the process: under x64 emulation on
|
||||
ARM64 Windows the process copy says AMD64, PROCESSOR_ARCHITEW6432 is unset (it is a
|
||||
WOW64-only variable), and platform.machine() says AMD64 too. Empty when unreadable
|
||||
or off Windows."""
|
||||
if not IS_WINDOWS:
|
||||
return ""
|
||||
try:
|
||||
import winreg
|
||||
with winreg.OpenKey(
|
||||
winreg.HKEY_LOCAL_MACHINE,
|
||||
r"SYSTEM\CurrentControlSet\Control\Session Manager\Environment",
|
||||
) as key:
|
||||
return str(winreg.QueryValueEx(key, "PROCESSOR_ARCHITECTURE")[0] or "")
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def _is_windows_arm64() -> bool:
|
||||
"""Windows on ARM, machine arch rather than process arch: platform.machine() reports
|
||||
AMD64 under an emulated x64 Python, and PROCESSOR_ARCHITEW6432 is ARM64 in exactly
|
||||
that case. Mirrors Get-HostMachineArch in install.ps1 / setup.ps1."""
|
||||
"""Windows on ARM, machine arch rather than process arch. The registry value leads
|
||||
because the per-process signals all say AMD64 under an emulated x64 Python; they stay
|
||||
as fallbacks for a native interpreter. Mirrors Get-HostMachineArch in install.ps1 /
|
||||
setup.ps1. Wheel availability is an interpreter question, not a machine one: see
|
||||
``_is_win_arm64_interpreter``."""
|
||||
if not IS_WINDOWS:
|
||||
return False
|
||||
return any(
|
||||
(value or "").strip().lower() in {"arm64", "aarch64"}
|
||||
for value in (
|
||||
_machine_arch_from_registry(),
|
||||
os.environ.get("PROCESSOR_ARCHITEW6432"),
|
||||
os.environ.get("PROCESSOR_ARCHITECTURE"),
|
||||
platform.machine(),
|
||||
|
|
@ -99,6 +121,44 @@ def _is_windows_arm64() -> bool:
|
|||
)
|
||||
|
||||
|
||||
@functools.lru_cache(maxsize = None)
|
||||
def _is_win_arm64_interpreter() -> bool:
|
||||
"""Windows on ARM, the arch of THIS INTERPRETER rather than of the machine.
|
||||
|
||||
The distinction decides which wheels exist. ``_is_windows_arm64`` above answers
|
||||
for the machine, and is true even under an emulated x64 Python -- which is what
|
||||
every install predating native ARM64 support is running, because install.ps1
|
||||
deliberately fetched an x64 interpreter there. Such a venv wants the x64 wheels
|
||||
and gets them: ``platform_machine == "ARM64"`` in a requirement marker is the
|
||||
INTERPRETER's arch, so the marker rows and this predicate have to agree or the
|
||||
same machine is served two different answers.
|
||||
|
||||
``sysconfig.get_platform()`` is what pip and uv tag wheels with, so it is the
|
||||
same authority; ``platform.machine()`` is the fallback and reports AMD64 under
|
||||
emulation, which is the answer we want there.
|
||||
"""
|
||||
if not IS_WINDOWS:
|
||||
return False
|
||||
try:
|
||||
tag = (sysconfig.get_platform() or "").strip().lower()
|
||||
if tag:
|
||||
return tag == "win-arm64"
|
||||
except Exception:
|
||||
pass
|
||||
return (platform.machine() or "").strip().lower() in {"arm64", "aarch64"}
|
||||
|
||||
|
||||
def _windows_arm64_has_torchaudio() -> bool:
|
||||
"""Does the CUDA index this install used publish a win_arm64 torchaudio?
|
||||
|
||||
NVIDIA's GA out-of-tree channel does (2.11.0+cu134); its nightly channel and
|
||||
download.pytorch.org do not. install.ps1 answers this in UNSLOTH_WOA_HAS_TORCHAUDIO.
|
||||
Unset means "assume not": asking for a wheel that does not exist makes the whole trio
|
||||
unresolvable, while skipping one that does costs only audio support.
|
||||
"""
|
||||
return (os.environ.get("UNSLOTH_WOA_HAS_TORCHAUDIO") or "").strip() == "1"
|
||||
|
||||
|
||||
# ── ROCm / AMD GPU support ─────────────────────────────────────────────────────
|
||||
_ROCM_TORCH_INDEX: dict[tuple[int, int], str] = {
|
||||
(7, 2): "rocm7.2", # torch 2.11.0
|
||||
|
|
@ -3119,8 +3179,7 @@ def _hsa_spoofed_physical_gfx(
|
|||
re-probe."""
|
||||
if physical == [inferred_gfx]:
|
||||
_safe_print(
|
||||
f" {source} reports {inferred_gfx} -- {probed} is a spoof of the "
|
||||
f"physical arch.\n"
|
||||
f" {source} reports {inferred_gfx} -- {probed} is a spoof of the physical arch.\n"
|
||||
)
|
||||
return inferred_gfx
|
||||
# Say so rather than leaving "Checking whether..." hanging: on a real gfx1100
|
||||
|
|
@ -3937,8 +3996,7 @@ def _ensure_cuda_torch() -> None:
|
|||
if _target_span is None or not _span_covers(_target_span, _sms):
|
||||
return
|
||||
_why = (
|
||||
f"torch is {_family} but this host has GPUs outside its "
|
||||
f"sm_{_span[0]}-{_span[1]} range"
|
||||
f"torch is {_family} but this host has GPUs outside its sm_{_span[0]}-{_span[1]} range"
|
||||
)
|
||||
elif (
|
||||
_marker == "cpu"
|
||||
|
|
@ -4971,6 +5029,10 @@ def _ensure_expected_torch_flavor(expected: "str | None" = None) -> bool:
|
|||
"""
|
||||
if NO_TORCH:
|
||||
return True
|
||||
if _is_win_arm64_interpreter() and _explicit_torch_index_url() is None:
|
||||
_installed = _probe_installed_torch_version()
|
||||
if _installed and _is_cuda_family_leaf(_torch_flavor_tag(_installed)):
|
||||
return True
|
||||
# rocm/xpu/cpu fall THROUGH: an explicit GPU pin sets _TORCH_BACKEND, and rejecting it
|
||||
# here would skip the invariant on the hosts that asked for that family.
|
||||
if _TORCH_BACKEND not in ("", "cuda", "rocm", "xpu", "cpu"):
|
||||
|
|
@ -5045,9 +5107,9 @@ def _ensure_expected_torch_flavor(expected: "str | None" = None) -> bool:
|
|||
_torch_pkg, _vision_pkg, _audio_pkg = (
|
||||
_XPU_TORCH_PKG_SPEC if expected == "xpu" else _TORCH_FLAVOR_REPAIR_PKG_SPEC
|
||||
)
|
||||
# No win_arm64 torchaudio wheel exists on any index ($WinArm64NoAudio in setup.ps1).
|
||||
# Keyed on the INTERPRETER, not the machine: an emulated x64 venv installs win_amd64 wheels.
|
||||
_trio = [_torch_pkg, _vision_pkg, _audio_pkg]
|
||||
if _is_windows_arm64():
|
||||
if _is_win_arm64_interpreter() and not _windows_arm64_has_torchaudio():
|
||||
_trio = [_torch_pkg, _vision_pkg]
|
||||
_label_before = str(installed_version)
|
||||
# --force-reinstall, not install.ps1's uv-only --reinstall-package: pip_install falls
|
||||
|
|
@ -5392,10 +5454,8 @@ def _ensure_rocm_torch() -> None:
|
|||
_torch_pkg, _vision_pkg, _audio_pkg = _WINDOWS_ROCM_TORCH_PKG_SPECS.get(
|
||||
gfx_arch, ("torch", "torchvision", "torchaudio")
|
||||
)
|
||||
# Same win_arm64 exception setup.ps1 applies: no torchaudio wheel exists
|
||||
# there, so asking for one makes the trio unresolvable.
|
||||
_rocm_trio = [_torch_pkg, _vision_pkg, _audio_pkg]
|
||||
if _is_windows_arm64():
|
||||
if _is_win_arm64_interpreter():
|
||||
_rocm_trio = [_torch_pkg, _vision_pkg]
|
||||
# Nonfatal: a transient AMD-index failure must not abort the install.
|
||||
# --force-reinstall resolves before uninstalling, so a failed index keeps the
|
||||
|
|
@ -6392,6 +6452,35 @@ def run(
|
|||
return result
|
||||
|
||||
|
||||
# First line of the overrides file install.ps1 generates; setup.ps1's merge copies it first.
|
||||
WOA_OVERRIDES_HEADER = "# Generated by install.ps1 for Windows on ARM"
|
||||
|
||||
|
||||
def _woa_overrides_are_load_bearing() -> bool:
|
||||
"""Is this a native win_arm64 resolve whose correctness depends on UV_OVERRIDE?
|
||||
|
||||
install.ps1 writes those overrides to lift the released torch cap -- no win_arm64 CUDA
|
||||
wheel satisfies it -- and to drop the packages with no win_arm64 build. pip has no
|
||||
override mechanism at all: constraints only narrow a requirement, they cannot replace
|
||||
one, so there is nothing to translate them into. Falling back to pip on this stack does
|
||||
not recover, it silently resolves the wrong thing.
|
||||
|
||||
Judged by the generated file, not by the variable: a caller's own override on a run that
|
||||
never configured the stack (--no-torch, a direct run) keeps the fallback every host has.
|
||||
"""
|
||||
if not _is_win_arm64_interpreter():
|
||||
return False
|
||||
for path in os.environ.get("UV_OVERRIDE", "").split():
|
||||
try:
|
||||
with open(path, encoding = "utf-8", errors = "replace") as handle:
|
||||
first = handle.readline()
|
||||
except OSError:
|
||||
continue
|
||||
if first.lstrip("\ufeff").startswith(WOA_OVERRIDES_HEADER):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _report_failed_command(label: str, result: subprocess.CompletedProcess[bytes]) -> None:
|
||||
"""Print a failed command's redacted output and exit with its code."""
|
||||
_step("error", f"{label} failed (exit code {result.returncode})", _red)
|
||||
|
|
@ -6458,6 +6547,706 @@ def _purge_recordless_distributions(output: "bytes | str | None") -> list[str]:
|
|||
# Packages to skip on Windows (require special build steps)
|
||||
WINDOWS_SKIP_PACKAGES = {"triton_kernels"}
|
||||
|
||||
# No win_arm64 wheel and no sdist buildable without MSVC / Rust / LLVM / FFmpeg. All optional:
|
||||
# tensorboard needs grpcio; librosa and openai-whisper need numba -> llvmlite (cp314 only).
|
||||
# Lowercase entries only: _filter_requirements lowercases the line and compares verbatim.
|
||||
WINDOWS_ARM64_SKIP_PACKAGES = {
|
||||
"mecab",
|
||||
"sqlite-vec",
|
||||
"tiktoken",
|
||||
"tensorboard",
|
||||
"librosa",
|
||||
"openai-whisper",
|
||||
"torch-c-dlpack-ext",
|
||||
"pytorch_tokenizers",
|
||||
"hf_transfer",
|
||||
"xformers",
|
||||
}
|
||||
|
||||
|
||||
def _wheel_matches_interpreter(filename: str) -> bool:
|
||||
"""Can THIS interpreter install the wheel named ``filename``?
|
||||
|
||||
A wheelhouse is not built for one interpreter: install.ps1 stages every win_arm64
|
||||
wheel it finds, cp311 through cp314, as the wheelhouse published them. So a filename
|
||||
is not proof on its own -- a cp311 tiktoken is invisible to a cp313 resolver, and
|
||||
counting it as available drops the skip and sends the resolve to an sdist that cannot
|
||||
build here. PEP 425: a wheel is installable when one of its (python, abi, platform)
|
||||
triples is one the interpreter supports, and each filename field may be a
|
||||
"."-separated set expanded as their cartesian product. Unparseable means not
|
||||
installable, which only leaves the conservative skip in place.
|
||||
"""
|
||||
stem = filename[:-4] if filename.lower().endswith(".whl") else filename
|
||||
parts = stem.split("-")
|
||||
if len(parts) < 5:
|
||||
return False
|
||||
py_tags, abi_tags, plat_tags = (set(field.split(".")) for field in parts[-3:])
|
||||
this_platform = (sysconfig.get_platform() or "").replace("-", "_").replace(".", "_").lower()
|
||||
if "any" not in plat_tags and this_platform not in plat_tags:
|
||||
return False
|
||||
major, minor = sys.version_info[:2]
|
||||
free_threaded = bool(sysconfig.get_config_var("Py_GIL_DISABLED"))
|
||||
this_cpython = f"cp{major}{minor}"
|
||||
this_abi = f"{this_cpython}t" if free_threaded else this_cpython
|
||||
for py_tag in py_tags:
|
||||
for abi_tag in abi_tags:
|
||||
# abi3 excluded HERE too: this branch shadows the one below (CPython #111506).
|
||||
if py_tag == this_cpython and (
|
||||
abi_tag in ("none", this_abi) or (abi_tag == "abi3" and not free_threaded)
|
||||
):
|
||||
return True
|
||||
if abi_tag == "none":
|
||||
pure = re.fullmatch(r"py(\d)(\d*)", py_tag)
|
||||
if pure and int(pure.group(1)) == major and int(pure.group(2) or 0) <= minor:
|
||||
return True
|
||||
elif abi_tag == "abi3" and not free_threaded:
|
||||
stable = re.fullmatch(r"cp(\d)(\d+)", py_tag)
|
||||
if stable and int(stable.group(1)) == major and 2 <= int(stable.group(2)) <= minor:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
@functools.lru_cache(maxsize = 1)
|
||||
def _find_links_wheel_versions() -> "dict[str, frozenset[str]]":
|
||||
"""Canonical name -> versions of the wheels in the configured find-links directories.
|
||||
|
||||
install.ps1 points UV_FIND_LINKS / PIP_FIND_LINKS at a local wheelhouse on Windows on
|
||||
ARM, holding the wheels PyPI does not publish for win_arm64. Anything served there is
|
||||
installable, so it must not also be filtered out as unavailable -- but only the wheels
|
||||
tagged for this interpreter are, which is what the resolver will agree to.
|
||||
|
||||
The VERSIONS come back too, not just the names: every requirement these gate is
|
||||
``==``-pinned, and a wheelhouse holding tiktoken 0.12.0 against a ``tiktoken==0.13.0``
|
||||
line satisfies nothing -- the resolver goes to PyPI, finds no win_arm64 wheel for the
|
||||
pinned version, and falls to an sdist that cannot build here. That is the exact
|
||||
failure the skip list exists to prevent, so the caller checks the pin.
|
||||
"""
|
||||
versions: "dict[str, set[str]]" = {}
|
||||
# UV_FIND_LINKS ONLY: uv does not consume PIP_FIND_LINKS. Comma-split, the way uv reads it.
|
||||
for value, separator in ((os.environ.get("UV_FIND_LINKS"), ","),):
|
||||
for entry in re.split(separator, value or ""):
|
||||
entry = entry.strip().strip('"')
|
||||
if not entry or "://" in entry:
|
||||
continue # a URL index cannot be listed cheaply; treat it as unknown
|
||||
try:
|
||||
for wheel in Path(entry).glob("*.whl"):
|
||||
if not _wheel_matches_interpreter(wheel.name):
|
||||
continue
|
||||
fields = wheel.name[:-4].split("-")
|
||||
if len(fields) < 5:
|
||||
continue # not a wheel filename; _wheel_matches_interpreter agrees
|
||||
name = _canonical_dist_name(fields[0])
|
||||
versions.setdefault(name, set()).add(fields[1])
|
||||
except OSError:
|
||||
continue
|
||||
return {name: frozenset(vers) for name, vers in versions.items()}
|
||||
|
||||
|
||||
def _find_links_wheel_names() -> frozenset[str]:
|
||||
"""Just the names from :func:`_find_links_wheel_versions`."""
|
||||
return frozenset(_find_links_wheel_versions())
|
||||
|
||||
|
||||
_find_links_wheel_names.cache_clear = _find_links_wheel_versions.cache_clear # type: ignore[attr-defined]
|
||||
|
||||
|
||||
def _parse_release(version: str) -> "tuple[int, ...] | None":
|
||||
"""The numeric release segment of a PEP 440 version, or None when it has none."""
|
||||
match = re.match(r"^\s*v?(\d+(?:\.\d+)*)", version or "")
|
||||
if not match:
|
||||
return None
|
||||
return tuple(int(part) for part in match.group(1).split("."))
|
||||
|
||||
|
||||
def _version_satisfies(version: str, specifier: str) -> "bool | None":
|
||||
"""Does ``version`` satisfy the PEP 440 specifier set ``specifier``?
|
||||
|
||||
None means "cannot tell" -- an epoch, an arbitrary-equality clause, anything this
|
||||
deliberately small comparison does not model. The caller treats that as it treated
|
||||
every version before this existed, so an exotic pin is no worse off than it was.
|
||||
"""
|
||||
specifier = (specifier or "").strip()
|
||||
if not specifier:
|
||||
return True
|
||||
for module_name in ("packaging.specifiers", "pip._vendor.packaging.specifiers"):
|
||||
try:
|
||||
module = importlib.import_module(module_name)
|
||||
except Exception:
|
||||
continue
|
||||
try:
|
||||
spec_set = module.SpecifierSet(specifier)
|
||||
# Prereleases off unless the specifier names one; packaging's default admits 0.13.0rc1.
|
||||
return bool(spec_set.contains(version, prereleases = bool(spec_set.prereleases)))
|
||||
except Exception:
|
||||
break
|
||||
if not re.fullmatch(r"\s*v?\d+(?:\.\d+)*\s*", version or ""):
|
||||
return False
|
||||
got = _parse_release(version)
|
||||
# "!" that is not part of "!=" is a PEP 440 epoch, which _parse_release does not model.
|
||||
if got is None or "!" in (version or "") or "!" in specifier.replace("!=", ""):
|
||||
return None
|
||||
for clause in specifier.split(","):
|
||||
clause = clause.strip()
|
||||
if not clause:
|
||||
continue
|
||||
match = re.fullmatch(r"(==|!=|>=|<=|~=|>|<)\s*([^\s]+)", clause)
|
||||
if not match:
|
||||
return None
|
||||
op, want_raw = match.group(1), match.group(2).rstrip(".*")
|
||||
wildcard = match.group(2).endswith(".*")
|
||||
want = _parse_release(want_raw)
|
||||
if want is None:
|
||||
return None
|
||||
# Pad to a common length so 2.11 and 2.11.0 compare equal, as PEP 440 says.
|
||||
width = max(len(got), len(want))
|
||||
lhs = got + (0,) * (width - len(got))
|
||||
rhs = want + (0,) * (width - len(want))
|
||||
if wildcard:
|
||||
# ==1.2.* / !=1.2.*: only the prefix is compared.
|
||||
prefix = got[: len(want)] + (0,) * max(0, len(want) - len(got))
|
||||
ok = prefix == want
|
||||
if op == "==":
|
||||
pass
|
||||
elif op == "!=":
|
||||
ok = not ok
|
||||
else:
|
||||
return None
|
||||
elif op == "==":
|
||||
ok = lhs == rhs
|
||||
elif op == "!=":
|
||||
ok = lhs != rhs
|
||||
elif op == ">=":
|
||||
ok = lhs >= rhs
|
||||
elif op == "<=":
|
||||
ok = lhs <= rhs
|
||||
elif op == ">":
|
||||
ok = lhs > rhs
|
||||
elif op == "<":
|
||||
ok = lhs < rhs
|
||||
else: # ~=X.Y[.Z] is ">=X.Y[.Z], ==X.Y.*" one level up
|
||||
if len(want) < 2:
|
||||
return None
|
||||
ok = lhs >= rhs and got[: len(want) - 1] == want[: len(want) - 1]
|
||||
if not ok:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _marker_is_active(marker: str) -> "bool | None":
|
||||
"""Does ``marker`` hold for THIS interpreter? None when it cannot be decided.
|
||||
|
||||
packaging is what pip and uv evaluate markers with, so borrowing it keeps the answer
|
||||
identical to the resolver's; pip vendors a copy, which is the fallback for a venv
|
||||
where packaging is not installed in its own right. Neither available means the
|
||||
caller keeps every clause instead of picking one.
|
||||
"""
|
||||
if not marker:
|
||||
return True
|
||||
for module_name in ("packaging.markers", "pip._vendor.packaging.markers"):
|
||||
try:
|
||||
module = importlib.import_module(module_name)
|
||||
except Exception:
|
||||
continue
|
||||
try:
|
||||
return bool(module.Marker(marker).evaluate())
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _requirement_pins(req: "Path | None") -> "dict[str, list[str]]":
|
||||
"""Canonical name -> the version specifiers a requirements file states for it.
|
||||
|
||||
A name can appear more than once, split by marker -- extras.txt carries
|
||||
``MeCab==0.996.13`` and ``MeCab==0.996.5`` on complementary markers -- so keeping one
|
||||
specifier per name let the row for another platform overwrite the row that actually
|
||||
applies. Markers are evaluated for this interpreter and inactive rows dropped;
|
||||
when they cannot be evaluated every clause is kept, and the caller takes any of them
|
||||
as satisfied, which is no stricter than the name-only check that came before.
|
||||
"""
|
||||
pins: "dict[str, list[str]]" = {}
|
||||
if req is None:
|
||||
return pins
|
||||
try:
|
||||
text = req.read_text(encoding = "utf-8-sig")
|
||||
except OSError:
|
||||
return pins
|
||||
for line in text.splitlines():
|
||||
line = line.split("#", 1)[0].strip()
|
||||
if not line or line.startswith("-") or "@" in line:
|
||||
continue
|
||||
line, _, marker = line.partition(";")
|
||||
if _marker_is_active(marker.strip()) is False:
|
||||
continue
|
||||
line = line.strip()
|
||||
match = re.match(r"^([A-Za-z0-9][A-Za-z0-9._-]*)\s*(?:\[[^\]]*\])?\s*(.*)$", line)
|
||||
if not match:
|
||||
continue
|
||||
pins.setdefault(_canonical_dist_name(match.group(1)), []).append(match.group(2).strip())
|
||||
return pins
|
||||
|
||||
|
||||
# Skipped for their DEPENDENCIES: whisper's metadata needs tiktoken unconditionally, so filtering the direct line does not stop the sdist arriving transitively.
|
||||
WINDOWS_ARM64_SKIP_UNBLOCKED_BY = {
|
||||
"tensorboard": ("grpcio",),
|
||||
# soxr as well as the numba pair: librosa 0.11.0 requires soxr>=0.3.2 and soxr has never published a win_arm64 wheel, so un-skipping librosa would build it from an sdist.
|
||||
"librosa": ("llvmlite", "numba", "soxr"),
|
||||
"openai_whisper": ("llvmlite", "numba", "tiktoken"),
|
||||
}
|
||||
|
||||
|
||||
# The blocker versions the packages' OWN metadata demands: too old, and the extras pass fails.
|
||||
# {blocker: (specifier, package it was read from, that package's pinned version)}. llvmlite arrives through numba.
|
||||
WINDOWS_ARM64_BLOCKER_FLOORS: "dict[str, tuple[str, str, str]]" = {
|
||||
"grpcio": (">=1.74.0", "tensorboard", "2.21.0"),
|
||||
"numba": (">=0.51.0", "librosa", "0.11.0"),
|
||||
"soxr": (">=0.3.2", "librosa", "0.11.0"),
|
||||
}
|
||||
|
||||
|
||||
# Excluded on win_arm64 by MARKER, so a hosted wheel has no requirement to satisfy. Floors given, because --no-deps otherwise takes whatever happens to be hosted.
|
||||
WINDOWS_ARM64_WHEELHOUSE_OPTIONALS = {
|
||||
"hf-transfer": "",
|
||||
"xformers": ">=0.0.22.post7",
|
||||
"sqlite-vec": "",
|
||||
}
|
||||
|
||||
|
||||
def _wheelhouse_hosts(name: str) -> bool:
|
||||
"""Does the resolver's own find-links carry a wheel for this distribution?"""
|
||||
return bool(_find_links_wheel_versions().get(_canonical_dist_name(name)))
|
||||
|
||||
|
||||
def _wheelhouse_best_version(name: str, floor: str) -> "str | None":
|
||||
"""The newest hosted version that clears ``floor``, or None if none does.
|
||||
|
||||
An unreadable comparison keeps the answer it would have had before this floor
|
||||
existed, matching _windows_arm64_skip_packages: an exotic version is no worse off.
|
||||
"""
|
||||
hosted = _find_links_wheel_versions().get(_canonical_dist_name(name)) or ()
|
||||
usable = [
|
||||
version
|
||||
for version in hosted
|
||||
if not floor or _version_satisfies(version, floor) is not False
|
||||
]
|
||||
if not usable:
|
||||
return None
|
||||
try:
|
||||
from packaging.version import Version
|
||||
return max(usable, key = Version)
|
||||
except Exception:
|
||||
return sorted(usable)[-1]
|
||||
|
||||
|
||||
def _wheelhouse_torchcodec_version(torch_version: "str | None") -> "str | None":
|
||||
"""The hosted torchcodec inside the window this torch selects, or None.
|
||||
|
||||
win_arm64 only: elsewhere the platform's own wheel decides. An unknown torch has no
|
||||
window, so it gets no answer either.
|
||||
"""
|
||||
if not _is_win_arm64_interpreter() or not torch_version:
|
||||
return None
|
||||
spec = _select_torchcodec_spec(torch_version)
|
||||
if spec is None:
|
||||
return None
|
||||
return _wheelhouse_best_version("torchcodec", spec.split("torchcodec", 1)[1])
|
||||
|
||||
|
||||
def _install_wheelhouse_optionals() -> None:
|
||||
"""Install the hosted optionals the metadata cannot ask for. Best effort.
|
||||
|
||||
--no-deps: the graph is already resolved and installed by the time this runs, and
|
||||
xformers names torch, so resolving here could walk the win_arm64 CUDA build off to
|
||||
whatever PyPI offers. A failure leaves the feature off, which is where it was.
|
||||
|
||||
Pinned to the selected version rather than installed by bare name, so the floor
|
||||
checked here is the version that actually lands.
|
||||
"""
|
||||
if not _is_win_arm64_interpreter():
|
||||
return
|
||||
for name, floor in WINDOWS_ARM64_WHEELHOUSE_OPTIONALS.items():
|
||||
version = _wheelhouse_best_version(name, floor)
|
||||
if version is None:
|
||||
if _wheelhouse_hosts(name):
|
||||
_note(f"windows on arm: the wheelhouse {name} is below {floor}; leaving it off")
|
||||
# Nothing to refresh with, but the copy an earlier run installed is still resident.
|
||||
if _canonical_dist_name(name) == "xformers":
|
||||
_evict_xformers_built_for_another_torch()
|
||||
continue
|
||||
installed = pip_install_try(
|
||||
f"Installing {name}=={version} from the Windows on ARM wheelhouse",
|
||||
"--no-deps",
|
||||
"--no-cache-dir",
|
||||
f"{name}=={version}",
|
||||
constrain = False,
|
||||
)
|
||||
if not installed:
|
||||
_note(f"windows on arm: could not install the wheelhouse {name}; feature stays off")
|
||||
# Checked even when the refresh failed: the copy an earlier torch left behind is still resident.
|
||||
if _canonical_dist_name(name) == "xformers" and _evict_xformers_built_for_another_torch():
|
||||
continue
|
||||
if not installed:
|
||||
continue
|
||||
_note(f"windows on arm: installed {name}=={version} from the wheelhouse")
|
||||
|
||||
|
||||
def _evict_xformers_built_for_another_torch() -> bool:
|
||||
"""Remove a resident xFormers whose extension was built against another torch. True iff removed.
|
||||
|
||||
xFormers links its extension against ONE (torch, CUDA) pair; beside any other it is mute,
|
||||
and a package install never uninstalls what an earlier run left behind.
|
||||
"""
|
||||
built_for = _resident_xformers_build_torch()
|
||||
resident = str(_probe_installed_torch_version() or "")
|
||||
if not (built_for and resident and built_for != resident):
|
||||
return False
|
||||
_uninstall_distribution("xformers")
|
||||
_note(
|
||||
f"windows on arm: the wheelhouse xformers was built for torch "
|
||||
f"{built_for}, not {resident} -- removed; attention uses torch SDPA"
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
WINDOWS_ARM64_PUBLIC_INDEX_WHEELS: "dict[str, dict[str, str]]" = {
|
||||
"llvmlite": {"cp314": "0.49.0"},
|
||||
"numba": {"cp314": "0.67.0"},
|
||||
}
|
||||
|
||||
|
||||
def _uv_config_files() -> "list[tuple[Path, str]]":
|
||||
"""The persistent configuration uv would discover, most specific first, as (path, table).
|
||||
|
||||
uv reads uv.toml or pyproject.toml [tool.uv] from the current directory or the nearest
|
||||
parent (uv.toml wins over pyproject.toml in the same directory), then the user file
|
||||
(%APPDATA%\\uv\\uv.toml on Windows, $XDG_CONFIG_HOME/uv/uv.toml elsewhere), then the
|
||||
system file (%PROGRAMDATA%\\uv\\uv.toml, /etc/uv/uv.toml). UV_CONFIG_FILE names one file
|
||||
instead of discovering; UV_NO_CONFIG discovers nothing. `table` is the prefix the index
|
||||
keys sit under: "" for uv.toml, "tool.uv" for pyproject.toml.
|
||||
"""
|
||||
if os.environ.get("UV_NO_CONFIG", "").strip().lower() not in ("", "0", "false"):
|
||||
return []
|
||||
explicit = os.environ.get("UV_CONFIG_FILE", "").strip()
|
||||
if explicit:
|
||||
return [(Path(explicit), "")]
|
||||
found: "list[tuple[Path, str]]" = []
|
||||
here = Path.cwd()
|
||||
for d in (here, *here.parents):
|
||||
uv_toml = d / "uv.toml"
|
||||
if uv_toml.is_file():
|
||||
found.append((uv_toml, ""))
|
||||
break
|
||||
pyproject = d / "pyproject.toml"
|
||||
if pyproject.is_file():
|
||||
try:
|
||||
text = pyproject.read_text(encoding = "utf-8")
|
||||
except OSError:
|
||||
text = ""
|
||||
if re.search(r"(?m)^\s*\[+tool\.uv(\.|\])", text):
|
||||
found.append((pyproject, "tool.uv"))
|
||||
break
|
||||
if IS_WINDOWS:
|
||||
user = os.environ.get("APPDATA", "")
|
||||
system = os.environ.get("PROGRAMDATA", "")
|
||||
if user:
|
||||
found.append((Path(user) / "uv" / "uv.toml", ""))
|
||||
if system:
|
||||
found.append((Path(system) / "uv" / "uv.toml", ""))
|
||||
else:
|
||||
xdg = os.environ.get("XDG_CONFIG_HOME", "") or str(Path.home() / ".config")
|
||||
found.append((Path(xdg) / "uv" / "uv.toml", ""))
|
||||
found.append((Path("/etc/uv/uv.toml"), ""))
|
||||
return [(p, table) for p, table in found if p.is_file()]
|
||||
|
||||
|
||||
def _uv_config_index_policy() -> "dict[str, object]":
|
||||
"""{no_index, default_index, unreadable} from uv's discovered configuration.
|
||||
|
||||
Project outranks user outranks system for a scalar, so the first file that sets a key
|
||||
decides it. Only the keys that decide where a resolve looks are read: no-index and
|
||||
default-index (index-url is the older spelling), at the top level and under [pip], and
|
||||
an [[index]] entry carrying default = true. A file this cannot parse is reported rather
|
||||
than guessed at.
|
||||
"""
|
||||
policy: "dict[str, object]" = {
|
||||
"no_index": None,
|
||||
"default_index": None,
|
||||
"unreadable": False,
|
||||
"extra_indexes": [],
|
||||
}
|
||||
try:
|
||||
import tomllib
|
||||
except ImportError: # 3.10: the native path is 3.11+, so only the x64 fallback lands here
|
||||
policy["unreadable"] = bool(_uv_config_files())
|
||||
return policy
|
||||
for path, table in _uv_config_files():
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
data = tomllib.load(fh)
|
||||
except (OSError, ValueError):
|
||||
policy["unreadable"] = True
|
||||
continue
|
||||
section = data
|
||||
for part in [p for p in table.split(".") if p]:
|
||||
section = section.get(part, {}) if isinstance(section, dict) else {}
|
||||
if not isinstance(section, dict):
|
||||
continue
|
||||
# uv pip (0.10.7): [pip] scalars beat top-level, and [[index]] default = true beats both.
|
||||
pip_scope = section.get("pip", {}) if isinstance(section.get("pip"), dict) else {}
|
||||
file_no_index = None
|
||||
for scope in (pip_scope, section):
|
||||
if file_no_index is None and isinstance(scope.get("no-index"), bool):
|
||||
file_no_index = scope["no-index"]
|
||||
file_default = None
|
||||
extras: list[str] = []
|
||||
indexes = section.get("index")
|
||||
if isinstance(indexes, list):
|
||||
for entry in indexes:
|
||||
if not isinstance(entry, dict) or not isinstance(entry.get("url"), str):
|
||||
continue
|
||||
if entry.get("explicit") is True:
|
||||
# uv: explicit serves only packages pinned via [tool.uv.sources]; with default = true it also removes PyPI as the default (not modelled: doubt).
|
||||
if entry.get("default") is True:
|
||||
policy["unreadable"] = True
|
||||
continue
|
||||
if entry.get("default") is True:
|
||||
if file_default is None:
|
||||
file_default = entry["url"]
|
||||
else:
|
||||
extras.append(entry["url"])
|
||||
for scope in (pip_scope, section):
|
||||
value = scope.get("extra-index-url")
|
||||
if isinstance(value, str):
|
||||
extras.append(value)
|
||||
elif isinstance(value, list):
|
||||
extras.extend(v for v in value if isinstance(v, str))
|
||||
policy["extra_indexes"] = list(policy["extra_indexes"]) + extras
|
||||
if file_default is None:
|
||||
for scope in (pip_scope, section):
|
||||
for key in ("default-index", "index-url"):
|
||||
if file_default is None and isinstance(scope.get(key), str):
|
||||
file_default = scope[key]
|
||||
if policy["no_index"] is None and file_no_index is not None:
|
||||
policy["no_index"] = file_no_index
|
||||
if policy["default_index"] is None and file_default is not None:
|
||||
policy["default_index"] = file_default
|
||||
return policy
|
||||
|
||||
|
||||
def _url_is_public_pypi(url: str) -> bool:
|
||||
"""The host, not a substring: "https://pypi.org.corp.example/simple" and
|
||||
".../api/pypi/pypi.org/simple" both contain the name and neither is public PyPI."""
|
||||
try:
|
||||
from urllib.parse import urlsplit
|
||||
host = urlsplit(url.strip()).hostname
|
||||
except ValueError:
|
||||
return False
|
||||
return host is not None and host.lower() == "pypi.org"
|
||||
|
||||
|
||||
def _public_pypi_is_reachable() -> bool:
|
||||
"""Can this resolution actually reach public PyPI?
|
||||
|
||||
The table below records what PyPI publishes, which is only availability if PyPI is where
|
||||
the resolve will look. Offline, or pointed at an exclusive corporate index, those wheels
|
||||
are neither cached nor served: unblocking librosa there drops the skip and then fails the
|
||||
whole extras pass on an unavailable numba, which is exactly what the skip prevents.
|
||||
|
||||
Judged for the resolver that runs the pass. uv reads UV_* and its configuration files and
|
||||
ignores PIP_*; pip reads PIP_* and ignores UV_*. Mixing the two reported PyPI reachable
|
||||
from a PIP_EXTRA_INDEX_URL that uv, the resolver in use, never consults. A default index
|
||||
REPLACES PyPI; an extra index adds to it, so PyPI named there is still consulted.
|
||||
Environment outranks uv's configuration files. Doubt resolves to False: that answer keeps
|
||||
the skip, the other fails the extras pass.
|
||||
"""
|
||||
if USE_UV:
|
||||
return _uv_reaches_public_pypi()
|
||||
return _pip_reaches_public_pypi()
|
||||
|
||||
|
||||
def _env_flag(name: str) -> bool:
|
||||
return os.environ.get(name, "").strip().lower() not in ("", "0", "false")
|
||||
|
||||
|
||||
def _uv_reaches_public_pypi() -> bool:
|
||||
if _uv_is_offline() or _env_flag("UV_NO_INDEX"):
|
||||
return False
|
||||
extra_is_pypi = any(
|
||||
_url_is_public_pypi(u)
|
||||
for var in ("UV_INDEX", "UV_EXTRA_INDEX_URL")
|
||||
for u in os.environ.get(var, "").split()
|
||||
)
|
||||
for var in ("UV_DEFAULT_INDEX", "UV_INDEX_URL"):
|
||||
value = os.environ.get(var, "").strip()
|
||||
if value:
|
||||
return extra_is_pypi or _url_is_public_pypi(value)
|
||||
policy = _uv_config_index_policy()
|
||||
if policy["unreadable"] or policy["no_index"] is True:
|
||||
return False
|
||||
extra_is_pypi = extra_is_pypi or any(
|
||||
_url_is_public_pypi(u) for u in policy["extra_indexes"] if isinstance(u, str)
|
||||
)
|
||||
default = policy["default_index"]
|
||||
if isinstance(default, str) and not _url_is_public_pypi(default):
|
||||
return extra_is_pypi
|
||||
return True
|
||||
|
||||
|
||||
def _pip_reaches_public_pypi() -> bool:
|
||||
"""pip's policy: its environment first, then the configuration files it would read.
|
||||
|
||||
PIP_* outranks every file. Below that, `pip config list` reports the effective values from
|
||||
the site, user and global files, an `[install]` key outranking its `[global]` twin for an
|
||||
install. A `no-index` or an exclusive `index-url` set there replaces PyPI just as the
|
||||
environment does. Doubt (a `pip config` that cannot be read) keeps the skip.
|
||||
"""
|
||||
if _env_flag("PIP_NO_INDEX"):
|
||||
return False
|
||||
extra_is_pypi = any(
|
||||
_url_is_public_pypi(u) for u in os.environ.get("PIP_EXTRA_INDEX_URL", "").split()
|
||||
)
|
||||
value = os.environ.get("PIP_INDEX_URL", "").strip()
|
||||
if value:
|
||||
return extra_is_pypi or _url_is_public_pypi(value)
|
||||
policy = _pip_config_index_policy()
|
||||
if policy["unreadable"] or policy["no_index"] is True:
|
||||
return False
|
||||
extra_is_pypi = extra_is_pypi or any(_url_is_public_pypi(u) for u in policy["extra_index_urls"])
|
||||
index_url = policy["index_url"]
|
||||
if isinstance(index_url, str) and not _url_is_public_pypi(index_url):
|
||||
return extra_is_pypi
|
||||
return True
|
||||
|
||||
|
||||
def _pip_config_index_policy() -> "dict[str, object]":
|
||||
"""The index keys pip's configuration files set, read from `pip config list`.
|
||||
|
||||
Lines are `<section>.<key>='<value>'`; `:env:` entries mirror PIP_* variables the caller
|
||||
already read, so they are skipped. `install.<key>` outranks `global.<key>`, as it does for
|
||||
pip itself. A `pip config` that cannot run or be parsed is reported unreadable.
|
||||
"""
|
||||
policy: "dict[str, object]" = {
|
||||
"no_index": None,
|
||||
"index_url": None,
|
||||
"extra_index_urls": [],
|
||||
"unreadable": False,
|
||||
}
|
||||
try:
|
||||
done = subprocess.run(
|
||||
[sys.executable, "-m", "pip", "config", "list"],
|
||||
capture_output = True,
|
||||
text = True,
|
||||
timeout = 60,
|
||||
**_windows_hidden_subprocess_kwargs(),
|
||||
)
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
policy["unreadable"] = True
|
||||
return policy
|
||||
if done.returncode != 0:
|
||||
policy["unreadable"] = True
|
||||
return policy
|
||||
found: "dict[str, dict[str, str]]" = {"global": {}, "install": {}}
|
||||
for line in done.stdout.splitlines():
|
||||
m = re.match(r"^(global|install)\.([a-z-]+)=(.*)$", line.strip())
|
||||
if not m:
|
||||
continue
|
||||
section, key, raw = m.groups()
|
||||
raw = raw.strip()
|
||||
if len(raw) >= 2 and raw[0] == raw[-1] and raw[0] in "'\"":
|
||||
raw = raw[1:-1]
|
||||
found[section][key] = raw
|
||||
for section in ("global", "install"):
|
||||
keys = found[section]
|
||||
if "no-index" in keys:
|
||||
policy["no_index"] = keys["no-index"].strip().lower() in ("1", "true", "yes", "on")
|
||||
if keys.get("index-url", "").strip():
|
||||
policy["index_url"] = keys["index-url"].strip()
|
||||
if "extra-index-url" in keys:
|
||||
# pip prints the repr, so a multi-line value arrives with a literal backslash-n.
|
||||
policy["extra_index_urls"] = [
|
||||
u for u in re.split(r"\s+|\\n", keys["extra-index-url"]) if u
|
||||
]
|
||||
return policy
|
||||
|
||||
|
||||
def _woa_pypi_provided_versions() -> "dict[str, set[str]]":
|
||||
"""{canonical name: versions} install.ps1 found PyPI publishing for THIS interpreter.
|
||||
|
||||
UNSLOTH_WOA_PYPI_PROVIDED carries space-separated `name==version` entries: the wheelhouse
|
||||
wheels install.ps1 discarded because PyPI serves the same version. The managed copy is
|
||||
gone, so find-links no longer answers for them; this is the only record that they resolve.
|
||||
"""
|
||||
provided: "dict[str, set[str]]" = {}
|
||||
for entry in os.environ.get("UNSLOTH_WOA_PYPI_PROVIDED", "").split():
|
||||
name, sep, version = entry.partition("==")
|
||||
if sep and name and version:
|
||||
provided.setdefault(_canonical_dist_name(name), set()).add(version)
|
||||
return provided
|
||||
|
||||
|
||||
def _public_index_win_arm64_versions(canonical: str) -> "set[str]":
|
||||
"""Versions the public index publishes a usable win_arm64 wheel of, for THIS build.
|
||||
|
||||
Empty off win_arm64, empty when the resolve cannot reach PyPI, and empty for an
|
||||
interpreter the recorded wheel is not tagged for. The tag is judged by
|
||||
_wheel_matches_interpreter rather than by comparing strings, so a free-threaded build does
|
||||
not claim a wheel built for the GIL one. install.ps1's own probe is added on top.
|
||||
"""
|
||||
if not _is_win_arm64_interpreter() or not _public_pypi_is_reachable():
|
||||
return set()
|
||||
return {
|
||||
version
|
||||
for tag, version in WINDOWS_ARM64_PUBLIC_INDEX_WHEELS.get(canonical, {}).items()
|
||||
if _wheel_matches_interpreter(f"{canonical}-{version}-{tag}-{tag}-win_arm64.whl")
|
||||
} | _woa_pypi_provided_versions().get(_canonical_dist_name(canonical), set())
|
||||
|
||||
|
||||
def _windows_arm64_skip_packages(req: "Path | None" = None) -> set[str]:
|
||||
"""WINDOWS_ARM64_SKIP_PACKAGES minus whatever the wheelhouse already provides, so
|
||||
hosting a wheel is all it takes to re-enable one of these features here.
|
||||
|
||||
``req`` is the requirements file about to be installed, when there is one. Its pins
|
||||
decide whether a hosted wheel is actually usable: a name match is not enough, because
|
||||
the resolver has to honour ``tiktoken==0.13.0`` and a staged 0.12.0 wheel leaves it
|
||||
with the unbuildable sdist rather than the skip this list is here to keep.
|
||||
"""
|
||||
available = _find_links_wheel_versions()
|
||||
if not available and not any(
|
||||
_public_index_win_arm64_versions(name)
|
||||
for name in set(WINDOWS_ARM64_PUBLIC_INDEX_WHEELS) | set(_woa_pypi_provided_versions())
|
||||
):
|
||||
return set(WINDOWS_ARM64_SKIP_PACKAGES)
|
||||
pins = _requirement_pins(req)
|
||||
|
||||
def hosted(name: str) -> bool:
|
||||
canonical = _canonical_dist_name(name)
|
||||
versions = set(available.get(canonical) or ()) | _public_index_win_arm64_versions(canonical)
|
||||
if not versions:
|
||||
return False
|
||||
clauses = [clause for clause in pins.get(canonical, []) if clause]
|
||||
if not clauses:
|
||||
floor = WINDOWS_ARM64_BLOCKER_FLOORS.get(canonical)
|
||||
if floor is None:
|
||||
return True
|
||||
clauses = [floor[0]]
|
||||
verdicts = [_version_satisfies(v, c) for v in versions for c in clauses]
|
||||
if any(verdict is True for verdict in verdicts):
|
||||
return True
|
||||
return any(verdict is None for verdict in verdicts)
|
||||
|
||||
keep_skipping: set[str] = set()
|
||||
for package in WINDOWS_ARM64_SKIP_PACKAGES:
|
||||
canonical = _canonical_dist_name(package)
|
||||
blockers = WINDOWS_ARM64_SKIP_UNBLOCKED_BY.get(canonical)
|
||||
if blockers:
|
||||
if all(hosted(b) for b in blockers):
|
||||
continue
|
||||
elif hosted(package):
|
||||
continue
|
||||
keep_skipping.add(package)
|
||||
return keep_skipping
|
||||
|
||||
|
||||
# Skipped without torch (Intel Mac GGUF-only), plus librosa, whose numba chain fails (#5046).
|
||||
NO_TORCH_SKIP_PACKAGES = {
|
||||
"torch-stoi",
|
||||
|
|
@ -8285,10 +9074,39 @@ def pip_install(
|
|||
if VERBOSE and result.stdout:
|
||||
_safe_print(_redact_install_output(result.stdout))
|
||||
return
|
||||
if _woa_overrides_are_load_bearing():
|
||||
_step("error", f"{label} failed and pip cannot stand in for it", _red)
|
||||
_safe_print(
|
||||
_red(
|
||||
" The Windows on ARM stack resolves through UV_OVERRIDE, which pip has no "
|
||||
"equivalent for: overrides REPLACE a requirement, and pip constraints can "
|
||||
"only narrow one."
|
||||
)
|
||||
)
|
||||
_safe_print(
|
||||
_red(
|
||||
" Falling back here would honour the released torch cap, which no "
|
||||
"win_arm64 CUDA wheel satisfies, and pull back the packages that have no "
|
||||
"win_arm64 build at all -- downgrading a working CUDA torch or failing "
|
||||
"later, with nothing to say why."
|
||||
)
|
||||
)
|
||||
_safe_print(_red(" Install uv and re-run, or re-run install.ps1."))
|
||||
_report_failed_command(label, result)
|
||||
_safe_print(_red(f" uv failed, falling back to pip..."))
|
||||
if result.stdout:
|
||||
_safe_print(_redact_install_output(result.stdout))
|
||||
|
||||
elif _woa_overrides_are_load_bearing():
|
||||
_step("error", f"{label} needs uv on the Windows on ARM stack", _red)
|
||||
_safe_print(
|
||||
_red(
|
||||
" The native ARM64 resolve depends on UV_OVERRIDE, which pip cannot express. "
|
||||
"Install uv and re-run, or re-run install.ps1."
|
||||
)
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
pip_cmd, pip_env = _pinned_cmd_and_env(
|
||||
_build_pip_cmd(args) + constraint_args_pip + req_args_pip
|
||||
)
|
||||
|
|
@ -8880,11 +9698,19 @@ def _effective_requirements(req: Path) -> "tuple[Path, list[Path]]":
|
|||
if IS_WINDOWS and WINDOWS_SKIP_PACKAGES:
|
||||
actual = _filter_requirements(actual, WINDOWS_SKIP_PACKAGES)
|
||||
temps.append(actual)
|
||||
if _is_win_arm64_interpreter():
|
||||
# Judged against the ORIGINAL file: the pins decide whether a hosted wheel is usable,
|
||||
# and a copy already filtered above no longer carries the rows they sit on.
|
||||
arm64_skip = _windows_arm64_skip_packages(req)
|
||||
if arm64_skip:
|
||||
actual = _filter_requirements(actual, arm64_skip)
|
||||
temps.append(actual)
|
||||
if NO_TORCH and NO_TORCH_SKIP_PACKAGES:
|
||||
actual = _filter_requirements(actual, NO_TORCH_SKIP_PACKAGES)
|
||||
temps.append(actual)
|
||||
if PLATFORM_LACKS_TORCHCODEC_WHEEL:
|
||||
# Linux aarch64 / Windows ARM64 / Intel Mac have no torchcodec wheel.
|
||||
if PLATFORM_LACKS_TORCHCODEC_WHEEL and not _wheelhouse_hosts("torchcodec"):
|
||||
# Linux aarch64 / Windows ARM64 / Intel Mac have no torchcodec wheel, unless the
|
||||
# win_arm64 wheelhouse hosts one -- step 13b installs that copy itself.
|
||||
actual = _filter_requirements(actual, {"torchcodec"})
|
||||
temps.append(actual)
|
||||
return actual, temps
|
||||
|
|
@ -10032,6 +10858,10 @@ def install_python_stack() -> int:
|
|||
# beside a physical GPU.
|
||||
torch_flavor_tag = _expected_torch_flavor_tag()
|
||||
|
||||
# 13x. Optional win_arm64 features whose released metadata excludes them outright. After the
|
||||
# invariant above, so xformers is validated against the torch the install ends with.
|
||||
_install_wheelhouse_optionals()
|
||||
|
||||
# 13b. torchcodec, pinned to the venv's torch minor (_select_torchcodec_spec), which
|
||||
# extras-no-deps.txt cannot do because markers cannot see torch. Must run after the
|
||||
# repair above: that can move torch onto another minor, staling an earlier choice.
|
||||
|
|
@ -10039,14 +10869,38 @@ def install_python_stack() -> int:
|
|||
# tolerate), so read the installed metadata before giving up: guessing here means
|
||||
# downgrading a matching codec onto the default and recreating the mismatch.
|
||||
_codec_torch_ver = None
|
||||
if not NO_TORCH and not PLATFORM_LACKS_TORCHCODEC_WHEEL:
|
||||
_codec_hosted = None
|
||||
if not NO_TORCH and (not PLATFORM_LACKS_TORCHCODEC_WHEEL or _wheelhouse_hosts("torchcodec")):
|
||||
_codec_torch_ver = _probe_installed_torch_version() or _installed_distribution_version(
|
||||
"torch"
|
||||
)
|
||||
if not NO_TORCH and PLATFORM_LACKS_TORCHCODEC_WHEEL:
|
||||
_codec_hosted = _wheelhouse_torchcodec_version(_codec_torch_ver)
|
||||
if NO_TORCH:
|
||||
_progress("torchcodec (skipped, no torch)")
|
||||
elif _codec_hosted:
|
||||
# The wheelhouse stands in for the wheel this platform lacks: --no-deps and pinned to
|
||||
# the hosted version inside the resident torch's window, like the other hosted optionals.
|
||||
_progress("torchcodec")
|
||||
if pip_install_try(
|
||||
f"Installing torchcodec=={_codec_hosted} from the Windows on ARM wheelhouse",
|
||||
"--no-deps",
|
||||
"--no-cache-dir",
|
||||
f"torchcodec=={_codec_hosted}",
|
||||
constrain = False,
|
||||
):
|
||||
_note(f"windows on arm: installed torchcodec=={_codec_hosted} from the wheelhouse")
|
||||
else:
|
||||
_note(
|
||||
"windows on arm: could not install the wheelhouse torchcodec -- audio decoding stays disabled"
|
||||
)
|
||||
elif PLATFORM_LACKS_TORCHCODEC_WHEEL:
|
||||
_progress("torchcodec (skipped, no wheel for this platform)")
|
||||
if _wheelhouse_hosts("torchcodec"):
|
||||
_note(
|
||||
f"windows on arm: the wheelhouse torchcodec is outside the window torch "
|
||||
f"{_codec_torch_ver or 'unknown'} selects -- leaving audio decoding disabled"
|
||||
)
|
||||
elif not _codec_torch_ver:
|
||||
_progress("torchcodec (skipped, torch version unknown)")
|
||||
_note("could not read the installed torch version -- leaving torchcodec alone")
|
||||
|
|
@ -10240,6 +11094,7 @@ def install_python_stack() -> int:
|
|||
expected_torch_tag = _recordable_torch_flavor_tag(torch_flavor_tag),
|
||||
expected_torch_tag_pinned = bool(_recordable_torch_flavor_tag(torch_flavor_tag))
|
||||
and _expected_torch_flavor_was_pinned(_recordable_torch_flavor_tag(torch_flavor_tag)),
|
||||
woa_torch_index = os.environ.get("UNSLOTH_WOA_SELECTED_TORCH_INDEX"),
|
||||
# What the NEXT run may skip. Digested from the CURRENT REQ_ROOT, which the core step
|
||||
# may have replaced: a file that moved during this pass must not read as unchanged.
|
||||
extra = {
|
||||
|
|
|
|||
|
|
@ -932,6 +932,31 @@ def release_asset_map(release: dict[str, Any]) -> dict[str, str]:
|
|||
}
|
||||
|
||||
|
||||
def release_asset_digests(release: dict[str, Any]) -> dict[str, str]:
|
||||
"""asset name -> bare sha256 hex, for the assets whose digest GitHub reports.
|
||||
|
||||
Separate from release_asset_map because the name -> url mapping is what nearly every
|
||||
caller wants and widening its return type would touch all of them. GitHub returns
|
||||
`digest` as an algorithm-prefixed string ("sha256:<hex>"); anything else is skipped
|
||||
rather than guessed at, so a future algorithm cannot be read as a sha256.
|
||||
"""
|
||||
assets = release.get("assets")
|
||||
if not isinstance(assets, list):
|
||||
return {}
|
||||
digests: dict[str, str] = {}
|
||||
for asset in assets:
|
||||
if not isinstance(asset, dict) or not isinstance(asset.get("name"), str):
|
||||
continue
|
||||
raw = asset.get("digest")
|
||||
# The prefix is required, not stripped for convenience: normalize_sha256_digest accepts a bare 64-hex string, and an unprefixed digest is one whose algorithm GitHub did not state.
|
||||
if not isinstance(raw, str) or not raw.lower().startswith("sha256:"):
|
||||
continue
|
||||
digest = normalize_sha256_digest(raw)
|
||||
if digest:
|
||||
digests[asset["name"]] = digest
|
||||
return digests
|
||||
|
||||
|
||||
def github_release(
|
||||
ops: ModuleOps,
|
||||
repo: str,
|
||||
|
|
|
|||
1012
studio/setup.ps1
1012
studio/setup.ps1
File diff suppressed because it is too large
Load diff
|
|
@ -1,3 +1,6 @@
|
|||
// Building for aarch64-pc-windows-msvc needs clang on PATH: ring and aws-lc-sys assemble their
|
||||
// aarch64 sources through cc-rs, which reaches for clang rather than cl.exe, and the failure
|
||||
// names ring rather than the missing toolchain. No other leg builds for aarch64 Windows.
|
||||
fn main() {
|
||||
tauri_build::build()
|
||||
}
|
||||
|
|
|
|||
|
|
@ -704,17 +704,19 @@ mod tests {
|
|||
let state = new_update_state();
|
||||
state.lock().unwrap().child = Some(child);
|
||||
|
||||
for _ in 0..50 {
|
||||
if child_pid_file.is_file() {
|
||||
break;
|
||||
// Wait for a pid that PARSES, not merely for the path to appear: the redirection above
|
||||
// creates the file before anything is written, so is_file() can win that race and read "".
|
||||
let mut descendant = None;
|
||||
for _ in 0..100 {
|
||||
if let Ok(text) = std::fs::read_to_string(&child_pid_file) {
|
||||
if let Ok(pid) = text.trim().parse::<i32>() {
|
||||
descendant = Some(pid);
|
||||
break;
|
||||
}
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(20));
|
||||
}
|
||||
let descendant = std::fs::read_to_string(&child_pid_file)
|
||||
.unwrap()
|
||||
.trim()
|
||||
.parse::<i32>()
|
||||
.unwrap();
|
||||
let descendant = descendant.expect("the test child never wrote a usable descendant pid");
|
||||
|
||||
stop_update(&state).unwrap();
|
||||
|
||||
|
|
|
|||
391
tests/python/test_sign_wheel_binaries.py
Normal file
391
tests/python/test_sign_wheel_binaries.py
Normal file
|
|
@ -0,0 +1,391 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""The wheelhouse signer: what it signs, what it refuses, and what it must not break.
|
||||
|
||||
A .whl cannot be Authenticode signed, so the native binaries inside it are, and the wheel is
|
||||
repacked. That makes RECORD the thing most likely to go quietly wrong -- signing changes every
|
||||
signed member's bytes, and a stale RECORD ships a wheel that reads as corrupt to anything that
|
||||
checks it. The signing call itself is stubbed here: this is about the packing, the refusals and
|
||||
the RECORD, none of which need Azure.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import struct
|
||||
import sys
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
SIGNER_PY = REPO_ROOT / ".github" / "scripts" / "sign_wheel_binaries.py"
|
||||
WORKFLOW = REPO_ROOT / ".github" / "workflows" / "woa-wheelhouse.yml"
|
||||
|
||||
|
||||
@pytest.fixture(scope = "module")
|
||||
def mod():
|
||||
spec = importlib.util.spec_from_file_location("_sign_wheel_binaries", SIGNER_PY)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def _pe(
|
||||
machine: int = 0xAA64,
|
||||
*,
|
||||
signed: bool = False,
|
||||
plus: bool = True,
|
||||
) -> bytes:
|
||||
"""The smallest byte string the reader will accept as a PE image.
|
||||
|
||||
Built rather than vendored: a real .pyd in the tree would be a binary nobody can review,
|
||||
and every field this tool reads is one of the four below.
|
||||
"""
|
||||
header_at = 0x80
|
||||
blob = bytearray(b"\0" * 0x400)
|
||||
blob[0:2] = b"MZ"
|
||||
struct.pack_into("<I", blob, 0x3C, header_at)
|
||||
blob[header_at : header_at + 4] = b"PE\0\0"
|
||||
struct.pack_into("<H", blob, header_at + 4, machine)
|
||||
magic_at = header_at + 24
|
||||
struct.pack_into("<H", blob, magic_at, 0x20B if plus else 0x10B)
|
||||
directory_at = magic_at + (112 if plus else 96) + 4 * 8
|
||||
if signed:
|
||||
struct.pack_into("<II", blob, directory_at, 0x300, 0x40)
|
||||
return bytes(blob)
|
||||
|
||||
|
||||
def _record(names_and_blobs) -> str:
|
||||
lines = []
|
||||
for name, blob in names_and_blobs:
|
||||
digest = base64.urlsafe_b64encode(hashlib.sha256(blob).digest()).rstrip(b"=").decode()
|
||||
lines.append(f"{name},sha256={digest},{len(blob)}")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def _wheel(path: Path, members: dict) -> Path:
|
||||
"""A wheel whose RECORD matches its contents, the way a real one arrives."""
|
||||
dist = "demo-1.0.dist-info"
|
||||
record_name = f"{dist}/RECORD"
|
||||
body = dict(members)
|
||||
body.setdefault(f"{dist}/WHEEL", b"Wheel-Version: 1.0\nTag: cp313-cp313-win_arm64\n")
|
||||
body.setdefault(f"{dist}/METADATA", b"Name: demo\nVersion: 1.0\n")
|
||||
with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as z:
|
||||
for name, blob in body.items():
|
||||
z.writestr(name, blob)
|
||||
z.writestr(record_name, _record(list(body.items())) + f"{record_name},,\n")
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fake_signer(tmp_path):
|
||||
"""Stands in for trusted-signing-cli: appends a certificate table to the file it is given.
|
||||
|
||||
The real one talks to Azure. What this file's tests are about is everything around that
|
||||
call, so the stub does the one thing the caller checks for -- the file comes back signed.
|
||||
"""
|
||||
script = tmp_path / "fake_signer.py"
|
||||
script.write_text(
|
||||
"import struct, sys\n"
|
||||
"from pathlib import Path\n"
|
||||
"p = Path(sys.argv[1]); d = bytearray(p.read_bytes())\n"
|
||||
"h = struct.unpack_from('<I', d, 0x3C)[0]\n"
|
||||
"magic_at = h + 24\n"
|
||||
"magic = struct.unpack_from('<H', d, magic_at)[0]\n"
|
||||
"at = magic_at + (112 if magic == 0x20b else 96) + 4 * 8\n"
|
||||
"struct.pack_into('<II', d, at, len(d), 0x40)\n"
|
||||
"d += b'CERTIFICATE'\n"
|
||||
"p.write_bytes(bytes(d))\n",
|
||||
encoding = "utf-8",
|
||||
)
|
||||
return [sys.executable, str(script)]
|
||||
|
||||
|
||||
class TestWhatCountsAsAPeImage:
|
||||
def test_a_pe_is_recognised(self, mod):
|
||||
assert mod.pe_machine(_pe()) == 0xAA64
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"blob, why",
|
||||
[
|
||||
(b"", "empty"),
|
||||
(b"not a pe at all, just text in a wheel", "plain data"),
|
||||
(b"MZ" + b"\0" * 8, "an MZ header too short to hold an offset"),
|
||||
(b"MZ" + b"\0" * 0x200, "an MZ header whose PE offset points at zeros"),
|
||||
],
|
||||
)
|
||||
def test_everything_else_is_not(self, mod, blob, why):
|
||||
assert mod.pe_machine(blob) is None, why
|
||||
|
||||
def test_an_unsigned_image_reads_as_unsigned(self, mod):
|
||||
assert mod.is_signed(_pe()) is False
|
||||
|
||||
@pytest.mark.parametrize("plus", [True, False], ids = ["PE32+", "PE32"])
|
||||
def test_a_signed_image_reads_as_signed(self, mod, plus):
|
||||
"""PE32 and PE32+ put the data directory at different offsets; a reader that assumed
|
||||
one would call every 32-bit signed DLL unsigned and re-sign somebody else's binary."""
|
||||
assert mod.is_signed(_pe(signed = True, plus = plus)) is True
|
||||
|
||||
|
||||
class TestSigningAWheel:
|
||||
def test_every_unsigned_binary_is_signed(self, mod, tmp_path, fake_signer):
|
||||
wheel = _wheel(
|
||||
tmp_path / "demo-1.0-cp313-cp313-win_arm64.whl",
|
||||
{
|
||||
"demo/__init__.py": b"x = 1\n",
|
||||
"demo/_core.cp313-win_arm64.pyd": _pe(),
|
||||
"demo/helper.dll": _pe(),
|
||||
},
|
||||
)
|
||||
out, signed, skipped = mod.sign_wheel(wheel, tmp_path / "out", fake_signer)
|
||||
assert sorted(signed) == ["demo/_core.cp313-win_arm64.pyd", "demo/helper.dll"]
|
||||
assert skipped == []
|
||||
with zipfile.ZipFile(out) as z:
|
||||
for name in signed:
|
||||
assert mod.is_signed(z.read(name)), name
|
||||
|
||||
def test_an_already_signed_binary_is_left_alone(self, mod, tmp_path, fake_signer):
|
||||
"""delvewheel vendors Microsoft's redistributables into pyarrow already signed by
|
||||
Microsoft. Re-signing one replaces their signature with ours."""
|
||||
original = _pe(signed = True)
|
||||
wheel = _wheel(
|
||||
tmp_path / "demo-1.0-cp313-cp313-win_arm64.whl",
|
||||
{
|
||||
"demo/_core.cp313-win_arm64.pyd": _pe(),
|
||||
"demo.libs/msvcp140-abcdef.dll": original,
|
||||
},
|
||||
)
|
||||
out, signed, skipped = mod.sign_wheel(wheel, tmp_path / "out", fake_signer)
|
||||
assert signed == ["demo/_core.cp313-win_arm64.pyd"]
|
||||
assert skipped == ["demo.libs/msvcp140-abcdef.dll"]
|
||||
with zipfile.ZipFile(out) as z:
|
||||
assert z.read("demo.libs/msvcp140-abcdef.dll") == original, "byte-for-byte untouched"
|
||||
|
||||
def test_the_wheel_keeps_its_name(self, mod, tmp_path, fake_signer):
|
||||
wheel = _wheel(
|
||||
tmp_path / "demo-1.0-cp313-cp313-win_arm64.whl",
|
||||
{
|
||||
"demo/_core.cp313-win_arm64.pyd": _pe(),
|
||||
},
|
||||
)
|
||||
out, _, _ = mod.sign_wheel(wheel, tmp_path / "out", fake_signer)
|
||||
assert out.name == wheel.name, "the index and every pin name the wheel by filename"
|
||||
|
||||
def test_non_binary_members_are_carried_through_unchanged(self, mod, tmp_path, fake_signer):
|
||||
wheel = _wheel(
|
||||
tmp_path / "demo-1.0-cp313-cp313-win_arm64.whl",
|
||||
{
|
||||
"demo/__init__.py": b"x = 1\n",
|
||||
"demo/data.json": b'{"a": 1}',
|
||||
"demo/_core.cp313-win_arm64.pyd": _pe(),
|
||||
},
|
||||
)
|
||||
out, _, _ = mod.sign_wheel(wheel, tmp_path / "out", fake_signer)
|
||||
with zipfile.ZipFile(out) as z:
|
||||
assert z.read("demo/__init__.py") == b"x = 1\n"
|
||||
assert z.read("demo/data.json") == b'{"a": 1}'
|
||||
|
||||
def test_member_order_is_preserved(self, mod, tmp_path, fake_signer):
|
||||
members = {
|
||||
"demo/__init__.py": b"x = 1\n",
|
||||
"demo/_core.cp313-win_arm64.pyd": _pe(),
|
||||
"demo/zzz.py": b"pass\n",
|
||||
}
|
||||
wheel = _wheel(tmp_path / "demo-1.0-cp313-cp313-win_arm64.whl", members)
|
||||
with zipfile.ZipFile(wheel) as z:
|
||||
before = z.namelist()
|
||||
out, _, _ = mod.sign_wheel(wheel, tmp_path / "out", fake_signer)
|
||||
with zipfile.ZipFile(out) as z:
|
||||
assert z.namelist() == before, "a diff should show signatures and nothing else"
|
||||
|
||||
|
||||
class TestTheRecordIsRegenerated:
|
||||
"""Signing appends a certificate table, so every signed member's digest and length move.
|
||||
A wheel shipped with its original RECORD is one `pip check` away from reading as corrupt."""
|
||||
|
||||
@staticmethod
|
||||
def _rows(wheel: Path) -> dict:
|
||||
with zipfile.ZipFile(wheel) as z:
|
||||
record = next(n for n in z.namelist() if n.endswith(".dist-info/RECORD"))
|
||||
rows = {}
|
||||
for line in z.read(record).decode().splitlines():
|
||||
if line.strip():
|
||||
parts = line.split(",")
|
||||
rows[parts[0]] = (parts[1], parts[2])
|
||||
return rows
|
||||
|
||||
def test_every_member_digest_matches_the_bytes_shipped(self, mod, tmp_path, fake_signer):
|
||||
wheel = _wheel(
|
||||
tmp_path / "demo-1.0-cp313-cp313-win_arm64.whl",
|
||||
{
|
||||
"demo/__init__.py": b"x = 1\n",
|
||||
"demo/_core.cp313-win_arm64.pyd": _pe(),
|
||||
},
|
||||
)
|
||||
out, _, _ = mod.sign_wheel(wheel, tmp_path / "out", fake_signer)
|
||||
rows = self._rows(out)
|
||||
with zipfile.ZipFile(out) as z:
|
||||
for name in z.namelist():
|
||||
if name.endswith(".dist-info/RECORD"):
|
||||
continue
|
||||
blob = z.read(name)
|
||||
expected = base64.urlsafe_b64encode(hashlib.sha256(blob).digest()).rstrip(b"=")
|
||||
assert rows[name] == (f"sha256={expected.decode()}", str(len(blob))), name
|
||||
|
||||
def test_the_signed_binary_digest_actually_changed(self, mod, tmp_path, fake_signer):
|
||||
wheel = _wheel(
|
||||
tmp_path / "demo-1.0-cp313-cp313-win_arm64.whl",
|
||||
{
|
||||
"demo/_core.cp313-win_arm64.pyd": _pe(),
|
||||
},
|
||||
)
|
||||
before = self._rows(wheel)["demo/_core.cp313-win_arm64.pyd"]
|
||||
out, _, _ = mod.sign_wheel(wheel, tmp_path / "out", fake_signer)
|
||||
after = self._rows(out)["demo/_core.cp313-win_arm64.pyd"]
|
||||
assert (
|
||||
before != after
|
||||
), "the fixture did not really change the bytes, so this proves nothing"
|
||||
|
||||
def test_record_names_itself_without_a_digest(self, mod, tmp_path, fake_signer):
|
||||
"""PEP 376: RECORD cannot hash itself, and a tool that writes one anyway makes the
|
||||
file self-contradictory."""
|
||||
wheel = _wheel(
|
||||
tmp_path / "demo-1.0-cp313-cp313-win_arm64.whl",
|
||||
{
|
||||
"demo/_core.cp313-win_arm64.pyd": _pe(),
|
||||
},
|
||||
)
|
||||
out, _, _ = mod.sign_wheel(wheel, tmp_path / "out", fake_signer)
|
||||
assert self._rows(out)["demo-1.0.dist-info/RECORD"] == ("", "")
|
||||
|
||||
|
||||
class TestItRefusesRatherThanShipSomethingWrong:
|
||||
def test_a_foreign_architecture_is_refused(self, mod, tmp_path, fake_signer):
|
||||
"""0x8664 is x64. A win_arm64 filename says nothing about what was linked in, and
|
||||
signing one would publish a correctly signed wheel that cannot load."""
|
||||
wheel = _wheel(
|
||||
tmp_path / "demo-1.0-cp313-cp313-win_arm64.whl",
|
||||
{
|
||||
"demo/_core.cp313-win_arm64.pyd": _pe(machine = 0x8664),
|
||||
},
|
||||
)
|
||||
with pytest.raises(SystemExit, match = "0x8664"):
|
||||
mod.sign_wheel(wheel, tmp_path / "out", fake_signer)
|
||||
|
||||
def test_a_foreign_architecture_is_allowed_when_asked(self, mod, tmp_path, fake_signer):
|
||||
wheel = _wheel(
|
||||
tmp_path / "demo-1.0-cp313-cp313-win_amd64.whl",
|
||||
{
|
||||
"demo/_core.cp313-win_amd64.pyd": _pe(machine = 0x8664),
|
||||
},
|
||||
)
|
||||
_, signed, _ = mod.sign_wheel(wheel, tmp_path / "out", fake_signer, require_machine = None)
|
||||
assert signed == ["demo/_core.cp313-win_amd64.pyd"]
|
||||
|
||||
def test_a_wheel_with_no_binaries_is_refused(self, mod, tmp_path, fake_signer):
|
||||
"""Nothing to sign means the build produced a pure-Python wheel where a native one was
|
||||
expected. Publishing it silently is how an unsigned wheelhouse comes back."""
|
||||
wheel = _wheel(tmp_path / "demo-1.0-py3-none-win_arm64.whl", {"demo/__init__.py": b"x\n"})
|
||||
with pytest.raises(SystemExit, match = "no PE images"):
|
||||
mod.sign_wheel(wheel, tmp_path / "out", fake_signer)
|
||||
|
||||
def test_a_signer_that_fails_fails_the_run(self, mod, tmp_path):
|
||||
wheel = _wheel(
|
||||
tmp_path / "demo-1.0-cp313-cp313-win_arm64.whl",
|
||||
{
|
||||
"demo/_core.cp313-win_arm64.pyd": _pe(),
|
||||
},
|
||||
)
|
||||
failing = [sys.executable, "-c", "import sys; sys.exit(3)"]
|
||||
with pytest.raises(SystemExit, match = "signing failed"):
|
||||
mod.sign_wheel(wheel, tmp_path / "out", failing)
|
||||
|
||||
def test_a_signer_that_silently_does_nothing_fails_the_run(self, mod, tmp_path):
|
||||
"""Exit 0 with the file untouched is the dangerous case: it would publish an unsigned
|
||||
wheel while every log line says it was signed."""
|
||||
wheel = _wheel(
|
||||
tmp_path / "demo-1.0-cp313-cp313-win_arm64.whl",
|
||||
{
|
||||
"demo/_core.cp313-win_arm64.pyd": _pe(),
|
||||
},
|
||||
)
|
||||
noop = [sys.executable, "-c", "import sys"]
|
||||
with pytest.raises(SystemExit, match = "no certificate after signing"):
|
||||
mod.sign_wheel(wheel, tmp_path / "out", noop)
|
||||
|
||||
|
||||
class TestTheWorkflowUsesTheProcedureTheDesktopReleaseUses:
|
||||
"""One signing procedure, one definition. A second copy of the endpoint or the identity
|
||||
drifts, and the drift is invisible until a release is signed by the wrong certificate."""
|
||||
|
||||
@pytest.fixture(scope = "class")
|
||||
def text(self):
|
||||
return WORKFLOW.read_text(encoding = "utf-8")
|
||||
|
||||
def test_it_delegates_to_the_shared_signing_helper(self, text):
|
||||
assert "studio/src-tauri/windows/sign-with-trusted-signing.ps1" in text
|
||||
assert "codesigning.azure.net" not in text, "the endpoint belongs to the helper only"
|
||||
|
||||
def test_the_signing_cli_is_digest_pinned(self, text):
|
||||
assert "TRUSTED_SIGNING_CLI_SHA256" in text
|
||||
desktop = (REPO_ROOT / ".github" / "workflows" / "release-desktop.yml").read_text(
|
||||
encoding = "utf-8"
|
||||
)
|
||||
import re
|
||||
|
||||
pinned = re.findall(r'TRUSTED_SIGNING_CLI_SHA256:\s*"([0-9a-f]{64})"', text)
|
||||
assert pinned, "no digest pin"
|
||||
assert pinned[0] in desktop, "the two workflows pin different signing binaries"
|
||||
|
||||
def test_the_secrets_are_gated_by_an_environment(self, text):
|
||||
assert "environment: release-signing" in text, (
|
||||
"without an environment there is nowhere to hang a required reviewer, so anyone "
|
||||
"who can dispatch this can sign under our identity"
|
||||
)
|
||||
|
||||
def test_only_the_signing_job_reads_the_azure_secrets(self, text):
|
||||
import yaml
|
||||
data = yaml.safe_load(text)
|
||||
for name, job in data["jobs"].items():
|
||||
uses_azure = "AZURE_" in yaml.safe_dump(job)
|
||||
if uses_azure:
|
||||
assert name == "sign", f"{name} reads the signing secrets"
|
||||
assert job.get("environment") == "release-signing"
|
||||
|
||||
def test_it_cannot_be_reached_from_a_pull_request(self, text):
|
||||
import yaml
|
||||
|
||||
data = yaml.safe_load(text)
|
||||
triggers = data[True] if True in data else data["on"]
|
||||
assert set(triggers) == {
|
||||
"workflow_dispatch"
|
||||
}, "this job holds signing credentials and writes a release"
|
||||
|
||||
def test_publishing_is_opt_in(self, text):
|
||||
import yaml
|
||||
|
||||
data = yaml.safe_load(text)
|
||||
publish = data[True]["workflow_dispatch"]["inputs"]["publish"]
|
||||
assert publish["default"] is False, "a dispatch must not publish unless asked"
|
||||
assert "inputs.publish" in str(data["jobs"]["publish"]["if"])
|
||||
|
||||
def test_write_permission_is_scoped_to_the_publish_job(self, text):
|
||||
import yaml
|
||||
|
||||
data = yaml.safe_load(text)
|
||||
assert data["permissions"] == {"contents": "read"}, "the default must be read"
|
||||
assert data["jobs"]["publish"]["permissions"] == {"contents": "write"}
|
||||
for name, job in data["jobs"].items():
|
||||
if name != "publish":
|
||||
assert "permissions" not in job or job["permissions"].get("contents") != "write"
|
||||
|
||||
def test_the_published_set_is_checked_before_signing(self, text):
|
||||
"""A short matrix leg would otherwise publish a wheelhouse missing an interpreter,
|
||||
which reads downstream as 'no wheel for this Python' rather than as a failed build."""
|
||||
assert "expected 5 wheels" in text
|
||||
344
tests/python/test_windows_arm64_requirement_markers.py
Normal file
344
tests/python/test_windows_arm64_requirement_markers.py
Normal file
|
|
@ -0,0 +1,344 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
|
||||
|
||||
"""Windows on ARM is pinned by splitting rows, so the split must be a true partition.
|
||||
|
||||
Every package that needs a different version on win_arm64 is written as two rows:
|
||||
|
||||
X==old ; sys_platform != "win32" or platform_machine != "ARM64"
|
||||
X>=new ; sys_platform == "win32" and platform_machine == "ARM64"
|
||||
|
||||
The second marker is the exact complement of the first, so in every environment exactly one
|
||||
row is live. Getting that wrong is silent either way: an OVERLAP makes pip intersect two
|
||||
specifiers and can render the row unsatisfiable, a GAP drops the package on some platform
|
||||
nobody tested. The compare is case-sensitive, which is what isolates Windows on ARM (macOS
|
||||
reports ``arm64`` and Linux ``aarch64``), so that is asserted here too.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import itertools
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from packaging.requirements import Requirement
|
||||
from packaging.specifiers import SpecifierSet
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
REQ_FILES = [
|
||||
REPO_ROOT / "studio/backend/requirements/extras.txt",
|
||||
REPO_ROOT / "studio/backend/requirements/no-torch-runtime.txt",
|
||||
REPO_ROOT / "studio/backend/requirements/single-env/constraints.txt",
|
||||
REPO_ROOT / "studio/backend/requirements/studio.txt",
|
||||
]
|
||||
|
||||
# (sys_platform, platform_system, platform_machine, os_name)
|
||||
PLATFORMS = [
|
||||
("linux", "Linux", "x86_64", "posix"),
|
||||
("linux", "Linux", "aarch64", "posix"),
|
||||
("linux", "Linux", "armv7l", "posix"),
|
||||
("linux", "Linux", "ppc64le", "posix"),
|
||||
("linux", "Linux", "s390x", "posix"),
|
||||
("darwin", "Darwin", "arm64", "posix"),
|
||||
("darwin", "Darwin", "x86_64", "posix"),
|
||||
("win32", "Windows", "AMD64", "nt"),
|
||||
("win32", "Windows", "x86", "nt"),
|
||||
("win32", "Windows", "ARM64", "nt"),
|
||||
]
|
||||
PYTHONS = ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
|
||||
WOA = ("win32", "Windows", "ARM64", "nt")
|
||||
|
||||
|
||||
def _env(plat, py):
|
||||
sys_platform, platform_system, platform_machine, os_name = plat
|
||||
return {
|
||||
"implementation_name": "cpython",
|
||||
"implementation_version": f"{py}.0",
|
||||
"os_name": os_name,
|
||||
"platform_machine": platform_machine,
|
||||
"platform_python_implementation": "CPython",
|
||||
"platform_release": "",
|
||||
"platform_system": platform_system,
|
||||
"platform_version": "",
|
||||
"python_full_version": f"{py}.0",
|
||||
"python_version": py,
|
||||
"sys_platform": sys_platform,
|
||||
"extra": "",
|
||||
}
|
||||
|
||||
|
||||
#: Every environment the rows below are evaluated in, built once.
|
||||
ENVS = [(plat, py, _env(plat, py)) for plat, py in itertools.product(PLATFORMS, PYTHONS)]
|
||||
|
||||
|
||||
def _live(rows, env):
|
||||
"""The rows pip would install in `env`."""
|
||||
return [r for r in rows if r.marker is None or r.marker.evaluate(env)]
|
||||
|
||||
|
||||
def _rows(path: Path) -> list[Requirement]:
|
||||
out = []
|
||||
for raw in path.read_text(encoding = "utf-8").splitlines():
|
||||
line = raw.strip()
|
||||
if not line or line.startswith(("#", "-")):
|
||||
continue
|
||||
line = line.split("#", 1)[0].strip()
|
||||
if line:
|
||||
out.append(Requirement(line))
|
||||
return out
|
||||
|
||||
|
||||
def _by_name(reqs: list[Requirement]) -> dict[str, list[Requirement]]:
|
||||
grouped: dict[str, list[Requirement]] = {}
|
||||
for req in reqs:
|
||||
grouped.setdefault(req.name.lower().replace("_", "-"), []).append(req)
|
||||
return grouped
|
||||
|
||||
|
||||
def _multi_row_groups(reqs):
|
||||
"""Packages stated more than once, minus the ones that are different targets.
|
||||
|
||||
unsloth[a] and unsloth[b] may legitimately co-exist, so a group whose rows differ in
|
||||
their extras is not a platform split at all.
|
||||
"""
|
||||
for name, group in _by_name(reqs).items():
|
||||
if len(group) < 2 or len({tuple(sorted(r.extras)) for r in group}) > 1:
|
||||
continue
|
||||
yield name, group
|
||||
|
||||
|
||||
def _markers(group) -> list[str]:
|
||||
return [str(r.marker).replace("'", '"') for r in group if r.marker is not None]
|
||||
|
||||
|
||||
def _pyproject_extras() -> dict[str, list[Requirement]]:
|
||||
try:
|
||||
import tomllib
|
||||
except ModuleNotFoundError: # pragma: no cover - 3.10 runs use tomli
|
||||
import tomli as tomllib
|
||||
data = tomllib.loads((REPO_ROOT / "pyproject.toml").read_text(encoding = "utf-8"))
|
||||
out = {}
|
||||
for extra, lines in data["project"]["optional-dependencies"].items():
|
||||
out[extra] = [Requirement(l) for l in lines]
|
||||
return out
|
||||
|
||||
|
||||
ALL_SOURCES: list[tuple[str, list[Requirement]]] = [(p.name, _rows(p)) for p in REQ_FILES] + [
|
||||
(f"pyproject[{k}]", v) for k, v in _pyproject_extras().items()
|
||||
]
|
||||
per_source = pytest.mark.parametrize("label,reqs", ALL_SOURCES, ids = [s[0] for s in ALL_SOURCES])
|
||||
|
||||
|
||||
@per_source
|
||||
def test_split_rows_never_overlap(label, reqs):
|
||||
"""Two rows for one package must never both be live: pip would intersect them."""
|
||||
for name, group in _multi_row_groups(reqs):
|
||||
for plat, py, env in ENVS:
|
||||
live = _live(group, env)
|
||||
assert len(live) <= 1, (
|
||||
f"{label}: {name} has {len(live)} live rows on "
|
||||
f"{plat[0]}/{plat[2]}/py{py}: {[str(r) for r in live]}"
|
||||
)
|
||||
|
||||
|
||||
@per_source
|
||||
def test_no_package_is_dropped_on_a_non_woa_platform(label, reqs):
|
||||
"""A split may remove a package on Windows ARM64 only."""
|
||||
if "constraints" in label:
|
||||
return # a constraints file may legitimately have no cap in force
|
||||
for name, group in _multi_row_groups(reqs):
|
||||
# Only the complement-pair shape; triton-windows is two disjoint Windows-only rows.
|
||||
markers = _markers(group)
|
||||
if not (
|
||||
any('platform_machine == "ARM64"' in m for m in markers)
|
||||
and any('platform_machine != "ARM64"' in m for m in markers)
|
||||
):
|
||||
continue
|
||||
for plat, py, env in ENVS:
|
||||
if plat == WOA:
|
||||
continue
|
||||
assert _live(
|
||||
group, env
|
||||
), f"{label}: {name} has no live row on {plat[0]}/{plat[2]}/py{py}"
|
||||
|
||||
|
||||
def test_arm64_marker_is_case_sensitive_and_windows_only():
|
||||
"""``ARM64`` must not match macOS ``arm64`` or Linux ``aarch64``."""
|
||||
woa = Requirement('x==1; sys_platform == "win32" and platform_machine == "ARM64"')
|
||||
# The complement really is the complement.
|
||||
other = Requirement('x==1; sys_platform != "win32" or platform_machine != "ARM64"')
|
||||
for plat, py, env in ENVS:
|
||||
live = woa.marker.evaluate(env)
|
||||
assert live == (plat == WOA), f"win-ARM64 marker fired on {plat[0]}/{plat[2]}"
|
||||
assert live != other.marker.evaluate(
|
||||
env
|
||||
), f"the two halves are not complementary on {plat[0]}/{plat[2]}"
|
||||
|
||||
|
||||
@per_source
|
||||
def test_no_row_is_dead_on_arrival(label, reqs):
|
||||
"""Every row must be live in at least one real environment."""
|
||||
for req in reqs:
|
||||
if req.marker is None:
|
||||
continue
|
||||
assert any(req.marker.evaluate(env) for _, _, env in ENVS), (
|
||||
f"{label}: `{req}` is live in none of the {len(ENVS)} environments tested, so it "
|
||||
f'can never install. A lowercase "arm64" next to sys_platform == "win32" is the '
|
||||
f'usual cause: Windows reports "ARM64".'
|
||||
)
|
||||
|
||||
|
||||
# Which packages carry a Windows-on-ARM row, PER SOURCE, and in which shape.
|
||||
# "split" -- a positive `platform_machine == "ARM64"` row giving a different version
|
||||
# "dropped" -- only the negative row, so the package is absent on Windows on ARM
|
||||
# Checked per source and by shape: studio.txt and pyproject[studio] mirror each other, so a global
|
||||
# check stays green when one loses a row.
|
||||
WOA_ROWS_BY_SOURCE = {
|
||||
"extras.txt": {"av": "split", "scikit-learn": "split"},
|
||||
"no-torch-runtime.txt": {"pymupdf": "split", "hf-transfer": "dropped", "sqlite-vec": "dropped"},
|
||||
"constraints.txt": {
|
||||
"av": "split",
|
||||
"cryptography": "split",
|
||||
"pandas": "split",
|
||||
"pyarrow": "split",
|
||||
},
|
||||
"studio.txt": {"cryptography": "split", "pandas": "split", "pymupdf": "split"},
|
||||
"pyproject[studio]": {"cryptography": "split", "pandas": "split", "pymupdf": "split"},
|
||||
"pyproject[triton]": {"triton-windows": "split"},
|
||||
"pyproject[huggingfacenotorch]": {"hf-transfer": "dropped"},
|
||||
"pyproject[windows]": {"xformers": "dropped"},
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("label,expected", sorted(WOA_ROWS_BY_SOURCE.items()))
|
||||
def test_the_woa_split_is_used_where_we_claim_it_is(label, expected):
|
||||
"""Guard against a Windows-on-ARM row silently disappearing in a future edit."""
|
||||
groups = _by_name(dict(ALL_SOURCES)[label])
|
||||
for name, shape in sorted(expected.items()):
|
||||
markers = _markers(groups.get(name, []))
|
||||
positive = [m for m in markers if 'platform_machine == "ARM64"' in m]
|
||||
negative = [m for m in markers if 'platform_machine != "ARM64"' in m]
|
||||
if shape == "split":
|
||||
assert positive, f"{label}: {name} lost its Windows-on-ARM row"
|
||||
else:
|
||||
assert negative and not positive, (
|
||||
f"{label}: {name} should be excluded on Windows on ARM by a negative "
|
||||
f"marker only; found positive={positive}"
|
||||
)
|
||||
|
||||
|
||||
# A package's OWN requires-python floor, for rows pinning into a range that does not exist for
|
||||
# every interpreter. Only floors above our own 3.9 can make a row unsatisfiable.
|
||||
PACKAGE_PYTHON_FLOORS = {
|
||||
"pandas": [(SpecifierSet(">=3.0"), (3, 11))],
|
||||
}
|
||||
|
||||
|
||||
def _minor(py: str) -> tuple:
|
||||
major, minor = py.split(".")
|
||||
return (int(major), int(minor))
|
||||
|
||||
|
||||
def _lowest_allowed(req) -> str:
|
||||
"""The smallest concrete version the row's specifier admits, for floor comparison."""
|
||||
lowers = [s.version for s in req.specifier if s.operator in (">=", "==", "~=", ">")]
|
||||
return lowers[0] if lowers else "0"
|
||||
|
||||
|
||||
@per_source
|
||||
def test_a_selected_row_is_installable_on_the_python_it_was_selected_for(label, reqs):
|
||||
"""Splitting on platform is not enough on its own: a row can be live for an interpreter
|
||||
that no release in its range supports, which is not a resolution failure anyone reads
|
||||
as a marker bug -- pip just reports that no version matches.
|
||||
"""
|
||||
for plat, py, env in ENVS:
|
||||
for req in _live(reqs, env):
|
||||
for spec, floor in PACKAGE_PYTHON_FLOORS.get(req.name.lower(), ()):
|
||||
# Does this row admit ONLY versions that need a newer interpreter?
|
||||
if not spec.contains(_lowest_allowed(req), prereleases = True):
|
||||
continue
|
||||
assert _minor(py) >= floor, (
|
||||
f"{label}: `{req}` is live on Python {py} {plat[2]}, but every "
|
||||
f"version it admits needs Python >= {floor[0]}.{floor[1]}. "
|
||||
"The row is unsatisfiable there; the marker needs a "
|
||||
"python_version bound as well as a platform one."
|
||||
)
|
||||
|
||||
|
||||
def test_the_woa_pandas_split_covers_every_supported_python():
|
||||
"""The complement of the test above: having added a python_version bound, no ARM64
|
||||
interpreter may be left with no pandas row at all.
|
||||
"""
|
||||
for label, reqs in ALL_SOURCES:
|
||||
rows = [r for r in reqs if r.name.lower() == "pandas"]
|
||||
if not rows:
|
||||
continue
|
||||
for py in PYTHONS:
|
||||
live = _live(rows, _env(WOA, py))
|
||||
assert len(live) == 1, (
|
||||
f"{label}: Windows ARM64 on Python {py} has {len(live)} live pandas "
|
||||
f"rows, expected exactly 1: {[str(r) for r in live]}"
|
||||
)
|
||||
if _minor(py) < (3, 11):
|
||||
assert "3.0" not in str(
|
||||
live[0].specifier
|
||||
), f"{label}: Python {py} must not be handed the pandas 3 row"
|
||||
|
||||
|
||||
# install_python_stack.py, loaded so the skip list is read rather than copied here.
|
||||
_SPEC = importlib.util.spec_from_file_location(
|
||||
"_ips_marker_skiplist", REPO_ROOT / "studio" / "install_python_stack.py"
|
||||
)
|
||||
IPS = importlib.util.module_from_spec(_SPEC)
|
||||
_SPEC.loader.exec_module(IPS)
|
||||
# Every name install_python_stack.py filters out of the requirements files on win_arm64.
|
||||
WOA_SKIPPED = {IPS._canonical_dist_name(n) for n in IPS.WINDOWS_ARM64_SKIP_PACKAGES}
|
||||
|
||||
|
||||
# Scoped to `studio` deliberately: it is the extra a Windows-on-ARM user installs. The other
|
||||
# 190-odd are x64 recipes, so an ARM64 marker there would assert what they never promised.
|
||||
WOA_INSTALLABLE_EXTRAS = ["studio"]
|
||||
per_extra = pytest.mark.parametrize("extra", WOA_INSTALLABLE_EXTRAS, ids = WOA_INSTALLABLE_EXTRAS)
|
||||
|
||||
|
||||
@per_extra
|
||||
def test_a_skipped_package_is_not_left_live_in_an_extra(extra):
|
||||
"""The runtime skip list cannot reach package METADATA, so the extra has to agree.
|
||||
|
||||
`pip install "unsloth[studio]"` never runs the installer's filtering: it resolves
|
||||
pyproject's rows directly, so a row left live on win_arm64 for a package with no wheel
|
||||
and no buildable sdist there fails the install outright. sqlite-vec was exactly this.
|
||||
"""
|
||||
live = [
|
||||
str(req)
|
||||
for req in _live(_pyproject_extras()[extra], _env(WOA, "3.13"))
|
||||
if IPS._canonical_dist_name(req.name) in WOA_SKIPPED
|
||||
]
|
||||
assert not live, (
|
||||
f"pyproject[{extra}] leaves these live on Windows ARM64 even though the installer "
|
||||
f"treats them as unavailable there, so `pip install unsloth[{extra}]` cannot "
|
||||
f"resolve: {live}"
|
||||
)
|
||||
|
||||
|
||||
@per_extra
|
||||
def test_dropping_a_package_on_woa_drops_it_nowhere_else(extra):
|
||||
"""A negative ARM64 marker is a scalpel: every other platform keeps the row.
|
||||
|
||||
Checked as an outcome rather than a spelling, over the same platform table the rest of
|
||||
this file uses, so a marker that reads correctly but excludes (say) Windows x86 as well
|
||||
is still caught.
|
||||
"""
|
||||
for req in _pyproject_extras()[extra]:
|
||||
if IPS._canonical_dist_name(req.name) not in WOA_SKIPPED or req.marker is None:
|
||||
continue
|
||||
for plat, py, env in ENVS:
|
||||
if plat == WOA:
|
||||
continue
|
||||
assert req.marker.evaluate(env), (
|
||||
f"pyproject[{extra}] {req.name} is dropped on {plat[0]}/{plat[2]}/"
|
||||
f"py{py} too, which is not what the ARM64 marker is for"
|
||||
)
|
||||
1476
tests/python/test_windows_arm64_wheelhouse_tags.py
Normal file
1476
tests/python/test_windows_arm64_wheelhouse_tags.py
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -1679,6 +1679,35 @@ def test_the_private_temp_removal_only_takes_what_it_created(tmp_path: Path):
|
|||
assert (data / "studio.port").exists()
|
||||
|
||||
|
||||
# Split-Path's -LiteralPath lives in its own parameter set in Windows PowerShell 5.1, so naming
|
||||
# -Parent with it throws AmbiguousParameterSet at runtime, not at parse time, which is why eight of
|
||||
# them reached a release. -LiteralPath alone already splits off the parent, and -Path globs.
|
||||
# Static, because CI has no Windows PowerShell 5.1 to run the scripts under.
|
||||
_SPLIT_PATH_LITERAL_PARENT = re.compile(
|
||||
r"Split-Path\b[^\r\n|;]*?-LiteralPath\b[^\r\n|;]*?-Parent\b"
|
||||
r"|Split-Path\b[^\r\n|;]*?-Parent\b[^\r\n|;]*?-LiteralPath\b"
|
||||
)
|
||||
|
||||
|
||||
# The two installers this change touches. scripts/uninstall.ps1 has its own eight of these and its
|
||||
# own PR (#10471); listed here it failed on this tree unconditionally.
|
||||
@pytest.mark.parametrize("name", ("install.ps1", "studio/setup.ps1"))
|
||||
def test_split_path_never_pairs_literalpath_with_parent(name: str) -> None:
|
||||
text = (REPO_ROOT / name).read_text(encoding = "utf-8")
|
||||
offenders = [
|
||||
f"{name}:{number}: {line.strip()}"
|
||||
for number, line in enumerate(text.splitlines(), start = 1)
|
||||
# Comments are prose about the rule, not a call the shell binds.
|
||||
if not line.strip().startswith("#") and _SPLIT_PATH_LITERAL_PARENT.search(line)
|
||||
]
|
||||
assert not offenders, (
|
||||
"Split-Path -LiteralPath cannot be combined with -Parent: Windows PowerShell 5.1 "
|
||||
"resolves no parameter set for the pair and the call throws at runtime. Drop "
|
||||
"-Parent -- -LiteralPath alone already returns the parent, and unlike -Path it does "
|
||||
"not treat [ ] in an install root as a wildcard.\n " + "\n ".join(offenders)
|
||||
)
|
||||
|
||||
|
||||
SETUP_PS1 = REPO_ROOT / "studio" / "setup.ps1"
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -233,12 +233,23 @@ def test_installer_never_installs_an_unpinned_xformers():
|
|||
necessarily built for the CUDA family the resident torch came from. The installer
|
||||
picks the exact version for (torch, cuda) instead, so every spec must be pinned."""
|
||||
source = _source()
|
||||
# The Windows on ARM drop list names packages the installer REMOVES, so a bare name there is
|
||||
# the opposite of an install spec and has to match the requirement line exactly.
|
||||
drop_list = re.search(r"\$WoaDropCandidates = @\(.*?\n\s*\)\n", source, re.DOTALL)
|
||||
drop_span = drop_list.span() if drop_list else (-1, -1)
|
||||
# Same for the drop floors: the key is a package NAME being looked up, not an install spec.
|
||||
floors = re.search(r"\$WoaDropFloors = @\{[^}]*\}", source, re.DOTALL)
|
||||
floor_span = floors.span() if floors else (-1, -1)
|
||||
for match in re.finditer(r'"xformers[^"]*"', source):
|
||||
spec = match.group(0)
|
||||
# A wheel FILENAME is not a spec: it names one exact file and cannot resolve to anything else, which is the
|
||||
# whole point of preferring it.
|
||||
if spec.endswith('.whl"'):
|
||||
continue
|
||||
if drop_span[0] <= match.start() < drop_span[1]:
|
||||
continue
|
||||
if floor_span[0] <= match.start() < floor_span[1]:
|
||||
continue
|
||||
assert spec == '"xformers==$_xfVersion"', f"unpinned xFormers spec in install.ps1: {spec}"
|
||||
|
||||
|
||||
|
|
|
|||
167
tests/studio/install/test_arm64_cuda_digest_verification.py
Normal file
167
tests/studio/install/test_arm64_cuda_digest_verification.py
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""The Windows ARM64 CUDA bundle was the one archive this installer would install unchecked.
|
||||
|
||||
Every other prebuilt goes through apply_approved_hashes, which drops an attempt our checksum
|
||||
manifest does not cover. The ARM64 CUDA branch could not: we publish no windows-arm64-cuda
|
||||
artifact, so nothing covers upstream's zip, and refusing outright would mean no CUDA
|
||||
llama.cpp on this hardware at all. It logged that it had no hash and installed anyway.
|
||||
|
||||
GitHub's release API now reports a `digest` for every asset (ggml-org/llama.cpp b10853: 27 of
|
||||
27, including llama-*-bin-win-cuda-13.4-arm64.zip and its paired cudart archive). That is
|
||||
weaker than a manifest we compute ourselves rather than read from the host serving the bytes,
|
||||
but it pins the download to what the API listed. An asset GitHub states no digest for is now
|
||||
refused, so no path installs an unverified archive.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
REPO_ROOT = pathlib.Path(__file__).resolve().parents[3]
|
||||
sys.path.insert(0, str(REPO_ROOT / "studio"))
|
||||
|
||||
import install_llama_prebuilt as ip # noqa: E402
|
||||
import prebuilt_core # noqa: E402
|
||||
|
||||
SHA_A = "a" * 64
|
||||
SHA_B = "b" * 64
|
||||
|
||||
|
||||
def _release(*assets: dict) -> dict:
|
||||
return {"tag_name": "b10853", "assets": list(assets)}
|
||||
|
||||
|
||||
def _asset(name: str, digest: str | None) -> dict:
|
||||
out = {"name": name, "browser_download_url": f"https://example.invalid/{name}"}
|
||||
if digest is not None:
|
||||
out["digest"] = digest
|
||||
return out
|
||||
|
||||
|
||||
class TestTheDigestMapReadsOnlyWhatGitHubStated:
|
||||
def test_a_sha256_digest_is_read_without_its_prefix(self):
|
||||
release = _release(_asset("bundle.zip", f"sha256:{SHA_A}"))
|
||||
assert prebuilt_core.release_asset_digests(release) == {"bundle.zip": SHA_A}
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("digest", "why"),
|
||||
[
|
||||
(None, "no digest field at all"),
|
||||
("", "empty"),
|
||||
(SHA_A, "bare hex: GitHub did not say which algorithm produced it"),
|
||||
(f"sha512:{'c' * 128}", "a different algorithm"),
|
||||
("sha256:nothex" + "0" * 58, "not hex"),
|
||||
("sha256:" + "a" * 63, "too short"),
|
||||
("sha256:" + "a" * 65, "too long"),
|
||||
],
|
||||
ids = ["absent", "empty", "unprefixed", "sha512", "not-hex", "short", "long"],
|
||||
)
|
||||
def test_anything_else_is_left_out_rather_than_guessed_at(self, digest, why):
|
||||
release = _release(_asset("bundle.zip", digest))
|
||||
assert prebuilt_core.release_asset_digests(release) == {}, why
|
||||
|
||||
def test_an_upper_case_digest_still_matches_a_lower_case_file_hash(self):
|
||||
"""The verifier compares hex strings, so the case has to be normalised here."""
|
||||
release = _release(_asset("bundle.zip", "SHA256:" + SHA_A.upper()))
|
||||
assert prebuilt_core.release_asset_digests(release) == {"bundle.zip": SHA_A}
|
||||
|
||||
def test_a_release_with_no_assets_is_empty_not_an_error(self):
|
||||
assert prebuilt_core.release_asset_digests({"tag_name": "b1"}) == {}
|
||||
assert prebuilt_core.release_asset_digests({"assets": "not a list"}) == {}
|
||||
|
||||
|
||||
class TestAnAttemptWithNoDigestIsDropped:
|
||||
"""The rule that makes this safe. [] is what the caller reads as "refuse"."""
|
||||
|
||||
def _choice(
|
||||
self,
|
||||
name = "bundle.zip",
|
||||
runtime = None,
|
||||
):
|
||||
return ip.AssetChoice(
|
||||
repo = "ggml-org/llama.cpp",
|
||||
tag = "b10853",
|
||||
name = name,
|
||||
url = f"https://example.invalid/{name}",
|
||||
source_label = "upstream",
|
||||
install_kind = "windows-arm64-cuda",
|
||||
runtime_name = runtime,
|
||||
runtime_url = f"https://example.invalid/{runtime}" if runtime else None,
|
||||
)
|
||||
|
||||
def test_a_covered_attempt_carries_the_digest_forward(self):
|
||||
kept = ip._apply_release_digests([self._choice()], {"bundle.zip": SHA_A})
|
||||
assert [c.expected_sha256 for c in kept] == [SHA_A]
|
||||
|
||||
def test_an_uncovered_attempt_is_dropped_not_installed(self):
|
||||
assert ip._apply_release_digests([self._choice()], {"other.zip": SHA_A}) == []
|
||||
|
||||
def test_an_empty_digest_map_refuses_everything(self):
|
||||
assert ip._apply_release_digests([self._choice()], {}) == []
|
||||
|
||||
def test_a_paired_runtime_gets_its_own_digest(self):
|
||||
kept = ip._apply_release_digests(
|
||||
[self._choice(runtime = "cudart.zip")],
|
||||
{"bundle.zip": SHA_A, "cudart.zip": SHA_B},
|
||||
)
|
||||
assert kept[0].runtime_sha256 == SHA_B
|
||||
|
||||
def test_an_uncovered_runtime_unpairs_rather_than_riding_along(self):
|
||||
"""Dropping the whole attempt would lose CUDA over a missing side archive, and
|
||||
keeping the pair would install one unverified. Unpairing is neither."""
|
||||
kept = ip._apply_release_digests(
|
||||
[self._choice(runtime = "cudart.zip")], {"bundle.zip": SHA_A}
|
||||
)
|
||||
assert len(kept) == 1
|
||||
assert kept[0].expected_sha256 == SHA_A
|
||||
assert kept[0].runtime_name is None
|
||||
assert kept[0].runtime_url is None
|
||||
assert kept[0].runtime_sha256 is None
|
||||
|
||||
|
||||
class TestTheFetcherFailsClosed:
|
||||
def test_an_api_failure_is_an_empty_map_not_an_exception(self, monkeypatch):
|
||||
"""A rate limit or an outage has to cost the CUDA bundle, not the whole install,
|
||||
and it must not be mistaken for "no digest needed"."""
|
||||
|
||||
def _boom(repo, tag):
|
||||
raise RuntimeError("429 rate limited")
|
||||
|
||||
monkeypatch.setattr(ip, "github_release", _boom)
|
||||
assert ip.github_release_asset_digests("ggml-org/llama.cpp", "b10853") == {}
|
||||
|
||||
def test_it_reads_the_digests_off_the_release_it_fetched(self, monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
ip, "github_release", lambda repo, tag: _release(_asset("b.zip", f"sha256:{SHA_A}"))
|
||||
)
|
||||
assert ip.github_release_asset_digests("r", "t") == {"b.zip": SHA_A}
|
||||
|
||||
|
||||
def test_no_branch_returns_attempts_that_were_never_hash_gated():
|
||||
"""The property this file exists for, asserted against the source.
|
||||
|
||||
Every `return` of asset choices in the release-plan resolver goes through
|
||||
apply_approved_hashes or _apply_release_digests. A future branch that returns a bare
|
||||
list would reintroduce exactly the hole this closes.
|
||||
"""
|
||||
source = (REPO_ROOT / "studio" / "install_llama_prebuilt.py").read_text(encoding = "utf-8")
|
||||
start = source.index("def resolve_release_asset_choice(")
|
||||
end = source.index("\ndef ", start + 1)
|
||||
body = source[start:end]
|
||||
returns = [
|
||||
line.strip()
|
||||
for line in body.splitlines()
|
||||
if line.strip().startswith("return ") and "attempts" not in line
|
||||
]
|
||||
for line in returns:
|
||||
assert (
|
||||
"apply_approved_hashes" in line
|
||||
or "_apply_release_digests" in line
|
||||
or "verified" in line
|
||||
or "published_choice" in line
|
||||
), f"a return that is not hash gated: {line}"
|
||||
|
|
@ -836,6 +836,7 @@ def _run_flavor_invariant(
|
|||
index_family = None,
|
||||
index_url = None,
|
||||
win_arm64 = False,
|
||||
win_arm64_interpreter = None,
|
||||
probe_cuda = None,
|
||||
probe_hip = None,
|
||||
):
|
||||
|
|
@ -913,6 +914,13 @@ def _run_flavor_invariant(
|
|||
patch.object(stack_mod, "_RECORDED_TORCH_TAG", recorded),
|
||||
patch.object(stack_mod.platform, "machine", return_value = "AMD64"),
|
||||
patch.object(stack_mod, "_is_windows_arm64", return_value = win_arm64),
|
||||
# The interpreter's arch, which the CUDA-preservation shortcut reads.
|
||||
# They separate on an ARM64 machine running an emulated x64 python.
|
||||
patch.object(
|
||||
stack_mod,
|
||||
"_is_win_arm64_interpreter",
|
||||
return_value = win_arm64 if win_arm64_interpreter is None else win_arm64_interpreter,
|
||||
),
|
||||
patch.object(stack_mod, "_has_usable_nvidia_gpu", return_value = nvidia),
|
||||
patch.object(stack_mod.shutil, "which", side_effect = _which),
|
||||
patch.object(stack_mod.os.path, "isfile", return_value = True),
|
||||
|
|
@ -1397,6 +1405,84 @@ class TestAPinnedCpuIndexIsEnforcedToo:
|
|||
mock_pip.assert_not_called()
|
||||
|
||||
|
||||
class TestWindowsOnArmPreservesCudaOnlyForAnInferredExpectation:
|
||||
"""The win_arm64 CUDA shortcut must not swallow an explicit CPU pin.
|
||||
|
||||
Preserving an installed cu* build is right for an expectation DERIVED from the driver:
|
||||
download.pytorch.org publishes no win_arm64 CUDA wheel, so that "repair" resolves
|
||||
nothing. A pin is not a derivation, and /cpu does publish win_arm64 torch and
|
||||
torchvision, so that repair has somewhere to go.
|
||||
"""
|
||||
|
||||
_PIN = "https://download.pytorch.org/whl/cpu"
|
||||
|
||||
def test_an_explicit_cpu_pin_still_repairs_a_cuda_venv(self):
|
||||
ok, mock_pip = _run_flavor_invariant(
|
||||
installed = "2.11.0+cu134",
|
||||
repaired = "2.10.0+cpu",
|
||||
expected_env = "cpu",
|
||||
index_url = self._PIN,
|
||||
backend = "cpu",
|
||||
win_arm64 = True,
|
||||
)
|
||||
assert ok is True
|
||||
assert mock_pip.call_count == 1
|
||||
assert _index_url(mock_pip) == self._PIN
|
||||
args = [str(a) for a in mock_pip.call_args.args]
|
||||
# No win_arm64 torchaudio on /cpu either, so the existing drop still applies.
|
||||
assert not any(a.startswith("torchaudio") for a in args)
|
||||
|
||||
def test_the_family_spelling_of_the_pin_is_honoured_too(self):
|
||||
ok, mock_pip = _run_flavor_invariant(
|
||||
installed = "2.11.0+cu134",
|
||||
repaired = "2.10.0+cpu",
|
||||
expected_env = "cpu",
|
||||
index_family = "cpu",
|
||||
backend = "cpu",
|
||||
win_arm64 = True,
|
||||
)
|
||||
assert ok is True
|
||||
assert mock_pip.call_count == 1
|
||||
|
||||
def test_a_driver_inferred_cuda_expectation_is_still_preserved(self):
|
||||
ok, mock_pip = _run_flavor_invariant(
|
||||
installed = "2.11.0+cu134",
|
||||
expected_env = "cu130",
|
||||
win_arm64 = True,
|
||||
)
|
||||
assert ok is True
|
||||
mock_pip.assert_not_called()
|
||||
|
||||
def test_a_probed_cpu_tag_with_no_pin_does_not_downgrade_it(self):
|
||||
# setup.ps1 also publishes "cpu" when its nvidia-smi probe comes back empty, and a probe
|
||||
# result is not evidence, so the shortcut keeps the CUDA build.
|
||||
ok, mock_pip = _run_flavor_invariant(
|
||||
installed = "2.11.0+cu134",
|
||||
expected_env = "cpu",
|
||||
nvidia = False,
|
||||
win_arm64 = True,
|
||||
)
|
||||
assert ok is True
|
||||
mock_pip.assert_not_called()
|
||||
|
||||
def test_an_emulated_x64_interpreter_on_an_arm64_machine_still_repairs(self):
|
||||
"""The machine and the interpreter are separate axes, and the shortcut reads the
|
||||
interpreter. Every Windows on ARM install predating native support runs an
|
||||
emulated x64 python against ordinary win_amd64 wheels from
|
||||
download.pytorch.org, so the repair must reach them exactly as it always did.
|
||||
"""
|
||||
ok, mock_pip = _run_flavor_invariant(
|
||||
installed = "2.9.1+cu118",
|
||||
repaired = "2.10.0+cu128",
|
||||
expected_env = "cu128",
|
||||
win_arm64 = True,
|
||||
win_arm64_interpreter = False,
|
||||
)
|
||||
assert ok is True
|
||||
assert mock_pip.call_count == 1
|
||||
assert _index_url(mock_pip).endswith("/cu128")
|
||||
|
||||
|
||||
class TestWindowsOnArmKeepsTheNoTorchaudioException:
|
||||
"""No win_arm64 torchaudio wheel is published on any index.
|
||||
|
||||
|
|
@ -2192,9 +2278,13 @@ class TestTheDelegatedRocmRepairKeepsTheArm64Exception:
|
|||
def test_the_windows_rocm_install_drops_torchaudio_on_arm64(self):
|
||||
source = inspect.getsource(stack_mod._ensure_rocm_torch)
|
||||
block = source[source.index("_WINDOWS_ROCM_TORCH_PKG_SPECS.get") :][:1200]
|
||||
# The interpreter's arch, not the machine's: an emulated x64 venv installs win_amd64.
|
||||
assert (
|
||||
"_is_windows_arm64()" in block
|
||||
"_is_win_arm64_interpreter()" in block
|
||||
), "the delegated ROCm repair needs the same exception as the flavor repair"
|
||||
assert (
|
||||
"_is_windows_arm64()" not in block
|
||||
), "the machine predicate would drop torchaudio from x64 venvs"
|
||||
assert "*_rocm_trio" in block, "the trio has to be built, not passed positionally"
|
||||
|
||||
def test_x64_windows_still_asks_for_all_three(self):
|
||||
|
|
|
|||
|
|
@ -19,8 +19,11 @@ import ast
|
|||
import io
|
||||
import pathlib
|
||||
import re
|
||||
import subprocess
|
||||
import tokenize
|
||||
|
||||
import pytest
|
||||
|
||||
REPO_ROOT = pathlib.Path(__file__).resolve().parents[3]
|
||||
REQ_ROOT = REPO_ROOT / "studio" / "backend" / "requirements"
|
||||
PIN_FILE = REQ_ROOT / "diffusers-pin.txt"
|
||||
|
|
@ -181,3 +184,70 @@ def test_install_sh_still_delegates_the_core_package_skip():
|
|||
"""The handoff flag skips core packages while allowing other base entries through."""
|
||||
assert 'SKIP_STUDIO_BASE="$_SKIP_BASE"' in INSTALL_SH.read_text(encoding = "utf-8")
|
||||
assert "_SKIP_BASE=1" in INSTALL_SH.read_text(encoding = "utf-8")
|
||||
|
||||
|
||||
def test_no_generated_filter_snapshot_is_tracked():
|
||||
"""They are a copy of a file already in the tree, and one got committed.
|
||||
|
||||
_filter_requirements writes beside the source so relative -r/-c includes resolve.
|
||||
pip_install unlinks them in a finally, but a test calling the helper directly, or an
|
||||
install killed mid-run, leaves them in the checkout, where `git add -A` picks them up.
|
||||
A stale snapshot then reads as a second, silently divergent copy of the pins.
|
||||
"""
|
||||
done = subprocess.run(
|
||||
["git", "ls-files", "-z", "--", "studio/backend/requirements/"],
|
||||
cwd = REPO_ROOT,
|
||||
capture_output = True,
|
||||
text = True,
|
||||
)
|
||||
if done.returncode != 0:
|
||||
pytest.skip("not a git checkout")
|
||||
tracked = [p for p in done.stdout.split("\0") if p]
|
||||
offenders = [p for p in tracked if _GENERATED_FILTER.fullmatch(pathlib.Path(p).name)]
|
||||
assert not offenders, f"generated filter snapshots are tracked: {offenders}"
|
||||
|
||||
|
||||
def test_gitignore_covers_the_generated_snapshots():
|
||||
"""So the next `git add -A` cannot put one back."""
|
||||
probe = REQ_ROOT / ".studio-filtered-abcd1234.txt"
|
||||
assert _GENERATED_FILTER.fullmatch(probe.name), "the probe must match the generated shape"
|
||||
done = subprocess.run(
|
||||
["git", "check-ignore", "-q", "--no-index", str(probe)],
|
||||
cwd = REPO_ROOT,
|
||||
capture_output = True,
|
||||
text = True,
|
||||
)
|
||||
if done.returncode == 128:
|
||||
pytest.skip("not a git checkout")
|
||||
assert done.returncode == 0, f"{probe.name} is not ignored; .gitignore needs the pattern"
|
||||
|
||||
|
||||
# A win_arm64 floor set above the first release that actually publishes one costs the
|
||||
# resolver every wheel in between, and for scikit-learn it cost the only one that exists on
|
||||
# a free-threaded 3.13. Each floor below is the earliest release carrying a win_arm64 wheel,
|
||||
# read off PyPI's own file list, so the pin can be checked against the index by hand.
|
||||
WIN_ARM64_FLOORS = [
|
||||
# scikit-learn 1.9.0 dropped cp313-cp313t; 1.8.0 is the only release with one, so a
|
||||
# >=1.9.0 floor leaves a free-threaded 3.13 with no wheel and an sdist to compile.
|
||||
("extras.txt", "scikit-learn", "1.8.0"),
|
||||
# av publishes cp311-abi3 plus a cp314t from 17.0.0. No release has a 3.13t wheel.
|
||||
("extras.txt", "av", "17.0.0"),
|
||||
("single-env/constraints.txt", "av", "17.0.0"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"relpath, dist, floor",
|
||||
WIN_ARM64_FLOORS,
|
||||
ids = [f"{r.split('/')[-1]}:{d}" for r, d, _ in WIN_ARM64_FLOORS],
|
||||
)
|
||||
def test_the_win_arm64_floor_is_the_first_release_that_has_a_wheel(relpath, dist, floor):
|
||||
text = (REQ_ROOT / relpath).read_text(encoding = "utf-8")
|
||||
wanted = f'{dist}>={floor}; sys_platform == "win32" and platform_machine == "ARM64"'
|
||||
assert wanted in text, f"{relpath} no longer floors {dist} at {floor}"
|
||||
# And nothing else floors the same dist higher on that marker.
|
||||
for line in text.splitlines():
|
||||
line = line.strip()
|
||||
if not line.startswith(f"{dist}>=") or 'platform_machine == "ARM64"' not in line:
|
||||
continue
|
||||
assert line == wanted, f"{relpath}: a second ARM64 floor for {dist}: {line}"
|
||||
|
|
|
|||
|
|
@ -132,6 +132,28 @@ def test_extras_cannot_shadow_a_field_verify_install_reads(tmp_path: pathlib.Pat
|
|||
assert payload["package_version"] != "9.9.9"
|
||||
|
||||
|
||||
def test_extras_cannot_name_the_windows_on_arm_torch_index(tmp_path: pathlib.Path) -> None:
|
||||
"""setup.ps1 reads woa_torch_index back as an index to install torch FROM, and the
|
||||
parameter that writes it accepts NVIDIA's own https channel only. Evidence reaching the
|
||||
same key would hand a fresh-shell update any URL the caller liked."""
|
||||
im.write_manifest(
|
||||
root = tmp_path,
|
||||
req_root = tmp_path,
|
||||
package_name = "pytest",
|
||||
extra = {"woa_torch_index": "https://evil.test/simple"},
|
||||
)
|
||||
assert "woa_torch_index" not in _payload(tmp_path)
|
||||
|
||||
im.write_manifest(
|
||||
root = tmp_path,
|
||||
req_root = tmp_path,
|
||||
package_name = "pytest",
|
||||
woa_torch_index = "https://pypi.nvidia.com/nvtorch_oot",
|
||||
extra = {"woa_torch_index": "https://evil.test/simple"},
|
||||
)
|
||||
assert _payload(tmp_path)["woa_torch_index"] == "https://pypi.nvidia.com/nvtorch_oot"
|
||||
|
||||
|
||||
def test_update_manifest_merges_without_touching_the_rest(tmp_path: pathlib.Path) -> None:
|
||||
im.write_manifest(root = tmp_path, req_root = tmp_path, package_name = "pytest")
|
||||
before = _payload(tmp_path)
|
||||
|
|
|
|||
183
tests/studio/install/test_llama_kind_gate_matches_selector.py
Normal file
183
tests/studio/install/test_llama_kind_gate_matches_selector.py
Normal file
|
|
@ -0,0 +1,183 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""The llama.cpp mismatch gate and the llama.cpp selector have to agree, or updates churn.
|
||||
|
||||
Before reinstalling, setup.ps1 deletes the existing llama.cpp tree when its recorded
|
||||
`install_kind` is not in the set the gate considers correct for this host; then
|
||||
install_llama_prebuilt.py picks a bundle. Nothing makes those two agree, and a disagreement
|
||||
is permanent: every update deletes the tree and refetches the identical bundle, forever, on
|
||||
a working host. The only relationship that matters is "the kind the selector installs is one
|
||||
the gate accepts", so that is asserted directly, over the product of what either side
|
||||
branches on, rather than the text of either.
|
||||
|
||||
Both churning combinations this found are on Windows ARM64 and invisible from an x64 box:
|
||||
|
||||
* NVIDIA. The gate expected windows-cuda; on an ARM64 venv the selector installs
|
||||
windows-arm64. That is every Windows ARM64 machine with an NVIDIA GPU.
|
||||
* ROCm. No ROCm bundle exists for Windows ARM64 at all, so the selector falls through to
|
||||
the ARM64 CPU bundle while the gate still expected windows-rocm or windows-hip.
|
||||
|
||||
Offline. direct_upstream_release_plan takes a release dict, and the gate's own block is lifted
|
||||
out of setup.ps1 and run with only its inputs replaced.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import itertools
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
REPO_ROOT = pathlib.Path(__file__).resolve().parents[3]
|
||||
SETUP_PS1 = REPO_ROOT / "studio" / "setup.ps1"
|
||||
SETUP_SRC = SETUP_PS1.read_text(encoding = "utf-8")
|
||||
|
||||
sys.path.insert(0, str(REPO_ROOT / "studio"))
|
||||
|
||||
import install_llama_prebuilt as ip # noqa: E402
|
||||
|
||||
PWSH = shutil.which("pwsh") or shutil.which("powershell")
|
||||
requires_pwsh = pytest.mark.skipif(PWSH is None, reason = "PowerShell is unavailable")
|
||||
|
||||
TAG = "b9334"
|
||||
|
||||
# Everything upstream publishes for Windows, so no row is starved of a candidate and a
|
||||
# disagreement is a disagreement rather than a missing asset. Note what is NOT here and
|
||||
# cannot be: there is no ROCm or CUDA bundle for Windows ARM64 upstream.
|
||||
ASSET_NAMES = (
|
||||
f"llama-{TAG}-bin-win-cpu-arm64.zip",
|
||||
f"llama-{TAG}-bin-win-cpu-x64.zip",
|
||||
f"llama-{TAG}-bin-win-cuda-12.4-x64.zip",
|
||||
f"llama-{TAG}-bin-win-cuda-13.0-x64.zip",
|
||||
f"llama-{TAG}-bin-win-vulkan-x64.zip",
|
||||
f"llama-{TAG}-bin-win-hip-radeon-x64.zip",
|
||||
)
|
||||
RELEASE = {
|
||||
"tag_name": TAG,
|
||||
"assets": [
|
||||
{"name": name, "browser_download_url": f"https://example.invalid/{name}"}
|
||||
for name in ASSET_NAMES
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _host(*, arm64: bool, nvidia: bool, rocm: bool) -> ip.HostInfo:
|
||||
"""The interpreter's own view of the machine, which is what the selector reads: an
|
||||
emulated x64 Python on an ARM64 box reports AMD64, and is an x64 host here."""
|
||||
return ip.HostInfo(
|
||||
system = "Windows",
|
||||
machine = "ARM64" if arm64 else "AMD64",
|
||||
is_windows = True,
|
||||
is_linux = False,
|
||||
is_macos = False,
|
||||
is_x86_64 = not arm64,
|
||||
is_arm64 = arm64,
|
||||
nvidia_smi = "nvidia-smi" if nvidia else None,
|
||||
driver_cuda_version = (13, 0) if nvidia else None,
|
||||
compute_caps = ["12.1"] if nvidia else [],
|
||||
visible_cuda_devices = None,
|
||||
has_physical_nvidia = nvidia,
|
||||
has_usable_nvidia = nvidia,
|
||||
has_rocm = rocm,
|
||||
rocm_gfx_target = "gfx1201" if rocm else None,
|
||||
)
|
||||
|
||||
|
||||
def _gate_block() -> str:
|
||||
"""setup.ps1's own expected-kinds computation, from the opt-out read to the delete."""
|
||||
start = SETUP_SRC.index("$_arm64CudaOptOut = ")
|
||||
end = SETUP_SRC.index("if ($existingKind -and", start)
|
||||
return SETUP_SRC[start:end]
|
||||
|
||||
|
||||
def _expected_kinds(*, arm64_venv: bool, nvidia: bool, rocm: bool, opt_out: bool) -> list[str]:
|
||||
block = _gate_block()
|
||||
# The parentheses stay: the source writes `if (Test-WinArm64Venv)`, and replacing the
|
||||
# whole parenthesised form would leave `if $true {`, which does not parse.
|
||||
block = block.replace("Test-WinArm64Venv", "$true" if arm64_venv else "$false")
|
||||
# The evidence index is read through a chain of markers and a venv on disk. Its only use
|
||||
# is Test-WoaPersistableIndex, so supply the answer and stub the predicate.
|
||||
block = re.sub(
|
||||
r"(?s)\$_woaEvidenceIndex = if.*?\n(\s*)\$_nvidiaEvidence",
|
||||
"$_woaEvidenceIndex = '{}'\n\\1$_nvidiaEvidence".format(
|
||||
"https://pypi.nvidia.com/nvtorch_oot" if nvidia else ""
|
||||
),
|
||||
block,
|
||||
)
|
||||
script = "\n".join(
|
||||
[
|
||||
"$ErrorActionPreference = 'Stop'",
|
||||
"function Test-WoaPersistableIndex { param($i) return ($i -like '*nvidia*') }",
|
||||
f"$env:UNSLOTH_LLAMA_ARM64_CUDA = '{'0' if opt_out else '1'}'",
|
||||
f"$HasNvidiaSmi = ${str(nvidia).lower()}",
|
||||
f"$HasROCm = ${str(rocm).lower()}",
|
||||
"$script:ROCmGfxArch = " + ("'gfx1201'" if rocm else "$null"),
|
||||
block,
|
||||
"Write-Output ('<<<' + ($expectedKinds -join ',') + '>>>')",
|
||||
]
|
||||
)
|
||||
done = subprocess.run(
|
||||
[PWSH, "-NoProfile", "-NonInteractive", "-Command", script],
|
||||
capture_output = True,
|
||||
timeout = 120,
|
||||
)
|
||||
out = done.stdout.decode("utf-8", "replace")
|
||||
assert "<<<" in out, done.stderr.decode("utf-8", "replace")
|
||||
return out[out.index("<<<") + 3 : out.rindex(">>>")].split(",")
|
||||
|
||||
|
||||
def _installed_kinds(host: ip.HostInfo) -> list[str]:
|
||||
plan = ip.direct_upstream_release_plan(RELEASE, host, "ggml-org/llama.cpp", "latest")
|
||||
return [attempt.install_kind for attempt in plan.attempts] if plan else []
|
||||
|
||||
|
||||
# nvidia and rocm together is not a host we ship to: the ROCm arm wins outright on both sides.
|
||||
COMBINATIONS = [
|
||||
(arm64_venv, nvidia, rocm, opt_out)
|
||||
for arm64_venv, nvidia, rocm, opt_out in itertools.product([True, False], repeat = 4)
|
||||
if not (nvidia and rocm)
|
||||
]
|
||||
|
||||
|
||||
@requires_pwsh
|
||||
@pytest.mark.parametrize(
|
||||
("arm64_venv", "nvidia", "rocm", "opt_out"),
|
||||
COMBINATIONS,
|
||||
ids = [
|
||||
"{}-{}{}".format(
|
||||
"arm64" if a else "x64",
|
||||
"nvidia" if n else ("rocm" if r else "cpu"),
|
||||
"-optout" if o else "",
|
||||
)
|
||||
for a, n, r, o in COMBINATIONS
|
||||
],
|
||||
)
|
||||
def test_the_gate_accepts_what_the_selector_installs(arm64_venv, nvidia, rocm, opt_out):
|
||||
expected = _expected_kinds(arm64_venv = arm64_venv, nvidia = nvidia, rocm = rocm, opt_out = opt_out)
|
||||
installed = _installed_kinds(_host(arm64 = arm64_venv, nvidia = nvidia, rocm = rocm))
|
||||
assert installed, "no candidate at all, so this row proves nothing"
|
||||
assert installed[0] in expected, (
|
||||
f"the gate would delete what the selector just installed, on every update: "
|
||||
f"selector picks {installed[0]}, gate accepts {expected}"
|
||||
)
|
||||
|
||||
|
||||
@requires_pwsh
|
||||
def test_no_arm64_row_expects_a_kind_only_published_for_x64():
|
||||
"""The shape behind both bugs: on an ARM64 venv a gate that expects an x64-only kind and
|
||||
nothing else is a delete on every update whatever the selector does. windows-vulkan counts
|
||||
as x64-only, since upstream's bundle is vulkan-x64."""
|
||||
x64_only = {"windows-cuda", "windows-rocm", "windows-hip", "windows-cpu", "windows-vulkan"}
|
||||
for nvidia, rocm, opt_out in itertools.product([True, False], repeat = 3):
|
||||
if nvidia and rocm:
|
||||
continue
|
||||
expected = set(_expected_kinds(arm64_venv = True, nvidia = nvidia, rocm = rocm, opt_out = opt_out))
|
||||
assert expected - x64_only, (
|
||||
f"nvidia={nvidia} rocm={rocm} optout={opt_out}: the gate expects only x64 kinds "
|
||||
f"on an ARM64 venv: {sorted(expected)}"
|
||||
)
|
||||
|
|
@ -161,9 +161,13 @@ class TestUnsupportedNameLookup:
|
|||
# ── The Windows WMI path end to end ──────────────────────────────────────────
|
||||
|
||||
|
||||
def _wmi_detect(names):
|
||||
def _wmi_detect(names, arm64 = False):
|
||||
"""Drive _detect_windows_gfx_arch over `names` with no hipinfo and no amd-smi,
|
||||
which is the reporter's host: Adrenalin driver only. Returns (arch, stdout)."""
|
||||
which is the reporter's host: Adrenalin driver only. Returns (arch, stdout).
|
||||
|
||||
`arm64` is the host the advice is being written for. Pinned here rather than read
|
||||
from the box running the tests, because the machine predicate is right to answer
|
||||
True on an ARM64 dev box and the reporter's box is x64."""
|
||||
ps_result = MagicMock()
|
||||
ps_result.returncode = 0
|
||||
amd = [n for n in names if re.search(r"AMD|Radeon", n, re.IGNORECASE)]
|
||||
|
|
@ -190,7 +194,8 @@ def _wmi_detect(names):
|
|||
with patch("shutil.which", return_value = None):
|
||||
with patch("os.path.isfile", return_value = False):
|
||||
with patch("subprocess.run", side_effect = _run):
|
||||
result = stack_mod._detect_windows_gfx_arch()
|
||||
with patch.object(stack_mod, "_is_windows_arm64", return_value = arm64):
|
||||
result = stack_mod._detect_windows_gfx_arch()
|
||||
return result, buf.getvalue()
|
||||
|
||||
|
||||
|
|
@ -308,8 +313,7 @@ class TestPythonStackWindowsArm64:
|
|||
ARM64 throw applies to it: setup.ps1 rejects the variable there."""
|
||||
|
||||
def test_arm64_gets_a_source_build_note_instead_of_the_setter(self):
|
||||
with patch.object(stack_mod, "_is_windows_arm64", return_value = True):
|
||||
_arch, out = _wmi_detect(["AMD Radeon RX 5700 XT"])
|
||||
_arch, out = _wmi_detect(["AMD Radeon RX 5700 XT"], arm64 = True)
|
||||
assert "gfx1010" in out, "the card is still named"
|
||||
assert (
|
||||
"UNSLOTH_LLAMA_CPP_BACKEND" not in out
|
||||
|
|
@ -318,29 +322,43 @@ class TestPythonStackWindowsArm64:
|
|||
|
||||
def test_x64_still_gets_the_setter(self):
|
||||
"""Positive control: the guard must not silence the advice everywhere."""
|
||||
with patch.object(stack_mod, "_is_windows_arm64", return_value = False):
|
||||
_arch, out = _wmi_detect(["AMD Radeon RX 5700 XT"])
|
||||
_arch, out = _wmi_detect(["AMD Radeon RX 5700 XT"])
|
||||
assert "UNSLOTH_LLAMA_CPP_BACKEND" in out
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"env,expected",
|
||||
"registry,env,expected",
|
||||
[
|
||||
({"PROCESSOR_ARCHITECTURE": "ARM64"}, True),
|
||||
({"PROCESSOR_ARCHITECTURE": "AMD64", "PROCESSOR_ARCHITEW6432": "ARM64"}, True),
|
||||
({"PROCESSOR_ARCHITECTURE": "AMD64", "PROCESSOR_ARCHITEW6432": ""}, False),
|
||||
("ARM64", {"PROCESSOR_ARCHITECTURE": "ARM64"}, True),
|
||||
# x64 emulation on an ARM64 box: the process copy says AMD64 and ARCHITEW6432 is
|
||||
# unset, so only the machine-scope registry value tells the truth.
|
||||
("ARM64", {"PROCESSOR_ARCHITECTURE": "AMD64"}, True),
|
||||
# ARCHITEW6432 still counts on the builds that do set it.
|
||||
("", {"PROCESSOR_ARCHITECTURE": "AMD64", "PROCESSOR_ARCHITEW6432": "ARM64"}, True),
|
||||
("AMD64", {"PROCESSOR_ARCHITECTURE": "AMD64", "PROCESSOR_ARCHITEW6432": ""}, False),
|
||||
# Unreadable registry falls back to the per-process signals, not to True.
|
||||
("", {"PROCESSOR_ARCHITECTURE": "AMD64"}, False),
|
||||
],
|
||||
ids = [
|
||||
"native-arm64",
|
||||
"emulated-x64-on-arm64",
|
||||
"architew6432-set",
|
||||
"real-x64",
|
||||
"no-registry-x64",
|
||||
],
|
||||
ids = ["native-arm64", "emulated-x64-on-arm64", "real-x64"],
|
||||
)
|
||||
def test_the_arch_probe_reads_the_machine_not_the_process(self, env, expected):
|
||||
def test_the_arch_probe_reads_the_machine_not_the_process(self, registry, env, expected):
|
||||
"""PROCESSOR_ARCHITECTURE describes the PROCESS, so an emulated x64 Python on an
|
||||
ARM64 box reports AMD64; ARCHITEW6432 is ARM64 in exactly that case."""
|
||||
ARM64 box reports AMD64 and so does platform.machine(). The machine-scope value in
|
||||
the registry is the one signal emulation cannot misreport. Patched per case so the
|
||||
answer is the same on an ARM64 host as on x64 CI."""
|
||||
with patch.object(stack_mod, "IS_WINDOWS", True):
|
||||
with patch.object(stack_mod.platform, "machine", return_value = "AMD64"):
|
||||
with patch.dict(os.environ, env, clear = False):
|
||||
for _k in ("PROCESSOR_ARCHITEW6432", "PROCESSOR_ARCHITECTURE"):
|
||||
if _k not in env:
|
||||
os.environ.pop(_k, None)
|
||||
assert stack_mod._is_windows_arm64() is expected
|
||||
with patch.object(stack_mod, "_machine_arch_from_registry", return_value = registry):
|
||||
with patch.object(stack_mod.platform, "machine", return_value = "AMD64"):
|
||||
with patch.dict(os.environ, env, clear = False):
|
||||
for _k in ("PROCESSOR_ARCHITEW6432", "PROCESSOR_ARCHITECTURE"):
|
||||
if _k not in env:
|
||||
os.environ.pop(_k, None)
|
||||
assert stack_mod._is_windows_arm64() is expected
|
||||
|
||||
def test_it_is_false_off_windows(self):
|
||||
with patch.object(stack_mod, "IS_WINDOWS", False):
|
||||
|
|
|
|||
54
tests/studio/install/test_seed_plugin_pandas_range.py
Normal file
54
tests/studio/install/test_seed_plugin_pandas_range.py
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""The bundled seed plugin is installed with --no-deps, so its own pandas range was never
|
||||
resolved. On Windows on ARM (Python 3.11+) constraints.txt installs pandas 3, which the
|
||||
plugin declared out of range: a dependency-aware reinstall of the plugin would then look for
|
||||
a pandas 2 that has no win_arm64 wheel. The plugin's range must admit every pandas the
|
||||
constraints can install."""
|
||||
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
from packaging.specifiers import SpecifierSet
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parents[3]
|
||||
PLUGIN = (
|
||||
ROOT / "studio" / "backend" / "plugins" / "data-designer-unstructured-seed" / "pyproject.toml"
|
||||
)
|
||||
CONSTRAINTS = ROOT / "studio" / "backend" / "requirements" / "single-env" / "constraints.txt"
|
||||
|
||||
|
||||
def _plugin_pandas_range():
|
||||
text = PLUGIN.read_text(encoding = "utf-8")
|
||||
match = re.search(r'"pandas([^"]*)"', text)
|
||||
assert match, "the plugin declares pandas"
|
||||
return SpecifierSet(match.group(1))
|
||||
|
||||
|
||||
def _constraint_pandas_pins():
|
||||
pins = []
|
||||
for line in CONSTRAINTS.read_text(encoding = "utf-8").splitlines():
|
||||
if line.startswith("pandas"):
|
||||
spec = line.split(";", 1)[0][len("pandas") :].strip()
|
||||
pins.append(SpecifierSet(spec))
|
||||
assert pins, "constraints.txt pins pandas"
|
||||
return pins
|
||||
|
||||
|
||||
def test_every_constrained_pandas_is_inside_the_plugins_range():
|
||||
plugin = _plugin_pandas_range()
|
||||
for pin in _constraint_pandas_pins():
|
||||
# The lowest version each constraint admits is the one that gets installed.
|
||||
floor = next(s.version for s in pin if s.operator in ("==", ">="))
|
||||
assert plugin.contains(
|
||||
floor
|
||||
), f"constraints install pandas {floor}, outside the plugin's {plugin}"
|
||||
|
||||
|
||||
def test_the_arm64_row_installs_pandas_3():
|
||||
"""The row this test exists for: without it the range check would pass on pandas 2 alone."""
|
||||
rows = [
|
||||
l for l in CONSTRAINTS.read_text(encoding = "utf-8").splitlines() if l.startswith("pandas>=3")
|
||||
]
|
||||
assert len(rows) == 1 and 'platform_machine == "ARM64"' in rows[0]
|
||||
|
|
@ -3019,6 +3019,137 @@ class TestPublishedWindowsCudaAttemptsDynamicMajor:
|
|||
assert result[0].runtime_line == "cuda12"
|
||||
|
||||
|
||||
class TestPublishedLegacyNamedArm64BundlesAreOrdered:
|
||||
"""The legacy-minor ordering synthesised -x64.zip names and called windows_cuda_attempts
|
||||
without the arch, so an approved llama-<tag>-bin-win-cuda-13.1-arm64.zip produced no
|
||||
runtime line and was skipped for the unchecksummed upstream fallback."""
|
||||
|
||||
TAG = "b8508"
|
||||
|
||||
def _release(
|
||||
self,
|
||||
minors_lines,
|
||||
ranks = None,
|
||||
cudart = False,
|
||||
):
|
||||
artifacts = [
|
||||
make_artifact(
|
||||
f"llama-{self.TAG}-bin-win-cuda-{minor}-arm64.zip",
|
||||
install_kind = "windows-arm64-cuda",
|
||||
runtime_line = line,
|
||||
supported_sms = ["75", "80", "86", "89", "90", "100", "120"],
|
||||
max_sm = 120,
|
||||
rank = (ranks or {}).get(minor, 100),
|
||||
)
|
||||
for minor, line in minors_lines
|
||||
]
|
||||
release = make_release(artifacts, upstream_tag = self.TAG)
|
||||
if cudart:
|
||||
for minor, _ in minors_lines:
|
||||
name = f"cudart-llama-bin-win-cuda-{minor}-arm64.zip"
|
||||
release.assets[name] = f"https://example.com/{name}"
|
||||
return release
|
||||
|
||||
def _host(self, driver):
|
||||
return make_host(
|
||||
system = "Windows", machine = "ARM64", driver_cuda_version = driver, compute_caps = ["120"]
|
||||
)
|
||||
|
||||
def test_the_arm64_legacy_bundle_is_selected(self, monkeypatch):
|
||||
mock_windows_runtime(monkeypatch, ["cuda13", "cuda12"])
|
||||
release = self._release([("13.1", "cuda13"), ("12.8", "cuda12")])
|
||||
result = published_windows_cuda_attempts(self._host((13, 1)), release, None, arch = "arm64")
|
||||
assert result, "no attempt at all: the arch was lost in the legacy ordering"
|
||||
assert result[0].runtime_line == "cuda13"
|
||||
assert result[0].name == f"llama-{self.TAG}-bin-win-cuda-13.1-arm64.zip"
|
||||
|
||||
def test_the_minor_gate_applies_to_arm64_too(self, monkeypatch):
|
||||
mock_windows_runtime(monkeypatch, ["cuda13", "cuda12"])
|
||||
release = self._release([("13.3", "cuda13"), ("12.8", "cuda12")])
|
||||
result = published_windows_cuda_attempts(self._host((13, 1)), release, None, arch = "arm64")
|
||||
assert result and result[0].runtime_line == "cuda12"
|
||||
assert all(not a.name.endswith("13.3-arm64.zip") for a in result)
|
||||
|
||||
def test_the_artifact_gate_reads_the_arm64_name(self, monkeypatch):
|
||||
"""Two legacy minors on one line, the higher one ranked first: the line survives on
|
||||
the lower minor, and the per-artifact gate must parse the -arm64 name to drop the
|
||||
higher one; an x64 spelling never matches, and the preferred 13.3 build slips through."""
|
||||
mock_windows_runtime(monkeypatch, ["cuda13"])
|
||||
release = self._release([("13.3", "cuda13"), ("13.1", "cuda13")], ranks = {"13.3": 10})
|
||||
result = published_windows_cuda_attempts(self._host((13, 1)), release, None, arch = "arm64")
|
||||
names = [a.name for a in result]
|
||||
assert names == [f"llama-{self.TAG}-bin-win-cuda-13.1-arm64.zip"], names
|
||||
|
||||
def test_the_cudart_bundle_pairs_by_the_arm64_name(self, monkeypatch):
|
||||
"""The same match feeds the cudart pairing: the arm64 runtime archive rides along."""
|
||||
mock_windows_runtime(monkeypatch, ["cuda13"])
|
||||
release = self._release([("13.1", "cuda13")], cudart = True)
|
||||
result = published_windows_cuda_attempts(self._host((13, 1)), release, None, arch = "arm64")
|
||||
assert result and result[0].runtime_name == f"cudart-llama-bin-win-cuda-13.1-arm64.zip"
|
||||
|
||||
def test_the_explicit_arch_wins_over_the_host(self, monkeypatch):
|
||||
"""The arch argument is the contract: it must reach the legacy ordering rather than
|
||||
being re-derived from the host, or an x64 host resolving arm64 bundles finds none."""
|
||||
mock_windows_runtime(monkeypatch, ["cuda13", "cuda12"])
|
||||
release = self._release([("13.1", "cuda13"), ("12.8", "cuda12")])
|
||||
host = make_host(
|
||||
system = "Windows", machine = "AMD64", driver_cuda_version = (13, 1), compute_caps = ["120"]
|
||||
)
|
||||
result = published_windows_cuda_attempts(host, release, None, arch = "arm64")
|
||||
assert result and result[0].name == f"llama-{self.TAG}-bin-win-cuda-13.1-arm64.zip"
|
||||
|
||||
|
||||
class TestAnUpstreamLookupFailureCostsOnlyCuda:
|
||||
"""On a Windows ARM64 NVIDIA host with no approved CUDA bundle, the upstream asset list is
|
||||
fetched from the release API. A rate limit or an outage there raised out of the planner,
|
||||
which catches only PrebuiltFallback, so the whole install aborted although the published
|
||||
ARM64 CPU bundle was there to fall through to. It now costs the CUDA bundle only, like the
|
||||
digest fetch beside it."""
|
||||
|
||||
TAG = "b8508"
|
||||
CPU = "app-b8508-windows-arm64-cpu.zip"
|
||||
|
||||
def _plan(self, monkeypatch, assets):
|
||||
mock_windows_runtime(monkeypatch, ["cuda13", "cuda12"])
|
||||
monkeypatch.setattr(
|
||||
INSTALL_LLAMA_PREBUILT,
|
||||
"detect_torch_cuda_runtime_preference",
|
||||
lambda host: CudaRuntimePreference(runtime_line = None, selection_log = []),
|
||||
)
|
||||
monkeypatch.setattr(INSTALL_LLAMA_PREBUILT, "github_release_assets", assets)
|
||||
monkeypatch.setattr(
|
||||
INSTALL_LLAMA_PREBUILT, "github_release_asset_digests", lambda repo, tag: {}
|
||||
)
|
||||
monkeypatch.delenv("UNSLOTH_LLAMA_ARM64_CUDA", raising = False)
|
||||
host = make_host(
|
||||
system = "Windows", machine = "ARM64", driver_cuda_version = (13, 4), compute_caps = ["121"]
|
||||
)
|
||||
release = make_release(
|
||||
[
|
||||
make_artifact(
|
||||
self.CPU,
|
||||
install_kind = "windows-arm64",
|
||||
runtime_line = None,
|
||||
bundle_profile = "windows-cpu-arm64",
|
||||
)
|
||||
],
|
||||
upstream_tag = self.TAG,
|
||||
)
|
||||
return resolve_release_asset_choice(host, self.TAG, release, make_checksums([self.CPU]))
|
||||
|
||||
def test_an_api_failure_falls_through_to_the_cpu_bundle(self, monkeypatch):
|
||||
def _boom(repo, tag):
|
||||
raise RuntimeError("429 rate limited")
|
||||
|
||||
result = self._plan(monkeypatch, _boom)
|
||||
assert [a.name for a in result] == [self.CPU]
|
||||
assert result[0].install_kind == "windows-arm64"
|
||||
|
||||
def test_an_empty_listing_lands_in_the_same_place(self, monkeypatch):
|
||||
result = self._plan(monkeypatch, lambda repo, tag: {})
|
||||
assert [a.name for a in result] == [self.CPU]
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# N.1e. resolve_release_asset_choice -- pin on the published install path
|
||||
# ===========================================================================
|
||||
|
|
@ -4288,8 +4419,7 @@ class TestCudaDriverToolkitMismatchMessage:
|
|||
"exceeds driver CUDA major 12 (12.9)."
|
||||
) in output
|
||||
assert (
|
||||
"Update the NVIDIA GPU driver to run CUDA Toolkit 13.3, or install "
|
||||
"a CUDA 12.x toolkit."
|
||||
"Update the NVIDIA GPU driver to run CUDA Toolkit 13.3, or install a CUDA 12.x toolkit."
|
||||
) in output
|
||||
assert "prebuilt CUDA bundle" in output
|
||||
|
||||
|
|
|
|||
|
|
@ -661,3 +661,41 @@ def test_the_rocm_trio_is_reinstalled_when_the_architecture_index_moves():
|
|||
# The record follows the install, never precedes it: a failed trio must not be recorded.
|
||||
failed = text.index("AMD ROCm PyTorch install failed -- falling back to CPU")
|
||||
assert failed < record
|
||||
|
||||
|
||||
class TestSetupPs1WindowsOnArmCudaPreservation:
|
||||
"""The win_arm64 CUDA shortcut is for an INFERRED expectation, not a stated one.
|
||||
|
||||
It runs ahead of the pin branch that raises $script:PinChangedForceReinstall, and that
|
||||
flag is the only thing that clears $SkipPythonDeps. So without the exemption an
|
||||
explicit pin skipped the dependency pass, install_python_stack.py and every
|
||||
--force-reinstall at once, and `studio update` kept the old CUDA build while reporting
|
||||
success. Exempting only /cpu was not enough: a user moving the venv to their own
|
||||
cu129 mirror is stating an instruction just as much, and the index selection further
|
||||
down is written to let a pin outrank the persisted NVIDIA channel.
|
||||
"""
|
||||
|
||||
_GUARD = "if ((Test-WinArm64Venv) -and $installedTorchTag -and"
|
||||
|
||||
def _condition(self) -> str:
|
||||
start = _SETUP_SRC.index(self._GUARD)
|
||||
return _SETUP_SRC[start : _SETUP_SRC.index("{", start + len(self._GUARD))]
|
||||
|
||||
def test_any_explicit_pin_is_exempt(self):
|
||||
condition = self._condition()
|
||||
assert "-not $_pinnedIdx" in condition
|
||||
assert (
|
||||
"$_woaCpuPinned" not in condition
|
||||
), "a cu129 mirror pin is as much an instruction as a /cpu one"
|
||||
|
||||
def test_the_exemption_reads_a_variable_that_is_always_assigned(self):
|
||||
# Not $_pinLeaf: it is assigned only inside `if ($_pinnedIdx)`, so reading it here would
|
||||
# be fatal under Set-StrictMode. $_pinnedIdx is assigned unconditionally above.
|
||||
block = _SETUP_SRC[: _SETUP_SRC.index(self._GUARD)]
|
||||
assert "$_pinnedIdx = Get-PinnedTorchIndexUrl" in block
|
||||
assert "$_pinLeaf" not in self._condition()
|
||||
|
||||
def test_it_still_sits_ahead_of_the_pin_branch(self):
|
||||
shortcut = _line_of(_SETUP_SRC, self._GUARD)
|
||||
pin_branch = _line_of(_SETUP_SRC, "Torch-index pin changed ($installedTorchTag)")
|
||||
assert shortcut < pin_branch
|
||||
|
|
|
|||
6269
tests/studio/install/test_woa_torch_index_persistence.py
Normal file
6269
tests/studio/install/test_woa_torch_index_persistence.py
Normal file
File diff suppressed because it is too large
Load diff
260
tests/studio/install/test_woa_wheelhouse_guard_behaviour.py
Normal file
260
tests/studio/install/test_woa_wheelhouse_guard_behaviour.py
Normal file
|
|
@ -0,0 +1,260 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""The PyPI-first wheelhouse guard, executed rather than read.
|
||||
|
||||
test_woa_wheelhouse_prefers_pypi.py pins the SHAPE of the guard against install.ps1's
|
||||
source, which cannot catch a helper that is wired correctly and answers wrongly. The three
|
||||
bugs covered here, all found reviewing the guard's first version:
|
||||
|
||||
1. A wheel the guard skipped stopped counting as available. $WoaWheelNames is rebuilt by
|
||||
scanning the staging directory, and a $WoaDropCandidates name missing from it is emitted
|
||||
as `name ; platform_machine == "AMD64"`, which EXCLUDES the package on ARM64. So the day
|
||||
PyPI published a win_arm64 hf_transfer or brotli, the guard turned "installed from our
|
||||
wheelhouse" into "not installed at all", firing on the event it exists for.
|
||||
|
||||
2. abi3 was treated as universally compatible. It is forward compatible from the version it
|
||||
was built against, so a cp314-abi3 wheel does not import on cp313, and calling ours
|
||||
redundant against one would leave the package uninstallable.
|
||||
|
||||
3. PyPI publishing a wheel is only availability if the resolve will look at PyPI. Offline,
|
||||
or pointed at an exclusive mirror, dropping our copy leaves it obtainable from nowhere.
|
||||
|
||||
Hermetic: no network. The live-PyPI leg of the guard is exercised on hardware, not here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import textwrap
|
||||
|
||||
import pytest
|
||||
|
||||
REPO_ROOT = pathlib.Path(__file__).resolve().parents[3]
|
||||
INSTALL_PS1 = REPO_ROOT / "install.ps1"
|
||||
|
||||
#: A corporate index that is not PyPI, and PyPI itself: the only distinction these make.
|
||||
CORP = "https://corp.example/simple"
|
||||
PYPI = "https://pypi.org/simple"
|
||||
|
||||
requires_pwsh = pytest.mark.skipif(shutil.which("pwsh") is None, reason = "PowerShell is unavailable")
|
||||
|
||||
HELPERS = (
|
||||
"Test-WoaVersionAtLeast",
|
||||
"Test-WoaWheelTags",
|
||||
"Test-WoaWheelTagsUsable",
|
||||
# Everything Test-WoaResolveReachesPyPI calls: a missing helper answers True and lets a
|
||||
# lookalike index pass unnoticed.
|
||||
"Test-WoaUrlIsPublicPyPI",
|
||||
"Remove-WoaTomlComment",
|
||||
"Split-WoaTomlKey",
|
||||
"Read-WoaUvInlineIndexArray",
|
||||
"Read-WoaUvTomlIndexKeys",
|
||||
"Get-WoaUvConfigIndexPolicy",
|
||||
"Test-WoaResolveReachesPyPI",
|
||||
)
|
||||
|
||||
|
||||
INSTALL_SRC = INSTALL_PS1.read_text(encoding = "utf-8-sig")
|
||||
|
||||
|
||||
def _function(source: str, name: str) -> str:
|
||||
match = re.search(r"(?ms)^ function " + re.escape(name) + r"\b.*?^ \}\r?$", source)
|
||||
assert match, f"install.ps1 no longer defines {name}"
|
||||
# Lifted out of Install-UnslothStudio, so the four-space body indent comes with it.
|
||||
return textwrap.dedent(match.group(0))
|
||||
|
||||
|
||||
def _drop_list_block(source: str) -> str:
|
||||
"""The real block that builds $WoaWheelNames and emits the override drop lines."""
|
||||
start = re.search(r"(?m)^ \$WoaWheelNames = @\{\}\s*$", source)
|
||||
end = re.search(r"(?m)^ \$WoaOverrideLines \+= 'torch>=2\.4'\s*$", source)
|
||||
assert start and end, "the drop-list block moved"
|
||||
return textwrap.dedent(source[start.start() : end.start()].replace("\r\n", "\n"))
|
||||
|
||||
|
||||
# PowerShell does not hoist, so every snippet below carries all of HELPERS in front of it.
|
||||
PRELUDE = "\n".join(_function(INSTALL_SRC, name) for name in HELPERS)
|
||||
|
||||
|
||||
def _run(script: str) -> str:
|
||||
proc = subprocess.run(
|
||||
["pwsh", "-NoProfile", "-NonInteractive", "-Command", PRELUDE + "\n" + script],
|
||||
capture_output = True,
|
||||
text = True,
|
||||
encoding = "utf-8",
|
||||
errors = "replace",
|
||||
timeout = 120,
|
||||
)
|
||||
assert proc.returncode == 0, f"pwsh failed: {proc.stdout}\n{proc.stderr}"
|
||||
return proc.stdout.strip()
|
||||
|
||||
|
||||
def _dropped_names(provided: str) -> set[str]:
|
||||
"""Which $WoaDropCandidates get excluded on ARM64, given what PyPI supplies.
|
||||
|
||||
The staging directory is empty on purpose: the only thing standing between a name and
|
||||
the AMD64 drop line is the $WoaPyPIProvided bookkeeping this exercises.
|
||||
"""
|
||||
out = _run(f"""
|
||||
$dir = Join-Path ([System.IO.Path]::GetTempPath()) ([guid]::NewGuid().ToString("N"))
|
||||
New-Item -ItemType Directory -Path $dir -Force | Out-Null
|
||||
$WoaWheelDir = $dir
|
||||
$WoaVenvMinor = "3.13"
|
||||
$WoaWheelTag = "cp313"; $WoaWheelAbi = "cp313"; $WoaWheelStable = $true; $WoaWheelMinor = 13
|
||||
$script:WoaTorchAudio = $false
|
||||
$script:WoaPyarrowWheelName = $null
|
||||
$script:WoaPyPIProvided = {provided}
|
||||
function substep {{ param($a, $b) }}
|
||||
{_drop_list_block(INSTALL_SRC)}
|
||||
Remove-Item -LiteralPath $dir -Recurse -Force
|
||||
($WoaOverrideLines | Where-Object {{ $_ -like "*platform_machine*" }} |
|
||||
ForEach-Object {{ ($_ -split ' ')[0] }} | Sort-Object -Unique) -join ","
|
||||
""")
|
||||
return {name for name in out.split(",") if name}
|
||||
|
||||
|
||||
@requires_pwsh
|
||||
@pytest.mark.parametrize(
|
||||
("wheel", "py_tag", "abi_tag", "usable"),
|
||||
[
|
||||
# The two interpreter-agnostic shapes the wheelhouse actually ships.
|
||||
("hf_transfer-0.1.9-cp38-abi3-win_arm64.whl", "cp313", "cp313", True),
|
||||
("sqlite_vec-0.1.9-py3-none-win_arm64.whl", "cp313", "cp313", True),
|
||||
# Bug 2: abi3 does not reach BACKWARDS to an older interpreter.
|
||||
("hf_transfer-0.2.0-cp314-abi3-win_arm64.whl", "cp313", "cp313", False),
|
||||
("foo-1.0-py314-none-win_arm64.whl", "cp313", "cp313", False),
|
||||
# Free-threaded CPython has no stable ABI (CPython #111506).
|
||||
("hf_transfer-0.1.9-cp38-abi3-win_arm64.whl", "cp313", "cp313t", False),
|
||||
# An exact match needs no special case.
|
||||
("regex-2026.9.3-cp313-cp313-win_arm64.whl", "cp313", "cp313", True),
|
||||
# A wheel built for another interpreter is not this venv's business either way.
|
||||
("regex-2026.9.3-cp312-cp312-win_arm64.whl", "cp313", "cp313", False),
|
||||
],
|
||||
)
|
||||
def test_a_wheel_is_usable_only_where_it_actually_imports(wheel, py_tag, abi_tag, usable):
|
||||
out = _run(f'Test-WoaWheelTagsUsable -Name "{wheel}" -PyTag "{py_tag}" -AbiTag "{abi_tag}"')
|
||||
assert out == str(usable), f"{wheel} on {py_tag}/{abi_tag}: expected {usable}, got {out}"
|
||||
|
||||
|
||||
@requires_pwsh
|
||||
@pytest.mark.parametrize(
|
||||
("env", "reaches"),
|
||||
[
|
||||
({}, True),
|
||||
# Bug 3: these REPLACE the default index, so PyPI is not consulted at all.
|
||||
({"UV_DEFAULT_INDEX": CORP}, False),
|
||||
({"UV_INDEX_URL": CORP}, False),
|
||||
# install.ps1 resolves with uv, which never reads pip's variables.
|
||||
({"PIP_INDEX_URL": CORP}, True),
|
||||
({"PIP_NO_INDEX": "1"}, True),
|
||||
({"UV_INDEX_URL": CORP, "PIP_EXTRA_INDEX_URL": PYPI}, False),
|
||||
# Pointed at PyPI explicitly is still PyPI.
|
||||
({"UV_DEFAULT_INDEX": PYPI}, True),
|
||||
({"UV_OFFLINE": "1"}, False),
|
||||
({"UV_NO_INDEX": "1"}, False),
|
||||
# An unset-looking value must not read as "offline".
|
||||
({"UV_OFFLINE": "0"}, True),
|
||||
({"UV_NO_INDEX": "false"}, True),
|
||||
],
|
||||
)
|
||||
def test_pypi_counts_only_when_the_resolve_would_reach_it(env, reaches):
|
||||
sets = "".join(f'$env:{key} = "{value}"; ' for key, value in env.items())
|
||||
assert _run(sets + "Test-WoaResolveReachesPyPI") == str(reaches)
|
||||
|
||||
|
||||
@requires_pwsh
|
||||
def test_a_wheel_taken_from_pypi_is_not_then_excluded_on_arm64():
|
||||
"""Bug 1, the one that matters: skipping our copy must not read as "no wheel exists"."""
|
||||
nothing_anywhere = _dropped_names("@{}")
|
||||
assert (
|
||||
{"brotli", "hf-transfer", "hf_transfer"} <= nothing_anywhere
|
||||
), f"a name with no wheel anywhere must be dropped on ARM64, got {nothing_anywhere}"
|
||||
|
||||
from_pypi = _dropped_names('@{ "brotli" = @("1.3.0"); "hf-transfer" = @("0.2.0") }')
|
||||
assert not (
|
||||
{"brotli", "hf-transfer", "hf_transfer"} & from_pypi
|
||||
), f"PyPI supplies these, so they must not be excluded on ARM64; dropped {from_pypi}"
|
||||
# What is genuinely unavailable is still dropped, so the fix did not blanket-allow.
|
||||
assert {"xformers", "torchcodec", "brotlicffi"} <= from_pypi
|
||||
|
||||
|
||||
@requires_pwsh
|
||||
@pytest.mark.parametrize(
|
||||
("version", "still_dropped"),
|
||||
[("0.0.20", True), ("0.0.30", False)],
|
||||
)
|
||||
def test_a_pypi_version_below_a_floor_keeps_the_drop(version, still_dropped):
|
||||
"""xformers carries a floor because the released metadata does, so a PyPI version below
|
||||
it is not a usable answer and the drop has to stay."""
|
||||
dropped = _dropped_names(f'@{{ "xformers" = @("{version}") }}')
|
||||
assert ("xformers" in dropped) is still_dropped
|
||||
|
||||
|
||||
# A uv config can take PyPI out of the resolve as decisively as UV_OFFLINE can. Every case below
|
||||
# is one install.ps1's subset parser got WRONG until it learned to scan for quotes, each answering
|
||||
# "PyPI is reachable" for a config that had disabled it, which deletes wheelhouse copies of wheels
|
||||
# the resolve can never fetch. Expectations derived with tomllib, not written by hand.
|
||||
UV_CONFIG_CASES = [
|
||||
# A comment needs no whitespace in front of it. `(^|\s)#` missed this one entirely.
|
||||
("comment_nospace", "uv.toml", "no-index = true# offline lab\n", False),
|
||||
("value_hash_nospace", "uv.toml", f'index-url = "{CORP}"#corp\n', False),
|
||||
# A quoted key is the same key as the bare spelling.
|
||||
("quoted_key", "uv.toml", f'"index-url" = "{CORP}"\n', False),
|
||||
# A dotted key is the same statement as writing the leaf under [pip].
|
||||
("dotted_no_index", "uv.toml", "pip.no-index = true\n", False),
|
||||
("dotted_index_url", "uv.toml", f'pip.index-url = "{CORP}"\n', False),
|
||||
("pyproject_dotted", "pyproject.toml", "[tool.uv]\npip.no-index = true\n", False),
|
||||
# The other side of the same scan: a `#` INSIDE a string is not a comment.
|
||||
("fragment_kept", "uv.toml", f'index-url = "{CORP}#frag"\n', False),
|
||||
("url_only_in_comment", "uv.toml", f'# index-url = "{CORP}"\n', True),
|
||||
# Cases that must keep answering True, so the fix cannot be a blanket "not reachable".
|
||||
("extra_index_only", "uv.toml", f'extra-index-url = "{CORP}"\n', True),
|
||||
("index_default_false", "uv.toml", f'[[index]]\nurl = "{CORP}"\ndefault = false\n', True),
|
||||
("explicit_pypi", "uv.toml", f'index-url = "{PYPI}"\n', True),
|
||||
("no_index_false", "uv.toml", f'no-index = false\nindex-url = "{PYPI}"\n', True),
|
||||
# [pip] outranks the top level, in both directions.
|
||||
(
|
||||
"pip_pypi_beats_top_corp",
|
||||
"uv.toml",
|
||||
f'index-url = "{CORP}"\n[pip]\nindex-url = "{PYPI}"\n',
|
||||
True,
|
||||
),
|
||||
(
|
||||
"pip_corp_beats_top_pypi",
|
||||
"uv.toml",
|
||||
f'index-url = "{PYPI}"\n[pip]\nindex-url = "{CORP}"\n',
|
||||
False,
|
||||
),
|
||||
# A host that merely contains the name is not PyPI.
|
||||
("lookalike_host", "uv.toml", 'index-url = "https://pypi.org.evil.com/simple"\n', False),
|
||||
# An inline `index = [...]` is not modelled, and unreadable resolves to "not PyPI".
|
||||
("inline_index_array", "uv.toml", f'index = [{{ url = "{CORP}", default = true }}]\n', False),
|
||||
]
|
||||
|
||||
|
||||
@requires_pwsh
|
||||
@pytest.mark.parametrize(
|
||||
("name", "filename", "text", "reaches"),
|
||||
UV_CONFIG_CASES,
|
||||
ids = [case[0] for case in UV_CONFIG_CASES],
|
||||
)
|
||||
def test_a_uv_config_decides_whether_pypi_is_in_the_resolve(
|
||||
tmp_path, name, filename, text, reaches
|
||||
):
|
||||
work = tmp_path / name
|
||||
work.mkdir()
|
||||
(work / filename).write_text(text, encoding = "utf-8")
|
||||
# APPDATA/ProgramData would let the developer's own uv.toml decide the answer.
|
||||
out = _run(f"""
|
||||
Remove-Item Env:UV_OFFLINE,Env:PIP_NO_INDEX,Env:UV_DEFAULT_INDEX,Env:UV_INDEX_URL,\
|
||||
Env:PIP_INDEX_URL,Env:UV_CONFIG_FILE,Env:UV_NO_CONFIG -ErrorAction SilentlyContinue
|
||||
$env:APPDATA = ''
|
||||
$env:ProgramData = ''
|
||||
Set-Location -LiteralPath '{work}'
|
||||
[string](Test-WoaResolveReachesPyPI)
|
||||
""")
|
||||
assert out == str(reaches), f"{name}: expected reaches={reaches}, got {out}"
|
||||
180
tests/studio/install/test_woa_wheelhouse_prefers_pypi.py
Normal file
180
tests/studio/install/test_woa_wheelhouse_prefers_pypi.py
Normal file
|
|
@ -0,0 +1,180 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""The wheelhouse is for wheels PyPI does not build, and only for those.
|
||||
|
||||
The Windows on ARM wheelhouse exists because PyPI publishes no win_arm64 build of pyarrow,
|
||||
tiktoken, grpcio, brotli, hf_transfer or sqlite-vec. regex is different: PyPI has shipped
|
||||
win_arm64 regex since 2025.7.29, and the staging directory is first in UV_FIND_LINKS, so a
|
||||
user asking for regex got our binary rather than the one the project released.
|
||||
|
||||
So staging asks PyPI first and skips the copy when PyPI publishes the same project at or
|
||||
above the version that would have been staged. The version half matters as much as the
|
||||
check: an upstream BEHIND the wheelhouse has to leave ours in place, or the guard turns
|
||||
into a downgrade.
|
||||
|
||||
Text-level tests, like the rest of the installer suite: install.ps1 is PowerShell, so the
|
||||
shape is asserted against its source rather than by running it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
import pytest
|
||||
|
||||
REPO_ROOT = pathlib.Path(__file__).resolve().parents[3]
|
||||
INSTALL_PS1 = REPO_ROOT / "install.ps1"
|
||||
|
||||
|
||||
@pytest.fixture(scope = "module")
|
||||
def source() -> str:
|
||||
return INSTALL_PS1.read_text(encoding = "utf-8-sig")
|
||||
|
||||
|
||||
def _function_body(source: str, name: str) -> str:
|
||||
"""One nested ` function Name {` block, to its closing ` }`."""
|
||||
match = re.search(r"(?ms)^ function " + re.escape(name) + r"\b.*?^ \}$", source)
|
||||
assert match, f"install.ps1 no longer defines {name}"
|
||||
return match.group(0)
|
||||
|
||||
|
||||
def _staging_loop(source: str) -> str:
|
||||
"""The generic wheelhouse mirror: the block that copies everything but pyarrow."""
|
||||
match = re.search(
|
||||
r"(?ms)^ \$WoaExtraStaged = 0$.*?^ if \(\$WoaExtraStaged -gt 0\)", source
|
||||
)
|
||||
assert match, "the wheelhouse staging loop moved"
|
||||
return match.group(0)
|
||||
|
||||
|
||||
def test_the_pypi_probe_is_pypi_only(source):
|
||||
"""Test-WoaWheelAvailable falls back to the wheelhouse, where the wheel being staged
|
||||
always is, so the guard would never fire. This probe has to ask PyPI and nothing else."""
|
||||
body = _function_body(source, "Test-WoaPyPIWheel")
|
||||
assert "pypi.org/simple" in body
|
||||
for other in ("WoaWheelhouse", "index.txt", "Get-ChildItem"):
|
||||
assert other not in body, f"Test-WoaPyPIWheel consults {other}; it must ask PyPI alone"
|
||||
|
||||
|
||||
def test_both_staging_branches_skip_what_pypi_publishes(source):
|
||||
"""Both the local-directory branch and the URL branch mirror the wheelhouse, so a guard
|
||||
on one of them leaves the other shipping our copy."""
|
||||
loop = _staging_loop(source)
|
||||
calls = re.findall(r"Test-WoaWheelhouseWheelIsRedundant", loop)
|
||||
assert (
|
||||
len(calls) == 2
|
||||
), f"expected the redundancy guard in both staging branches, found {len(calls)}"
|
||||
|
||||
|
||||
def test_the_guard_is_version_aware(source):
|
||||
"""Published is not enough: upstream behind the wheelhouse means dropping ours
|
||||
downgrades the install, so the floor is the staged wheel's own version."""
|
||||
body = _function_body(source, "Test-WoaWheelhouseWheelIsRedundant")
|
||||
assert "-Floor $fields[1]" in body, "the guard must floor PyPI at the staged version"
|
||||
probe = _function_body(source, "Test-WoaPyPIWheel")
|
||||
assert "Test-WoaVersionAtLeast" in probe, "the probe must compare versions, not just presence"
|
||||
|
||||
|
||||
def test_the_guard_only_judges_wheels_this_venv_could_use(source):
|
||||
"""A cp312 wheel in the wheelhouse is not made redundant by a cp313 wheel on PyPI."""
|
||||
body = _function_body(source, "Test-WoaWheelhouseWheelIsRedundant")
|
||||
assert "Test-WoaWheelTagsUsable" in body
|
||||
|
||||
|
||||
def test_interpreter_agnostic_wheels_still_count(source):
|
||||
"""hf_transfer ships cp38-abi3 and sqlite_vec py3-none, both usable on cp313: an
|
||||
exact-tag test would call them foreign and go on shipping ours forever."""
|
||||
body = _function_body(source, "Test-WoaWheelTagsUsable")
|
||||
assert '"abi3"' in body and '"none"' in body
|
||||
# Free-threaded venvs are the exception the exact-tag helper exists for.
|
||||
assert '$AbiTag -like "*t"' in body
|
||||
|
||||
|
||||
def test_pyarrow_keeps_its_own_pypi_first_path(source):
|
||||
"""Get-WoaPyarrowSource already asks PyPI first and the generic loop skips pyarrow for
|
||||
that reason. Both halves have to stay, or pyarrow is staged twice or never probed."""
|
||||
body = _function_body(source, "Get-WoaPyarrowSource")
|
||||
assert 'return "pypi"' in body
|
||||
loop = _staging_loop(source)
|
||||
assert re.search(
|
||||
r'-like "pyarrow-\*"\) \{ continue \}', loop
|
||||
), "the generic loop must leave pyarrow to Get-WoaPyarrowSource"
|
||||
|
||||
|
||||
# The default wheelhouse URL had no test at all: changing it to a working but wrong host left
|
||||
# the whole suite green. It is the only source for the pyarrow that gates the native path, so a
|
||||
# typo sends every Windows on ARM host back to the emulated x64 stack, and it is fetched over
|
||||
# the network, so a wrong host is a wrong download.
|
||||
DEFAULT_WHEELHOUSE = "https://huggingface.co/danielhanchen/unsloth-blackwell-docker/resolve/main/windows-arm64-wheels"
|
||||
|
||||
PWSH = shutil.which("pwsh")
|
||||
requires_pwsh = pytest.mark.skipif(PWSH is None, reason = "pwsh not available")
|
||||
|
||||
|
||||
def _wheelhouse_assignment(source: str) -> str:
|
||||
"""The `$script:WoaWheelhouse = if (...) {...} else {...}` block, lifted verbatim."""
|
||||
match = re.search(r"(?ms)^ \$script:WoaWheelhouse = if .*?^ \}$", source)
|
||||
assert match, "install.ps1 no longer assigns $script:WoaWheelhouse in one block"
|
||||
return match.group(0)
|
||||
|
||||
|
||||
def test_the_default_wheelhouse_url_is_exactly_this(source):
|
||||
assert DEFAULT_WHEELHOUSE in _wheelhouse_assignment(source)
|
||||
|
||||
|
||||
def test_the_default_is_an_https_resolve_url_on_hugging_face(source):
|
||||
"""`resolve/main` serves the file; a plain repo URL serves an HTML page, which the
|
||||
staging code would happily save as a .whl. The path may continue past `resolve/main`:
|
||||
the wheelhouse is a folder of a repo we own."""
|
||||
assert DEFAULT_WHEELHOUSE.startswith("https://huggingface.co/danielhanchen/")
|
||||
assert "/resolve/main" in DEFAULT_WHEELHOUSE
|
||||
assert not DEFAULT_WHEELHOUSE.endswith("/"), "Join-UrlPath adds the slash"
|
||||
|
||||
|
||||
@requires_pwsh
|
||||
@pytest.mark.parametrize(
|
||||
("configured", "expected", "why"),
|
||||
[
|
||||
(None, DEFAULT_WHEELHOUSE, "unset falls back to the published wheelhouse"),
|
||||
("", DEFAULT_WHEELHOUSE, "empty is not a configuration"),
|
||||
("https://example.test/wheels", "https://example.test/wheels", "a mirror is honoured"),
|
||||
("https://example.test/wheels/", "https://example.test/wheels", "one trailing slash goes"),
|
||||
("https://example.test/wheels///", "https://example.test/wheels", "so do several"),
|
||||
(
|
||||
" https://example.test/wheels ",
|
||||
"https://example.test/wheels",
|
||||
"surrounding space goes",
|
||||
),
|
||||
(r"C:\wheels", r"C:\wheels", "a local directory survives untouched"),
|
||||
(
|
||||
"C:" + chr(92) + "wheels" + chr(92),
|
||||
"C:" + chr(92) + "wheels" + chr(92),
|
||||
"TrimEnd takes '/' only, so a trailing backslash stays",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_the_wheelhouse_override_is_normalised(source, configured, expected, why):
|
||||
setup = (
|
||||
"Remove-Item Env:UNSLOTH_WOA_WHEELHOUSE -ErrorAction SilentlyContinue; "
|
||||
if configured is None
|
||||
else f"$env:UNSLOTH_WOA_WHEELHOUSE = '{configured}'; "
|
||||
)
|
||||
script = (
|
||||
"$ErrorActionPreference = 'Stop'; "
|
||||
+ setup
|
||||
+ _wheelhouse_assignment(source).strip()
|
||||
+ '; Write-Output "<<<$script:WoaWheelhouse>>>"'
|
||||
)
|
||||
done = subprocess.run(
|
||||
[PWSH, "-NoProfile", "-NonInteractive", "-Command", script],
|
||||
capture_output = True,
|
||||
text = True,
|
||||
timeout = 120,
|
||||
)
|
||||
assert done.returncode == 0, done.stderr
|
||||
out = done.stdout
|
||||
assert out[out.index("<<<") + 3 : out.rindex(">>>")] == expected, why
|
||||
314
tests/studio/install/woa_ps_harness.py
Normal file
314
tests/studio/install/woa_ps_harness.py
Normal file
|
|
@ -0,0 +1,314 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""Shared plumbing for the Windows-on-ARM installer tests.
|
||||
|
||||
install.ps1 and studio/setup.ps1 cannot dot-source each other and cannot be imported, so a
|
||||
behavioural test lifts a function body or a live block out of the script under test, pastes
|
||||
it into a bare pwsh session with the few stubs that block needs, and asserts on what it
|
||||
writes out. The sources, the lifting and the stub bundles are the same from one test to the
|
||||
next, so they live here rather than being restated in every module that wants them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
PACKAGE_ROOT = pathlib.Path(__file__).resolve().parents[3]
|
||||
MANIFEST_PY = PACKAGE_ROOT / "studio" / "install_manifest.py"
|
||||
SETUP_PS1 = PACKAGE_ROOT / "studio" / "setup.ps1"
|
||||
STACK_PY = PACKAGE_ROOT / "studio" / "install_python_stack.py"
|
||||
STACK_LLAMA = PACKAGE_ROOT / "studio" / "install_llama_prebuilt.py"
|
||||
INSTALL_PS1 = PACKAGE_ROOT / "install.ps1"
|
||||
|
||||
# Read once. Well over a hundred tests want one of these whole files, and none of them
|
||||
# mutate what they read.
|
||||
INSTALL_SRC = INSTALL_PS1.read_text(encoding = "utf-8")
|
||||
SETUP_SRC = SETUP_PS1.read_text(encoding = "utf-8")
|
||||
STACK_SRC = STACK_PY.read_text(encoding = "utf-8")
|
||||
LLAMA_SRC = STACK_LLAMA.read_text(encoding = "utf-8")
|
||||
CONSTRAINTS_SRC = (
|
||||
PACKAGE_ROOT / "studio" / "backend" / "requirements" / "single-env" / "constraints.txt"
|
||||
).read_text(encoding = "utf-8")
|
||||
|
||||
PWSH = shutil.which("pwsh")
|
||||
requires_pwsh = pytest.mark.skipif(PWSH is None, reason = "pwsh not available")
|
||||
|
||||
# The channels, mirrors and indexes these tests name over and over. Spelled once so a
|
||||
# parametrize row stays one readable line instead of six.
|
||||
NV_GA = "https://pypi.nvidia.com/nvtorch_oot"
|
||||
NV_NIGHTLY = "https://pypi.nvidia.com/nvtorch_oot_nightly"
|
||||
PYPI = "https://pypi.org/simple"
|
||||
CORP_INDEX = "https://pypi.corp.test/simple"
|
||||
|
||||
|
||||
def _script(*lines: str) -> str:
|
||||
"""The lines of a PowerShell snippet, joined the way pwsh -Command wants them."""
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _ps(
|
||||
script,
|
||||
timeout = 120,
|
||||
**kwargs,
|
||||
):
|
||||
"""Run a PowerShell snippet and hand back the completed process."""
|
||||
return subprocess.run(
|
||||
[PWSH, "-NoProfile", "-NonInteractive", "-Command", script],
|
||||
capture_output = True,
|
||||
text = True,
|
||||
timeout = timeout,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
|
||||
def _ps_ok(
|
||||
script,
|
||||
timeout = 120,
|
||||
**kwargs,
|
||||
):
|
||||
"""Same, but fail the test with PowerShell's own stderr when it does not exit 0."""
|
||||
done = _ps(script, timeout = timeout, **kwargs)
|
||||
assert done.returncode == 0, done.stderr
|
||||
return done
|
||||
|
||||
|
||||
def _ps_last(script, **kwargs) -> str:
|
||||
"""Run a snippet that must succeed and hand back its last line of output.
|
||||
|
||||
Every one of these scripts ends in a Write-Output the assertion is about, and a
|
||||
PowerShell prelude can print before it, so the last line is the answer.
|
||||
"""
|
||||
return _ps_ok(script, **kwargs).stdout.strip().splitlines()[-1]
|
||||
|
||||
|
||||
def _ps_kv(script, **kwargs) -> dict:
|
||||
"""A snippet whose Write-Outputs are NAME=value lines, read back as a dict."""
|
||||
return dict(l.split("=", 1) for l in _ps_ok(script, **kwargs).stdout.splitlines() if "=" in l)
|
||||
|
||||
|
||||
def _function_source(text: str, name: str) -> str:
|
||||
"""Extract a PowerShell function by matching balanced braces."""
|
||||
match = re.search(rf"(?im)^[ \t]*function[ \t]+{re.escape(name)}\b", text)
|
||||
assert match, f"{name} is not defined in setup.ps1"
|
||||
start = text.index("{", match.start())
|
||||
depth = 0
|
||||
for index in range(start, len(text)):
|
||||
if text[index] == "{":
|
||||
depth += 1
|
||||
elif text[index] == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
return text[match.start() : index + 1]
|
||||
raise AssertionError(f"unbalanced braces in {name}")
|
||||
|
||||
|
||||
def functions(text: str, *names: str) -> str:
|
||||
"""Several bodies in one lift, in the order given.
|
||||
|
||||
PowerShell does not hoist and a helper the prelude does not lift is a
|
||||
command-not-found, not a false answer, so a body's helpers come with it.
|
||||
"""
|
||||
return "\n".join(_function_source(text, name) for name in names)
|
||||
|
||||
|
||||
def _ps_function(path: pathlib.Path, name: str) -> str:
|
||||
"""A function body by SCRIPT PATH, for the tests that parametrize over the two files."""
|
||||
src = {INSTALL_PS1: INSTALL_SRC, SETUP_PS1: SETUP_SRC}.get(path)
|
||||
return _function_source(src if src is not None else path.read_text(encoding = "utf-8"), name)
|
||||
|
||||
|
||||
def _ps_copies(name: str) -> tuple:
|
||||
"""install.ps1's and setup.ps1's copies of one function, bar comments and indentation.
|
||||
|
||||
Neither script can dot-source the other, so each is carried twice and the parity is
|
||||
pinned instead. Returned as a pair rather than compared here so a mismatch fails with
|
||||
pytest's own diff of the two bodies.
|
||||
"""
|
||||
|
||||
def normalized(source: str) -> str:
|
||||
lines = [
|
||||
line.rstrip()
|
||||
for line in _function_source(source, name).splitlines()
|
||||
if line.strip() and not line.strip().startswith("#")
|
||||
]
|
||||
indent = min(len(line) - len(line.lstrip()) for line in lines)
|
||||
return "\n".join(line[indent:] for line in lines)
|
||||
|
||||
return normalized(INSTALL_SRC), normalized(SETUP_SRC)
|
||||
|
||||
|
||||
def slice_between(
|
||||
src: str,
|
||||
start_marker: str,
|
||||
end_marker: str,
|
||||
*,
|
||||
include_end = False,
|
||||
) -> str:
|
||||
"""The live text from one marker to the next, sliced out of the script itself.
|
||||
|
||||
Restating a block in the test instead means a copy that passes forever after the
|
||||
original stopped matching it, which is the failure these tests exist to catch.
|
||||
"""
|
||||
start = src.index(start_marker)
|
||||
end = src.index(end_marker, start)
|
||||
return src[start : end + len(end_marker)] if include_end else src[start:end]
|
||||
|
||||
|
||||
# ── Stub bundles ──────────────────────────────────────────────────────────────────────────
|
||||
|
||||
SUBSTEP_NOOP = "function substep { param($m, $c) }"
|
||||
|
||||
|
||||
def substep_collector(var: str = "Messages") -> str:
|
||||
"""substep, collected into $script:<var> so a test can assert on what was said."""
|
||||
return _script(
|
||||
f"$script:{var} = @()",
|
||||
f"function substep {{ param($m, $c) $script:{var} += $m }}",
|
||||
)
|
||||
|
||||
|
||||
# The real one shortens a path to its 8.3 form; a passthrough is what every test that is
|
||||
# not about 8.3 wants, and it keeps the assertions readable.
|
||||
UV_SAFE_PATH = "function Get-UvSafePath { param([string]$Path) return $Path }"
|
||||
|
||||
JOIN_URL_PATH = _script(
|
||||
"function Join-UrlPath { param([string]$Base,[string]$Path)",
|
||||
" return ($Base.TrimEnd('/') + '/' + $Path.TrimStart('/')) }",
|
||||
)
|
||||
JOIN_URL_RETURNS_BASE = "function Join-UrlPath { param($Base, $Path) return $Base }"
|
||||
JOIN_URL_RETURNS_PATH = "function Join-UrlPath { param($Base, $Path) return $Path }"
|
||||
|
||||
|
||||
def invoke_restmethod(body: str) -> str:
|
||||
"""Invoke-RestMethod serving one synthetic PEP 503 page, so the probes are offline."""
|
||||
return (
|
||||
"function Invoke-RestMethod { param([Parameter(ValueFromRemainingArguments=$true)]$a) "
|
||||
f"return @'\n{body}\n'@ }}"
|
||||
)
|
||||
|
||||
|
||||
INVOKE_RESTMETHOD_OFFLINE = (
|
||||
"function Invoke-RestMethod { param([Parameter(ValueFromRemainingArguments=$true)]$a) "
|
||||
"throw 'offline' }"
|
||||
)
|
||||
INVOKE_RESTMETHOD_NO_NETWORK = "function Invoke-RestMethod { throw 'no network in this test' }"
|
||||
|
||||
# The resolver settings a lifted block must not inherit from the session running the tests.
|
||||
UV_INDEX_ENV = (
|
||||
"UV_NO_INDEX",
|
||||
"PIP_NO_INDEX",
|
||||
"UV_DEFAULT_INDEX",
|
||||
"UV_INDEX_URL",
|
||||
"PIP_INDEX_URL",
|
||||
"UV_INDEX",
|
||||
"UV_EXTRA_INDEX_URL",
|
||||
"PIP_EXTRA_INDEX_URL",
|
||||
"UV_CONFIG_FILE",
|
||||
)
|
||||
UV_POLICY_ENV = ("UV_OFFLINE",) + UV_INDEX_ENV + ("UV_NO_CONFIG",)
|
||||
UV_ONLY_INDEX_ENV = (
|
||||
"UV_NO_INDEX",
|
||||
"UV_DEFAULT_INDEX",
|
||||
"UV_INDEX_URL",
|
||||
"UV_INDEX",
|
||||
"UV_EXTRA_INDEX_URL",
|
||||
"UV_CONFIG_FILE",
|
||||
)
|
||||
|
||||
|
||||
def clear_env(names) -> str:
|
||||
"""Remove every one of `names` from the environment the lifted block will read."""
|
||||
listed = ",".join(f"'{name}'" for name in names)
|
||||
return f'foreach ($n in {listed}) {{ Remove-Item "Env:$n" -ErrorAction SilentlyContinue }}'
|
||||
|
||||
|
||||
# The four marker helpers travel together: the path builder, the write guard, the writer and
|
||||
# the reader. Injecting one without the others is a command-not-found inside the body.
|
||||
MARKER_FUNCS = functions(
|
||||
SETUP_SRC,
|
||||
"Get-WoaTorchIndexMarkerPath",
|
||||
"Test-WoaPersistableIndex",
|
||||
"Save-WoaTorchIndexMarker",
|
||||
"Get-WoaTorchIndexMarker",
|
||||
)
|
||||
|
||||
# Tag matching, the version order it needs, the floor and the pyarrow gate built on both.
|
||||
PYARROW_FLOOR = '$script:WoaPyarrowFloor = "21.0.0"'
|
||||
WHEEL_TAG_FUNCS = functions(INSTALL_SRC, "Test-WoaWheelTags", "Test-WoaWheelTagsUsable")
|
||||
PYARROW_USABLE_FUNCS = _script(
|
||||
WHEEL_TAG_FUNCS,
|
||||
_function_source(INSTALL_SRC, "Test-WoaVersionAtLeast"),
|
||||
PYARROW_FLOOR,
|
||||
_function_source(INSTALL_SRC, "Test-WoaPyarrowWheelUsable"),
|
||||
)
|
||||
|
||||
|
||||
def pyarrow_source_script(
|
||||
*,
|
||||
wheelhouse: str,
|
||||
local: str = "$false",
|
||||
rest_method: str = INVOKE_RESTMETHOD_NO_NETWORK,
|
||||
reaches_pypi: str | None = None,
|
||||
lifts: tuple = (),
|
||||
preamble: tuple = (),
|
||||
tail: tuple = ("Write-Output ('[' + (Get-WoaPyarrowSource -PythonMinor '3.13') + ']')",),
|
||||
) -> str:
|
||||
"""Get-WoaPyarrowSource with its network branches stubbed out.
|
||||
|
||||
`wheelhouse` is the PowerShell expression for $script:WoaWheelhouse, `lifts` names any
|
||||
extra install.ps1 helper the branch under test reaches for.
|
||||
"""
|
||||
return _script(
|
||||
SUBSTEP_NOOP,
|
||||
JOIN_URL_RETURNS_BASE,
|
||||
f"function Test-WoaWheelhouseIsLocal {{ {local} }}",
|
||||
rest_method,
|
||||
*((f"function Test-WoaResolveReachesPyPI {{ {reaches_pypi} }}",) if reaches_pypi else ()),
|
||||
f"$script:WoaWheelhouse = {wheelhouse}",
|
||||
PYARROW_USABLE_FUNCS,
|
||||
functions(INSTALL_SRC, "Test-ZipArchiveReadable", *lifts, "Get-WoaPyarrowSource"),
|
||||
*preamble,
|
||||
*tail,
|
||||
)
|
||||
|
||||
|
||||
def native_probe_script(
|
||||
*,
|
||||
driver: str,
|
||||
driver_leaf: str = "$null",
|
||||
stubs: tuple = (),
|
||||
lifts: tuple = (),
|
||||
outputs: tuple = (),
|
||||
) -> str:
|
||||
"""Initialize-WoaNativeCudaTorch run against a stubbed host.
|
||||
|
||||
`stubs` carries the index list and the wheel answers, which is all the callers differ
|
||||
by; `outputs` names the extra script variables to read back between NATIVE and MSG.
|
||||
"""
|
||||
return _script(
|
||||
"$SkipTorch = $false",
|
||||
substep_collector(),
|
||||
"function Get-HostMachineArch { 'arm64' }",
|
||||
"function Get-WoaAbiTag { param($PythonMinor, $FreeThreaded) 'cp313' }",
|
||||
"function Test-WoaNvidiaPresent { $true }",
|
||||
"function Test-WoaResolverPathsUsable { $true }",
|
||||
f"function Get-WoaDriverCudaLeaf {{ {driver_leaf} }}",
|
||||
f"function Get-WoaDriverCudaVersion {{ {driver} }}",
|
||||
*stubs,
|
||||
"function Get-WoaPyarrowSource { param($PythonMinor, $AbiTag) 'pypi' }",
|
||||
"function Test-WoaWheelAvailable { $true }",
|
||||
functions(
|
||||
INSTALL_SRC, *lifts, "Test-WoaAudioMatchesTorch", "Initialize-WoaNativeCudaTorch"
|
||||
),
|
||||
"Initialize-WoaNativeCudaTorch -PythonMinor '3.13'",
|
||||
"Write-Output ('NATIVE=' + $script:WoaNativeCudaTorch)",
|
||||
*outputs,
|
||||
"Write-Output ('MSG=' + ($script:Messages -join ' | '))",
|
||||
)
|
||||
|
|
@ -3,8 +3,8 @@
|
|||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
# Unit test for install.ps1's New-UnslothTorchOverridesFile, the Windows twin of install.sh's
|
||||
# _build_unsloth_torch_overrides. It folds a caller's UV_OVERRIDE file into the frozen torch-trio
|
||||
# pins without corrupting it: non-ASCII lines must survive, and the merged file must stay beside
|
||||
# the caller's override, as uv resolves `-r nested.txt` relative to THAT file's own directory.
|
||||
# pins without corrupting it: non-ASCII lines must survive, and every relative reference must come
|
||||
# across REBASED, since uv resolves them against the file that contains them.
|
||||
# Run: pwsh -NoProfile -File tests/studio/test_torch_overrides_merge.ps1
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
|
@ -21,6 +21,14 @@ $fn = $ast.FindAll({ param($n)
|
|||
$n.Name -eq "New-UnslothTorchOverridesFile"
|
||||
}, $true)
|
||||
if ($fn.Count -ne 1) { throw "expected exactly one New-UnslothTorchOverridesFile in install.ps1, found $($fn.Count)" }
|
||||
# PowerShell does not hoist, so the helpers the merge calls have to be defined here too.
|
||||
foreach ($helper in "Get-WoaRequirementEntries", "Resolve-WoaOverrideLine") {
|
||||
$h = $ast.FindAll({ param($n)
|
||||
$n -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $n.Name -eq $helper
|
||||
}, $true)
|
||||
if ($h.Count -ne 1) { throw "expected exactly one $helper in install.ps1, found $($h.Count)" }
|
||||
Invoke-Expression $h[0].Extent.Text
|
||||
}
|
||||
Invoke-Expression $fn[0].Extent.Text
|
||||
|
||||
$failures = 0
|
||||
|
|
@ -77,14 +85,14 @@ try {
|
|||
|
||||
Check "non-ASCII caller requirement survives the merge (no '?' substitution)" `
|
||||
($mergedText -match "caf${acute}pkg==1\.0" -and $mergedText -notmatch 'caf\?pkg')
|
||||
Check "merged file sits where the caller's relative includes still resolve" `
|
||||
(Test-Path -LiteralPath (Join-Path (Split-Path -Parent $merged) "nested.txt"))
|
||||
Check "the caller's relative include is rebased, not carried as a relative line" `
|
||||
($mergedText -notmatch '(?m)^\s*-r\s')
|
||||
Check "and its contents come across instead" ($mergedText -match '(?m)^idna==3\.6$')
|
||||
|
||||
# Guards on the behaviour that already worked, so the fix cannot regress it.
|
||||
Check "frozen torch trio is pinned first" ($mergedText -match '^torch==2\.11\.0\+cu130')
|
||||
Check "caller's torch-trio lines are dropped" `
|
||||
($mergedText -notmatch '(?m)^torch==1\.0$' -and $mergedText -notmatch '(?m)^torchvision==0\.1$')
|
||||
Check "caller's relative include line is carried over" ($mergedText -match '(?m)^-r nested\.txt$')
|
||||
Check "caller's ordinary requirements are carried over" ($mergedText -match '(?m)^plainpkg==2\.0$')
|
||||
Check "merged file is newline terminated" ($mergedText.EndsWith("`n"))
|
||||
|
||||
|
|
@ -141,7 +149,7 @@ try {
|
|||
# Three Windows-only hazards, so each is asserted behaviourally where it can be and
|
||||
# structurally from the AST, which runs everywhere.
|
||||
|
||||
$env:UV_OVERRIDE = $ovPath
|
||||
$env:UV_OVERRIDE = $callerOv
|
||||
$script:TorchOverridesFile = $null
|
||||
$merged = New-UnslothTorchOverridesFile -PythonExe $fakePy
|
||||
$made += $merged
|
||||
|
|
@ -156,8 +164,14 @@ try {
|
|||
# Get-Content decodes a BOM-less file with the ANSI code page on PS 5.1, which is where the
|
||||
# mojibake came from. pwsh on Linux defaults to UTF-8, so the round trip cannot fail here
|
||||
# even unfixed; the assertion holding the fix in place is the one on the reader.
|
||||
# The reading moved into Get-WoaRequirementEntries; the non-ASCII check above holds it.
|
||||
$scanSrc = ($ast.FindAll({ param($n)
|
||||
$n -is [System.Management.Automation.Language.FunctionDefinitionAst] -and
|
||||
$n.Name -eq "Get-WoaRequirementEntries"
|
||||
}, $true))[0].Extent.Text
|
||||
Check "the caller's override is read as UTF-8, not by the shell's default codepage" (
|
||||
($src -match '\[System\.IO\.File\]::ReadAllLines\(') -and ($src -notmatch 'Get-Content -LiteralPath \$ovFile')
|
||||
($scanSrc -match '\[System\.IO\.File\]::ReadAllLines\(') -and
|
||||
($src -notmatch 'Get-Content -LiteralPath \$ovFile')
|
||||
)
|
||||
Check "the inherited ACL is replaced rather than kept" (
|
||||
($src -match 'SetAccessRuleProtection\(\$true, \$false\)') -and ($src -match 'Set-Acl -LiteralPath \$f')
|
||||
|
|
|
|||
|
|
@ -97,7 +97,8 @@ Remove-Item -LiteralPath $leakFile -Force -ErrorAction SilentlyContinue
|
|||
Write-Host "the inherited-override filter drops the torch trio in any casing"
|
||||
# PowerShell's -notmatch is case-insensitive, so a caller override `Torch<2.11` is dropped.
|
||||
$filterPattern = $null
|
||||
if ($fnText -match '\$_ -notmatch ''([^'']+)''') { $filterPattern = $Matches[1] }
|
||||
# The merge folds entry by entry, so the trio filter reads `-match ... { continue }` instead.
|
||||
if ($fnText -match '\$ovEntry\.Line -match ''([^'']+)''') { $filterPattern = $Matches[1] }
|
||||
Check "filter pattern extracted from the helper" ($null -ne $filterPattern)
|
||||
$inherited = @(
|
||||
"# comment survives",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue