mirror of
https://github.com/unslothai/unsloth.git
synced 2026-10-21 14:00:32 +00:00
922e62297e
* Keep the regenerable caches inside the Studio root Unsloth's own data already lives under one root, but the caches it creates along the way do not. Torch inductor, Triton, matplotlib, numba and the CUDA JIT cache all land in the home directory, and DataDesigner builds ~/.data-designer off Path.home() at import time. This pins the regenerable, process-scoped ones in every mode. Part of #8865, which asks for an install contained in a single directory. This half needs no new flags and helps default installs too. Added to the defaults dict in _setup_cache_env, all under the existing cache_root(): TORCHINDUCTOR_CACHE_DIR (was /tmp/torchinductor_<user>), TRITON_HOME and TRITON_CACHE_DIR (~/.triton), TORCH_EXTENSIONS_DIR (~/.cache/torch_extensions), CUDA_CACHE_PATH (~/.nv/ComputeCache), MPLCONFIGDIR (~/.config/matplotlib and ~/.cache/matplotlib), NUMBA_CACHE_DIR (~/.cache/numba), and DATA_DESIGNER_HOME with DATA_DESIGNER_MANAGED_ASSETS_PATH (~/.data-designer). DATA_DESIGNER_HOME is read at import time by data_designer's constants module, so it has to be set before the Data Recipes worker imports the package. The workers are spawned, so they copy this environment and need no change. diffusion_compile_cache.py also stops hardcoding ~/.cache/unsloth/diffusion_compile_cache. It resolves through cache_root() per call now, and still reads a bundle from the old path when one is there, so nobody pays a cold compile for the move alone. Fixes ~/unsloth_compiled_cache. unsloth train, inference and export import the backend in-process and never seeded the cache environment, so they inherited unsloth-zoo's bare relative UNSLOTH_COMPILE_LOCATION and resolved it against whatever directory the shell was in. Running from the home directory left an ~/unsloth_compiled_cache that cache_cleanup then refused to remove, because nothing marked it as ours. Seeding from ensure_studio_backend_path() covers every command that reaches the backend, without making `unsloth --help` pay for importing it. What deliberately does not move: the Hugging Face hub cache stays at ~/.cache/huggingface. It was taken out of the Unsloth tree on purpose in #4591 so that models downloaded before Unsloth was installed, or shared with LM Studio, Ollama and plain huggingface_hub, keep being found instead of downloaded a second time. HF_HOME never moves in any mode either: it owns the token path, and credentials should not follow a cache onto a removable volume. * Tighten the comments this branch adds * Correct the runtime-root docstrings and note the matplotlib version gap * Install the managed runtimes at the master root and contain the modules cache * Point the whisper.cpp builder at the master root and harden the legacy cache probe * Normalize the master root the same way in both installer scripts * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Assert ownership of the runtime children a master root moves * Remove the master root's runtime children on uninstall * Keep a shared staging directory and stop locked master-root runtimes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove only a file from an install-lock path in a master root prebuilt_core.install_lock creates these locks with os.open(O_CREAT | O_EXCL) and writes a pid, so an install lock is always a regular file. The master-root cleanup reached them through _remove_path, which is rm -rf: in a user-chosen root a directory that happens to be named .node.install.lock was deleted whole, with none of the owner-marker proof the runtime children beside it require. The names are fixed and nothing else routes here, so the shape test costs nothing. A symlink is unlinked rather than followed, since -f is true for a link to a file and rm takes the link. The test grew a case for each shape and now names every helper it extracts: an extraction that stops matching went command-not-found inside a subshell the harness swallows, which reads as "nothing was removed" and quietly passes the keep-assertions. * Lift install.ps1's new master-root helper into the preflight harness Two Windows checks in this file were failing on the branch. Get-ManagedLlamaCppDir now calls Get-MasterRootOverride, which the harness's list of functions to copy into the child did not name. A helper that is not lifted does not announce itself: the child dies on the first call, every Write-Host after it is lost, and the two assertions read as though the preflight had resolved the wrong directory. So the list is now checked against the commands the lifted bodies actually call, and a missing one is reported by name. The ownership guard's flag moved into a parameter so the runtime children beside studio\ under a master root are covered even when the Studio home is the legacy one. The shape assertion still names the old variable; it now names the new one, plus the line that defaults it back to the script-level flag, which is what made the old assertion worth making. * Do not pin the compiler caches into a root that contains a space Four fixes, all reproduced against origin/main before and after. 1. torch.compile stopped working when the Studio root contains a space. torch/_inductor/cpp_builder.py joins its arguments with spaces, interpolates the output path unquoted, and reparses the result with shlex.split, so a cache directory containing a space splits mid-path: g++: fatal error: input file .../my is the same as output file Nothing here can fix that quoting, so this stops creating the situation: the five variables a C++ or CUDA toolchain receives on a command line are left unset when the path has whitespace, and Inductor goes back to its own whitespace-free temporary directory, which is what shipped before these were pinned. Everything else still moves. An explicit value from the caller is still honoured; only a default we invented is ours to withhold. Measured on origin/main and on this branch, same fixture: main compiles, the branch before this commit raises CppCompileError, the branch after it compiles. Not a rare shape -- "C:\Users\First Last" is an ordinary Windows account name, so this is the DEFAULT root for a large share of installs. 2. A missing master root ended the whole uninstall under sh. uninstall.sh runs with set -e and a bare assignment takes the exit status of its command substitution, so the canonicalizing cd failed the run whenever the root was already gone: a second uninstall, or a portable install on a disconnected drive. Ordinary bash hides this by clearing errexit inside command substitution; dash, sh and bash --posix do not, and the advertised one-liner pipes into sh on Debian, Ubuntu and WSL. setup.sh already guarded the same call. 3. setup.ps1's Node ownership guard did not cover a master root. The guard and the owner marker both hung off $NodeOverride, which only the UNSLOTH_STUDIO_HOME branch sets. The master-root branch sets $NodeParent and leaves it null, so <master>\node reached the whole-directory os.replace() in install_node_prebuilt.py with no ownership evidence, and the tree the run then created stayed unmarked, which makes the uninstaller keep it as somebody else's. setup.sh had already moved both sites. 4. uninstall.ps1 resolved a relative master root differently from setup. [IO.Path]::GetFullPath anchors at [Environment]::CurrentDirectory, which PowerShell does not keep in step with its own location. After a Set-Location a relative UNSLOTH_HOME named one install to setup and another here, and the owner marker does not help: it spares trees that are not Unsloth's, and the wrongly resolved path is another Unsloth install. Resolved through the provider path now, as Get-CanonicalDir does. Also: the PowerShell uninstaller gets uninstall.sh's install-lock rule, its stale-lock glob keeps the leading dot the rename actually produces, and both uninstallers apply the rule to the default root as well as a master one. tests/sh/test_with_llama_cpp_dir_link_behavior.sh had been failing on this branch: it seeds the variables the lifted block reads and never learned the runtime ownership flag, so the block died on an unbound variable under set -u and printed nothing, which read as seven wrong answers instead of one missing seed. Seeded, and it now says which it is. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stub git and cmake for the extracted whisper block, so its tests do not need a toolchain Four of this branch's own tests failed on a runner without cmake: AssertionError: ERROR: cmake is required They lift the root selection out of scripts/build_whisper_cpp.sh as shipped text, and that slice carries the builder's `command -v` preflight along with it, so the block exits 1 before choosing any directory and the tests fail for a reason unrelated to what they assert. They passed here because cmake is at /usr/bin/cmake and the fixture hardcoded PATH=/usr/bin:/bin. Stubbed rather than cut out of the slice: excising the preflight would mean the tests no longer run the shipped text, which is the only reason to lift it instead of restating it. Verified by running the extracted block directly under a PATH with no cmake, which reports the preflight error and names no directory, and under the same PATH with the stubs prepended, which resolves <root>/whisper.cpp. * Fix six review findings about the master root All six reproduce; each has a test that fails without the fix. 1. UNSLOTH_HOME=$HOME/.unsloth named the root a default install already uses, and the runtime paths did not move, but the ownership flag was set from the variable being present rather than from where the runtimes land. A legacy source-built ~/.unsloth/llama.cpp carries neither the owner marker nor prebuilt metadata, so the assertion rejected an update that used to reuse that build. Customness now comes from comparing the resolved root against the legacy one, canonicalised the same way so a symlinked $HOME does not compare unequal to itself. 2. setup.ps1's Node ownership guard and marker hung off $NodeOverride, which only the UNSLOTH_STUDIO_HOME branch sets, so <master>\node reached install_node_prebuilt.py's whole-directory os.replace() with no ownership evidence and the tree it created stayed unmarked. Both sites now consult the runtime ownership state, as llama.cpp and whisper.cpp already did. 3. _assert_studio_owned_or_absent tested -d, so a dangling symlink and a regular file both read as "nothing is here" and the caller rm -rf'd the path or replaced it. A dangling link is the ordinary case, its target volume merely unmounted, and following it later installs onto somebody's other disk. Every existing entry is now occupied unless it carries ownership, and the refusal says which shape it found. 4. The uninstaller's master-root child gate tested -e, which is false for a dangling symlink, so an unmarked one fell through to _remove_path, which treats -L as present and unlinks it. 5. A whitespace-only UNSLOTH_STUDIO_HOME or STUDIO_HOME is unset to every resolver, but a bare truthiness test called it present and suppressed the master-root branch. The path was then discarded, so an uninstall carrying that environment removed the master root's runtime siblings and left <UNSLOTH_HOME>/studio installed. Both uninstallers trim first. 6. setup.ps1 persists TORCHINDUCTOR_CACHE_DIR to the USER environment, so every later Studio process inherits it and _setup_cache_env's fill-if-unset default never applied on Windows: the containment this branch is for did not happen there, and with long paths off the cache sat at C:\tc even in portable mode. The persisted value now names what the resolver would have chosen. Long paths off still keeps the short drive-root directory for MAX_PATH headroom, and a path containing a space is refused for the same reason storage_roots refuses one. Two more, from the same round: The uninstaller could not find a master root that was named for a single command: `UNSLOTH_HOME=/mnt/portable unsloth studio update` installs the runtimes there and leaves nothing behind, so a later uninstall removed the Studio tree and stranded them. setup.sh now records the root in the Studio tree and the uninstaller reads it. The deny list and the marker gate still apply to whatever the note names, so it cannot license a removal the environment could not. install.sh and install.ps1 do not read UNSLOTH_HOME yet, so a machine with it merely exported had the CLI resolve <master>/studio while the installers put everything at the legacy root, and `unsloth studio` reported that Studio was not set up. The master root now wins only when it has an install; otherwise the one that exists does. When the installers learn the flag this fallback stops being reachable. test_setup_staged_root.sh matched the old spelling of the branch that chooses the compiler cache rather than what it does. It now checks that a staged run takes the stage root and is asked first, which is what that assertion was for. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record the Windows master root, and clear the cache path uninstall leaves behind setup.sh writes a .unsloth-master-root note in the Studio tree and uninstall.sh reads it, so an uninstall run without UNSLOTH_HOME still finds the runtimes that `UNSLOTH_HOME=... unsloth studio update` installed under a root named for one command. setup.ps1 never wrote that note and uninstall.ps1's _MasterRoot consulted only the current environment, so on Windows the same run removed <master>\studio and stranded node, llama.cpp and whisper.cpp beside it. Both halves are here now, matching the POSIX rules: only a master root is noted, never a staging one, the note is staged then renamed, and the first line alone is read. setup.ps1 also persists TORCHINDUCTOR_CACHE_DIR to the User environment, and this branch made that path install-specific. The uninstaller deleted the tree and left the variable, so every later PyTorch process on the account, including ones with nothing to do with Unsloth, compiled into the removed directory and rebuilt part of it. It is cleared when it names a root this run owned. A directory the user chose is theirs, and the shared C:\\tc fallback is not install-specific and is not deleted here either. * Two ways the diffusion migration raised instead of missing Both are pre-existing weaknesses this branch made reachable, so it should carry the fix. json.loads returns [] for "[]" and None for "null", and manifest.get() on either raises AttributeError out of a function whose entire contract is that a bad cache entry is a miss. The manifest being validated now comes from the pre-relocation root, written by an older build on a disk this run has never checked. Path.exists() raises rather than answering False when a parent denies traversal. The write root moved, so it can land on a directory the process does not own, and the unguarded probe left begin() before the legacy fallback could run: exactly the fallback the relocation depends on for a warm start after an upgrade. * Keep a matplotlibrc written under the managed root The legacy probe re-runs on every launch, so on its own it hands a matplotlibrc written under the managed MPLCONFIGDIR to a ~/.config/matplotlib created later by some other tool. Measured across four launches: figure.dpi 177 from the managed config, 222 once a legacy rc appeared, 177 again when it was removed, with the managed rc present throughout. A style that changes on a later launch and changes back is worse than either choice made once. _data_designer_defaults already applies exactly this rule to its own home, and the pre-existing rule is unchanged: an empty managed directory still defers, since _setup_cache_env creates it on the first launch and existence alone would drop a real user matplotlibrc. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop the workspace guard pinning text the master-root fix moved Two stale assertions, both caught by a full local run rather than by CI. The setup.sh half sliced a fixed 600 characters from _assert_studio_owned_or_absent. Adding the dangling-symlink check pushed the second assertion past the window, so the test failed while the guard it is about was intact. The PowerShell half beside it already slices to the end of the function and says why; the sh half now does too. The other assertion named the exact line `[ -z "$_MASTER_ROOT" ] || _RUNTIME_ROOT_IS_CUSTOM=true`, which is no longer the rule: UNSLOTH_HOME can name ~/.unsloth, the root a default install already uses, and treating that as custom made an update reject a legacy source-built llama.cpp carrying no owner marker. tests/test_setup_master_root.py runs the shipped derivation for that case, so this one holds the two halves being present rather than their wording. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Four more ways a generated path could be published or taken wrongly A blank inherited toolchain variable is now dropped rather than left blank when the generated default is refused for whitespace. Inductor distinguishes an absent TORCHINDUCTOR_CACHE_DIR from a present one, so " " became a relative compiler path and was then split by the very unquoted command construction the refusal exists to avoid. "Blank counts as unset" now holds on a spaced root too. A generated toolchain path that could not be made is withheld instead of exported. The placement error is swallowed on purpose so startup survives, which is what let a regular file at <studio>/cache/torchinductor, or a parent with a restrictive ACL, stay published: torch treats the value as authoritative and fails every compile, where an unset variable would have used its own temporary cache. Only these keys. UNSLOTH_COMPILE_LOCATION unset falls back to a bare relative name resolving against the CWD, which is the bug that pin exists to fix. setup.ps1's Node guard asked -PathType Container, so a regular file or a symlink named node under a user-selected master root was invisible to it and then displaced by _swap_into_place. Anything at the path counts as occupied now, and a non-directory is refused outright since ownership evidence lives inside one. Neither uninstaller will take a Studio root that is also the master root. The resolver accepts that flat layout, and a Studio root is removed whole once it carries the ownership marker, so anything else kept in that directory went with the install. Every other master-root child is individually marker-gated for exactly that reason; this was the one hole in the rule. Kept rather than pruned: data left behind is recoverable and printed, a deleted file is not. * A link at managed-assets is use, not the empty layout we create _setup_cache_env only ever makes a plain directory there, so a symlink or a Windows junction is the user sending their assets to another disk. entry.is_dir() follows it, and an empty target therefore read as the untouched layout Unsloth creates: with a standalone ~/.data-designer also present, DATA_DESIGNER_HOME was dropped and the redirect went with it. os.path.isjunction is 3.12, and the Studio venv can be 3.11, so its absence means there are no junctions to worry about rather than an error. * Four ways the master-root cleanup could miss its target or take someone else's The master root is resolved once, before anything is deleted. _master_root can read its answer from a note INSIDE a Studio tree, and the custom-root loop removes that tree, so asking again afterwards returned nothing and stranded the marked llama.cpp, Node and whisper.cpp siblings: the exact failure the note was added to prevent, reintroduced by asking too late. Windows already computed the value early for the stop pass and then re-probed; it reuses it now. The note probe trims the Studio overrides, as _custom_studio_roots and setup.sh both do. UNSLOTH_STUDIO_HOME=" /mnt/studio " puts the note at /mnt/studio/share, and appending to the padded value looked somewhere that does not exist, so the note was never found while the loop still removed /mnt/studio. The stale-lock sweep matches the installer's own shape rather than any dotted name. prebuilt_core.py leaves <component>.install.lock.stale.<pid>; the glob also matched a user's .backup.install.lock.stale.copy, and in a root they chose that file is theirs. Both sweeps, both scripts. And _nothing_at looks at the link before following it. os.scandir follows a symlink or junction at stylelib, so one pointing at an empty directory, or a dangling one whose volume is not mounted, read as an empty directory and let MPLCONFIGDIR move away from a redirect the user created. The Data Designer probe already checked the link itself. * Refuse a non-directory at a managed runtime path on Windows * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the CLI, the backend and both setup halves on one master root Four defects found by simulating the branch rather than reading it. The CLI keeps an installed legacy root when UNSLOTH_HOME names a directory with no install in it, because install.sh and install.ps1 do not read the variable yet. That fallback is not custom, and _ensure_studio_env_exported returned early on non-custom, so nothing was exported at all while storage_roots.studio_root() went on honouring UNSLOTH_HOME with no install check. The CLI ran the legacy venv and the backend under it wrote studio.db, auth and the pid file to <master>/studio. The root is now exported without being marked custom, so the ownership guards are untouched. setup.ps1 derived the runtime ownership flag from whether UNSLOTH_HOME was set, where setup.sh derives it from where the runtimes land. UNSLOTH_HOME pointed at the legacy root moves nothing, and calling it custom there demanded an owner marker from a markerless source-built .unsloth\\llama.cpp that predates the marker, so Assert-StudioOwnedOrAbsent exited an update the shell half adopts. uninstall.ps1 removed the master root and its .staging only when empty, but asked with Get-ChildItem -ErrorAction SilentlyContinue and then called _RemovePath, which is recursive: a swallowed enumeration error read as an empty directory, and a file arriving between the check and the delete went with it. Directory.Delete(path, false) is one call that refuses a non-empty directory, which is what uninstall.sh's rmdir does and what this file's own TEMP prune already does. The bash and PowerShell harnesses in test_pr10648_offline_and_status.py stand in for the setup section that derives these flags, and the guards inside the block they extract now read the runtime flag, so both harnesses have to define it. Nine tests there failed under set -u with 'unbound variable'; the file passes 102 on the merge base and 102 here. * Hold the preflight lift check to install.ps1's own helpers The check resolved every command the lifted bodies call, so main's new Get-SecuritySoftwareNote, which reaches Get-CimInstance and Get-MpPreference, read as two unlifted helpers on a Linux runner and as nothing on Windows. Ask install.ps1 whether it defines the name instead: a helper the lift leaves behind is still caught, on either platform. * Stage the master-root note under an unpredictable name The note setup writes so the uninstaller can find a root that only one command named was staged at "<share>/.unsloth-master-root.$$" and written with a shell redirection. Anyone who can write share/ could precreate that name as a symlink and have setup truncate whatever it points at, which is the shape that matters here: a master root is normally a volume, and a volume is the thing two accounts share. setup.sh states the rule for the uv cache probe 2400 lines earlier, "mktemp (O_EXCL) not $$"; the note follows it now, and setup.ps1 opens its staging file CreateNew under a random name rather than writing through WriteAllText. The rename already replaced a link at the final path rather than following it, so only the staging name was open. tests/test_setup_master_root.py plants the link from the same shell that then runs the shipped block, so the guessed pid matches the way an attacker's would. The PowerShell half is held structurally: the Linux runners have no Windows filesystem to plant on. * Treat a whitespace-only Studio home as unset when exporting it _resolve_studio_home strips before deciding, and so does storage_roots, so " " means unset to both. The export guard read the raw value, where " " is truthy, and left it in place. With a master root exported and an install at the legacy root, that split the two: this process ran the legacy venv while the backend under it resolved <UNSLOTH_HOME>/studio and wrote studio.db, auth and the pid file there. Blank was already covered; whitespace was the shape that got through. uninstall.ps1 had the same bug and this branch already fixed it there. UNSLOTH_LLAMA_CPP_PATH gets the same treatment: llama_cpp.py's discovery takes a whitespace value as a directory to scan, so leaving it costs the master root's build and hands the scan a path that cannot exist. * Migrate the legacy diffusion bundle through a temp file Every other write in this module goes to a temp file in the destination directory and is renamed into place, because a save runs on a daemon thread that interpreter exit can kill anywhere. The migration out of the legacy root used shutil.copyfile straight onto the live names, so an interrupted copy published a partial manifest and a second backend migrating the same key interleaved its writes into one destination file. A torn manifest reads as a miss, which costs exactly the cold compile the migration exists to avoid. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Point the matplotlib containment tests at each platform's config dir The helper hardcoded _get_config_or_cache_dir's Linux branch, so on a mac runner these nine tests wrote the rc into ~/.config/matplotlib, which macOS matplotlib never reads. The guard correctly found nothing to strand, pinned MPLCONFIGDIR, and the assertions failed for a reason that was in the tests rather than in the code they cover; test_matplotlib_reads_the_config_the_pin _would_have_hidden proved it by watching real matplotlib fall back to its built-in matplotlibrc. The derivation is held against real matplotlib on each spoofed platform by the two tests above it, so it cannot drift into agreeing with storage_roots.py alone. * Give the master-root probes an environment Python can start in _resolve and _discover build the child's environment from scratch, and the list left out what the interpreter itself needs: Windows python exits 1 before running a line when SYSTEMROOT is absent. check = True then raised CalledProcessError with the child's stderr discarded, so all nine tests reported as though the resolver had answered wrongly on every Windows runner. The environment now carries SYSTEMROOT and its neighbours, and a non-zero exit asserts with the child's stderr in the message, since a dead interpreter passing for a wrong answer is the failure that hid this. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write the llama-server fixture under the name this platform looks for _find_llama_server_binary appends .exe on Windows, so the fixture's POSIX-only name meant discovery correctly found nothing and the three discovery tests failed on a Windows runner for a reason that was in the fixture. * Skip the setup.sh blocks on Windows rather than run them under Git Bash Sixteen tests here execute blocks of the POSIX installer and were guarded by "is bash on PATH". Git Bash puts it there on a Windows runner, so they ran against Windows path semantics and UTF-16 output and failed sixteen times about a script that platform never uses: `assert '\x00' == 'ALLOWED'` is the shape of it. setup.ps1 is held separately in this file and under tests/studio, so the guard now asks for bash on a POSIX filesystem and the Windows half of a cross-platform run reports only what it really covers. * Probe the cache dirs for writability, keep linked lock paths, harden the Node guard Three defects found in review. storage_roots._usable_dir accepted any existing directory. The mkdir above it passes exist_ok = False and swallows every OSError, so a directory that exists but cannot be written to arrived here having proven nothing, and the value was published anyway. torch treats these variables as authoritative: cache_dir() and triton_cache_dir() fall back to their own temporary directory only when the variable is UNSET, then call os.makedirs(exist_ok = True), which succeeds on an existing read-only directory and leaves every later compile write to fail (torch/_inductor/runtime/cache_dir_utils.py). Probe with a real create via tempfile.mkstemp and unlink it, so an unwritable directory is withheld and torch keeps its own fallback. scripts/uninstall.sh removed a symlink sitting at an install-lock path, so a lock name planted as a link to something outside the install took that link with it. uninstall.ps1 already excludes reparse points at the same names. Keep a link, and keep any other non-file, reporting both. studio/setup.ps1 probed the Node directory with Test-Path -PathType Container. Under Windows PowerShell 5.1 that answers for the link TARGET, so a dangling `node` link read as absent and install_node_prebuilt.py then installed straight through it. Probe with Get-Item -Force, as Assert-StudioOwnedOrAbsent already does, and treat a reparse point as not-a-directory however it resolves. Tests: a read-only managed cache is withheld while NUMBA_CACHE_DIR stays pinned, the write probe leaves nothing behind, a linked and a dangling-linked lock are both kept, and the two Node-guard structure tests now hold the stronger invariant. The sh suite's assert_link uses -L so a dangling link does not read as gone. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Recover the master root from the note setup already writes UNSLOTH_HOME is documented as settable for a single command, `UNSLOTH_HOME=/mnt/portable unsloth studio update`, and that command installs node, llama.cpp and whisper.cpp as SIBLINGS of studio/ under the root it names. Nothing persisted the variable, so once the command's environment went away unsloth_home() returned None and all three resolvers looked somewhere else. Measured against a real master-root tree: later launch, bare node -> ~/.unsloth/node (absent) later launch, UNSLOTH_STUDIO_HOME set node -> <master>/studio/node (absent) what setup installed node -> <master>/node The second row is the one that matters: even with the studio root known correctly, the runtimes resolved to its studio/ CHILD while the installer had put them beside it. whisper.cpp and the llama.cpp path pinned into every worker follow the same call. setup.sh and setup.ps1 already record the root at share/.unsloth-master-root, for the uninstallers. Read it here too, so the process agrees with the install on disk. One fallback in unsloth_home() covers all three resolvers, since each of them keys on it. Two guards keep a note from doing harm. The recorded root must exist, and its studio/ child must be the very directory the note was read from, so a note left behind by a tree that has since moved, or copied into an unrelated install, is ignored rather than adopted for caches and runtimes both. The candidate studio roots are derived without calling studio_root(), which asks unsloth_home() where the studio tree is and would recurse. Tests: the recorded root outlives the command that set it, a moved root and a copied note are both ignored, an empty note is not a root, and a default install still reads none. Each one fails with the fallback removed or a guard weakened. * Read the master-root note from this install's Studio root only Two installs on one box, one legacy and one named by UNSLOTH_STUDIO_HOME. Every note reader walked a list and took the first readable note, with the legacy tree first, so a note belonging to the OTHER install won while the rest of the run stayed on the named one. In storage_roots that split the process against itself: studio_root() returned the named tree while unsloth_home() returned the other install's master root, sending node, whisper.cpp, llama.cpp, the portable caches and orphan cleanup into a tree Studio was not running from. In both uninstallers it was worse than wrong, because a master root licenses removals: the run deleted the named Studio and then followed the other install's root and took its marker-owned runtime children with it. All three now consult exactly one Studio root, the one studio_root() itself would answer with the master root out of the picture: UNSLOTH_STUDIO_HOME, then STUDIO_HOME, then the legacy tree, and no falling through from one to the next. An explicit override names that exact directory, so its absence of a note is an answer, not a reason to keep looking. Also moves this file's three pwsh call sites onto run_pwsh, which tests/studio/ test_pwsh_calls_use_the_shared_runner.py requires: a direct spawn shares one startup cache with every other xdist worker and dies with `Stack overflow.` at a few percent per process. Tests: an unrelated install's note is not adopted and the named root's own note still is, in Python, in sh and in PowerShell, each running the shipped function. Restoring the legacy-first order fails all three. * Require a master-root note to describe the tree it was found in A Studio tree copied from master root A to B carries a note still naming A. Every reader took that at face value, and in the uninstallers it licenses removals: A's llama.cpp, node, whisper.cpp and sd.cpp carry exactly the owner markers B's would, so the marker gates cannot tell them apart and uninstalling the copy deleted the ORIGINAL install's runtimes. All three readers now require the Studio directory a note was read from to lie INSIDE the root it names. Containment rather than an exact <root>/studio match, including in storage_roots, which had the stricter rule: the flat layout, UNSLOTH_HOME and UNSLOTH_STUDIO_HOME naming one directory, is supported and an exact match refuses it. A Studio root deliberately placed outside its master root is not recovered from a note, which storage_roots already calls not self-contained and warns about on every resolve. An explicit UNSLOTH_HOME skips the check, being the user speaking rather than a file found on disk. Both uninstallers also read the default-mode studio.conf when no override names a Studio root. A master-root install puts Studio at <master>/studio, so with a bare environment that conf is the only thing naming the directory the note lives in, and the legacy path could never satisfy containment. Read inline rather than through _custom_studio_roots / _CustomStudioRoots, which call back into this function. unsloth_cli exports the recovered root rather than only reading it. setup runs as a subprocess, so an update with the root in a note but not the environment refreshed the runtimes at <master>/studio/ while the backend, which does read the note, kept launching the stale ones at <master>/. The CLI keeps its own aligned copy of the reader, as it does for studio_root(); tests/test_unsloth_home_root_agreement.py is what holds the two together. Tests: a copied note is refused in Python, in sh and in PowerShell, a flat recorded root is still honoured, and the CLI both recovers a real note and refuses a carried-in one. Four sh fixtures described a master root with no Studio tree under it, which cannot occur; they now describe the real layout. Every case fails with its guard removed. * Give the CLI the same note validation as the backend Relaxing the note check to containment in storage_roots and both uninstallers left the CLI's copy on the stricter <master>/studio match. UNSLOTH_HOME=/root with UNSLOTH_STUDIO_HOME=/root/custom/studio is a real pairing: setup records /root, the Studio directory is inside it, and it is not the default child. The backend accepts that note and the CLI declined it, so a later `unsloth studio update` passed setup no master root at all and refreshed the runtimes under the Studio directory while the backend went on loading the ones at /root. That is the same split this recovery exists to close, reopened one level up. Test: the CLI and the backend agree on a nested Studio root, in tests/test_unsloth_home_root_agreement.py. It fails with the strict rule restored. * Only record a master root a reader will honour Two shapes of the note were being written and then refused by every reader, so the feature recorded nothing while the file on disk said otherwise. Both were found by running the shipped setup.sh block against a fixture and handing the result to the real resolver, not by reading. Containment. setup.sh writes the note at $STUDIO_HOME/share, and STUDIO_HOME is derived only from UNSLOTH_STUDIO_HOME/STUDIO_HOME, never from UNSLOTH_HOME. Every reader requires the Studio directory to lie inside the root the note names. install.sh does not read UNSLOTH_HOME yet, so the documented case, UNSLOTH_HOME=/mnt/portable unsloth studio update on an ordinary install, leaves Studio at ~/.unsloth/studio and puts llama.cpp, node and whisper.cpp at /mnt/portable, which is exactly the shape all four readers reject. The note is now skipped there and the stranding is reported with the two ways out, instead of being recorded in a file nobody reads. The legacy default root. UNSLOTH_HOME=$HOME/.unsloth names the directory the install is already in. Both uninstallers refuse that value outright, but storage_roots honoured it, and honouring it made portable_mode() true on every later bare launch: HF_HUB_CACHE moved off ~/.cache/huggingface, which is the one cache this branch promises not to move, and every model already downloaded went missing. One command that named the default root converted the install permanently, with nothing set at the time. It is no longer recorded, and a note an earlier build left carrying it is cleared on the next run, since the writer only runs when UNSLOTH_HOME is set again and the user who hit this set it once. Also in storage_roots: - A note read that fails for a reason other than absence is no longer cached. _recorded_master_root memoises per studio path and the backend is long-lived, so an EACCES on share/ or an EIO off a mount that came back pinned that process to the legacy runtime paths for its whole lifetime, silently, with the trees it should have found beside it. Nothing in production calls forget_recorded_master_root. A genuine miss is still cached. - _usable_dir guards os.close like it already guards the unlink. Every other failure there is a False; an OSError escaping took _setup_cache_env, and the backend start that called it, down with it. - The UNSLOTH_PORTABLE warning no longer says 0/false/off/no will leave portable mode off. An off value is not a veto: a master root is what makes an install portable, as the comment three lines below already said. Tests: five new cases over the gate, the legacy-root sweep, the cached-note rule and the close guard, plus the setup.ps1 halves held structurally. 205 in test_setup_cache_env_containment and the cache suites, 356 across the master root, root agreement, workspace guard and system32 guard suites, 6710 in tests/studio/install, and all five tests/sh scripts pass; tests/studio/install matches the merge base exactly. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Prove a studio.db is absent before skipping the read, on Windows too The skip added here reads os.stat(studio_db_path()) and treats FileNotFoundError as "no database, nothing to read". On POSIX that is sound: a parent component that is a file raises NotADirectoryError and one that cannot be traversed raises PermissionError, each its own type. Windows collapses all of them into ERROR_PATH_NOT_FOUND, which Python surfaces as FileNotFoundError, so there the exception means "absent, OR something above it is not a directory, OR I could not look" and the guard took the first reading. The result was the cache home chosen in Settings being discarded on Windows and nowhere else. This is the branch's own new test failing, not a theory: test_uninspectable_studio_db_keeps_the_stored_cache_home[not_a_directory] is green on Linux and macOS and red on windows-latest. Measured as a differential on owned staging runners, base against head, same sixteen files, same three operating systems: ubuntu-latest has no real failure on either arm, macos-15 has the same five on both, and windows-latest has 82 on the base and 83 on the head. The one extra is that test. Nothing else the branch touches moved on any of the three. _absence_is_real answers the question the skip actually needs: the nearest ancestor that can be stat'ed decides. A directory means the tree is real and the file genuinely is not in it. Anything else means a component is a file or a reparse point, so the path could never have existed and the caller has learned nothing. An ancestor that cannot be inspected at all is not proof either way. Running out of ancestors means nothing on the path exists, which is the machine that has never opened Studio, and the skip has to hold there or the read builds the 250 KB database it was added to avoid. Checking only the immediate parent is not enough: account_path puts studio.db several levels under the Studio home, so the non-directory is usually further up. Held by a test that reproduces the Windows error shape rather than the platform, since the Linux runners cannot raise it, plus its opposite, an absent Studio home that must still skip. The first fails on the code as it was and passes after, so it measures the fix rather than restating it. 207 pass in the cache suites, 365 with 1 skipped across the master root, root agreement, workspace guard and system32 guard suites, 6754 with 32 skipped in tests/studio/install, 2 in the CLI export suite. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop the module-level errno import the probe does not need errno is used only inside _WINDOWS_SHAPED_GUARD_PROBE, which is a string run in a subprocess and imports it itself. The hoisted import was unused at module level and the source lint blocks on exactly that. * Decline a recorded legacy default root in the reader, not only the writer Found by enumerating the resolver's input space rather than reading it: 512 cells over UNSLOTH_STUDIO_HOME x STUDIO_HOME x UNSLOTH_HOME x UNSLOTH_PORTABLE x the recorded note, base against head, one subprocess per cell. In 12 of them the Hugging Face hub cache still moved off ~/.cache/huggingface with nothing set in the process at all. The write side was already gated: setup.sh and setup.ps1 refuse to record $HOME/.unsloth and sweep an old note away. But that only fires when setup runs again, and the user who hit this set UNSLOTH_HOME once, for one command. Until the reader declines it too the repair never reaches them, and both uninstallers were meanwhile refusing the same value outright, so the four readers the branch claims to keep aligned were not. _is_legacy_default_root declines it in storage_roots, and the CLI copy gets the identical rule, which is what test_unsloth_home_root_agreement.py holds. Recorded roots only: unsloth_home() returns an explicit UNSLOTH_HOME before it ever reads the note, and narrowing that would change what the variable means rather than what a stale file is allowed to claim. Re-run over the same 512 cells: root resolution 480 identical and 32 changed, which is the new UNSLOTH_HOME feature, with 0 violations; HF_HOME moves in 0 cells; of the 192 cells with no UNSLOTH_HOME and no UNSLOTH_PORTABLE, the caches move in 30 and all 30 carry a real recorded master root. Legacy-root notes: 12 before, 0 now. Foreign notes: 0 throughout. Also closes a gap in this file's own tests. Reverting the production hunks and keeping the tests left 25 of 113 containment tests green, because each asserts only that some variable is ABSENT, which is equally true when no pinning code exists: assertions, not negative controls. Three new cases pair each decline with the pins that must still happen in the same run, so the family fails when the resolver does nothing. All five new tests fail with the production hunks reverted and pass here. 119 pass in the containment suite, 147 across it plus the root agreement and managed tools suites. * Refuse a multi-line master-root note in both uninstallers The comment beside each read said "One line, first only: a note that grew a second line is not one we wrote", and neither did that. uninstall.sh took `head -n 1` and carried on; uninstall.ps1 took `Get-Content -TotalCount 1`. The second line was never looked for. That is the four readers disagreeing in the one place it costs something. storage_roots.py and unsloth_cli/commands/studio.py both read_text().strip() the WHOLE file, so a two-line note is not a directory to them and they decline it. The two uninstallers accepted line 1 and went on to authorise removing <master>/llama.cpp, node, whisper.cpp and stable-diffusion.cpp for a master root no runtime reader had agreed to. Measured on the fixture, all four readers, one note: storage_roots.unsloth_home() -> None CLI _recorded_master_root() -> None uninstall.sh _master_root() -> /<master> <- authorises the delete uninstall.ps1 _MasterRoot -> same shape Both now refuse when a second non-blank line is present, which is what the comments already promised. A trailing newline on an ordinary note is not a second line, and the one-line note keeps working -- both are asserted, so the refusal cannot pass by reading nothing. Found by the release-safety fan-out over the install and uninstall paths, then reproduced against the shipped helpers rather than taken on report. tests/sh/test_uninstall_master_root.sh grows three cases and runs the real extracted block: 45 pass here, and the two-line case fails against uninstall.sh exactly as submitted in this PR. The PowerShell half is held structurally, as the rest of that file's Windows coverage is. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not rewrite the master-root note when it already says this Move-Item -Force is not an atomic replace. PowerShell's filesystem provider deletes the destination and then moves the source, so an interruption between the two leaves NO note at all, while the complete staging file sits beside it unreferenced. A backend that looks next reads "no master root" and, since the miss is cached, keeps that answer for its whole lifetime. The window cannot be removed under Windows PowerShell 5.1, which has no atomic replace primitive this code can reach. What can be removed is re-entering it on every single update to write bytes that are already there, which is every run after the first. The note is compared first and the write skipped when it matches, and the staging file is now cleared in a finally rather than left behind when the move throws. Two tests, and the second one took two attempts to make honest: the first version compared LastWriteTime across two runs a millisecond apart, which can land on one filesystem tick, and it passed against the unfixed writer. It back-dates the note instead, so any rewrite is unmistakable, and then changes the value to prove the writer still writes when it should. Both fail against setup.ps1 exactly as submitted in this PR and pass here. Found by the release-safety fan-out over idempotency and interrupted runs, then reproduced against the shipped block under pwsh rather than taken on report. 43 pass in test_setup_master_root, 176 across it plus the containment, root agreement and managed tools suites, and 45 in tests/sh/test_uninstall_master_root.sh. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not read a Windows path error as proof the user's config is absent _nothing_at's docstring is explicit that Path.exists and is_dir must not be used because they report ENOTDIR and ELOOP as absence and the redirect would then hide the user's files. It then took FileNotFoundError at face value, which is the same reading restricted to one platform: POSIX distinguishes "not there" from "a parent component is a file" (NotADirectoryError) and "I may not traverse it" (PermissionError), while Windows collapses all three into ERROR_PATH_NOT_FOUND. So with a file where ~/.matplotlib belongs, POSIX declined the pin and Windows took it, moving MPLCONFIGDIR for the whole process and hiding whatever was underneath. Same defect and same cause as the studio.db guard fixed in 37f23c576, in a second probe, and _nothing_above is the same rule: the nearest ancestor that can be stat'ed decides, a directory means real absence, anything else means the path could never have existed, an uninspectable ancestor proves nothing, and no ancestors at all is the default install and is real absence. Found by the windows-latest staging leg, on a negative control written for the POSIX shape and added earlier in this review. base arm 82 real failures, head 83, the one extra being that test. It is the only new failure on any of the three operating systems. Held by a test that reproduces the error shape rather than the platform, since the Linux runners cannot raise it, with both opposite cases asserted so the fix cannot pass by declining everything: a genuinely empty present directory still reads as empty, and a path with nothing along it is still absence. It fails against storage_roots.py exactly as submitted in this PR. 584 pass with 1 skipped across the eleven suites this branch touches. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comment prose around the master-root resolvers * Give the cache-pin probes one table each, and the PowerShell probes one helper * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse a master-root note that names the legacy tree, and stop Node discovery seeding caches Three fixes found while auditing what happens to an existing install. A note left at <studio>/share/.unsloth-master-root that names the legacy tree itself passed every containment check, so a bare environment resolved a master root and the uninstaller then treated a second marked install under it as a root to remove. Both uninstallers and both setup scripts now decline a note whose directory is the legacy Studio dir. A staged update derived the runtime ownership flag from the master root, so a staging pass could mark runtimes custom when they were still legacy. The flag is now assigned only when no stage root is in play, and follows where the runtimes actually land. Node tool discovery imported the Studio backend path purely to find the bundled tools and built the whole compiler cache tree as a side effect. The import now takes a seed_cache_env flag and the discovery path opts out. Each fix has a regression test checked against the unfixed code. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <moonshotaisubstack@gmail.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
87 lines
4 KiB
Bash
Executable file
87 lines
4 KiB
Bash
Executable file
#!/bin/sh
|
|
# Build whisper.cpp's whisper-server for Unsloth's GGUF dictation engine.
|
|
# Installs into the managed Unsloth home so the backend's binary discovery
|
|
# (core/inference/stt_ggml_sidecar.py::find_whisper_server_binary) picks it up:
|
|
# <UNSLOTH_HOME>/whisper.cpp/build/bin/whisper-server (master root)
|
|
# <UNSLOTH_STUDIO_HOME>/whisper.cpp/build/bin/whisper-server (custom home)
|
|
# ~/.unsloth/whisper.cpp/build/bin/whisper-server (default)
|
|
# Usage:
|
|
# ./scripts/build_whisper_cpp.sh # build the pinned tag
|
|
# WHISPER_CPP_TAG=v1.9.0 ./scripts/build_whisper_cpp.sh
|
|
# Requires: git, cmake, a C/C++ toolchain (the same prerequisites as a
|
|
# llama.cpp source build). GPU backends are auto-detected by whisper.cpp's
|
|
# CMake (Metal on macOS; set GGML_CUDA=1 to force a CUDA build on Linux).
|
|
|
|
set -eu
|
|
|
|
WHISPER_CPP_SOURCE="${WHISPER_CPP_SOURCE:-https://github.com/ggml-org/whisper.cpp}"
|
|
WHISPER_CPP_TAG="${WHISPER_CPP_TAG:-v1.9.1}"
|
|
|
|
# Stripped and tilde-expanded before it can outrank anything, as setup.sh and the Python
|
|
# resolvers do: ${VAR:-} only treats the EMPTY string as unset, so a whitespace-only value
|
|
# would win here and name a relative " /whisper.cpp" the backend never looks in.
|
|
_root_value() {
|
|
_rv=$(printf '%s' "${1:-}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')
|
|
case "$_rv" in
|
|
"~") _rv="$HOME" ;;
|
|
"~/"*) _rv="$HOME/${_rv#'~/'}" ;;
|
|
esac
|
|
printf '%s' "$_rv"
|
|
}
|
|
|
|
# UNSLOTH_HOME first: whisper.cpp is a SIBLING of studio/ under the master root, and the CLI
|
|
# exports UNSLOTH_STUDIO_HOME=<root>/studio beside it, so taking that one would install a level
|
|
# below where stt_ggml_sidecar._managed_whisper_cpp_dir() looks.
|
|
_STUDIO_HOME_ALIAS="${STUDIO_HOME:-}" # read before the name below is reassigned
|
|
STUDIO_HOME="$(_root_value "${UNSLOTH_HOME:-}")"
|
|
[ -n "$STUDIO_HOME" ] || STUDIO_HOME="$(_root_value "${UNSLOTH_STUDIO_HOME:-}")"
|
|
[ -n "$STUDIO_HOME" ] || STUDIO_HOME="$(_root_value "$_STUDIO_HOME_ALIAS")"
|
|
CUSTOM_STUDIO_HOME=false
|
|
if [ -n "$STUDIO_HOME" ]; then
|
|
CUSTOM_STUDIO_HOME=true
|
|
INSTALL_DIR="$STUDIO_HOME/whisper.cpp"
|
|
else
|
|
INSTALL_DIR="$HOME/.unsloth/whisper.cpp"
|
|
fi
|
|
|
|
command -v git >/dev/null 2>&1 || { echo "ERROR: git is required" >&2; exit 1; }
|
|
command -v cmake >/dev/null 2>&1 || { echo "ERROR: cmake is required" >&2; exit 1; }
|
|
|
|
# Same policy as studio/setup.sh's _assert_studio_owned_or_absent: never delete
|
|
# a directory under a custom Unsloth home unless Unsloth itself created it (the
|
|
# marker file below). Protects a user-managed whisper.cpp/src from rm -rf.
|
|
STUDIO_OWNED_MARKER=".unsloth-studio-owned"
|
|
if [ "$CUSTOM_STUDIO_HOME" = true ] && [ -e "$INSTALL_DIR" ] && \
|
|
[ ! -f "$INSTALL_DIR/$STUDIO_OWNED_MARKER" ]; then
|
|
echo "ERROR: $INSTALL_DIR already exists and is not marked as an Unsloth-owned whisper.cpp build tree." >&2
|
|
echo " Move it aside or choose an empty UNSLOTH_STUDIO_HOME before re-running." >&2
|
|
exit 1
|
|
fi
|
|
|
|
echo "==> Building whisper.cpp ($WHISPER_CPP_TAG) into $INSTALL_DIR"
|
|
mkdir -p "$INSTALL_DIR"
|
|
: > "$INSTALL_DIR/$STUDIO_OWNED_MARKER"
|
|
|
|
if [ ! -d "$INSTALL_DIR/src/.git" ]; then
|
|
rm -rf "$INSTALL_DIR/src"
|
|
git clone --depth 1 --branch "$WHISPER_CPP_TAG" "$WHISPER_CPP_SOURCE" "$INSTALL_DIR/src"
|
|
else
|
|
git -C "$INSTALL_DIR/src" fetch --depth 1 origin "$WHISPER_CPP_TAG"
|
|
git -C "$INSTALL_DIR/src" checkout FETCH_HEAD
|
|
fi
|
|
|
|
CMAKE_FLAGS="-DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF"
|
|
if [ "${GGML_CUDA:-0}" = "1" ]; then
|
|
CMAKE_FLAGS="$CMAKE_FLAGS -DGGML_CUDA=ON"
|
|
fi
|
|
|
|
# shellcheck disable=SC2086
|
|
cmake -S "$INSTALL_DIR/src" -B "$INSTALL_DIR/src/build" $CMAKE_FLAGS
|
|
NCPU="$(getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4)"
|
|
cmake --build "$INSTALL_DIR/src/build" --config Release --target whisper-server -j"$NCPU"
|
|
|
|
mkdir -p "$INSTALL_DIR/build/bin"
|
|
cp "$INSTALL_DIR/src/build/bin/whisper-server" "$INSTALL_DIR/build/bin/whisper-server"
|
|
|
|
echo "==> Installed $INSTALL_DIR/build/bin/whisper-server"
|
|
"$INSTALL_DIR/build/bin/whisper-server" --help >/dev/null 2>&1 && echo "==> Binary runs OK"
|