5 commits

Author	SHA1	Message	Date
Etherl	cd9162f49e	Studio: add audio.cpp as a native engine for speech, music and dictation (#12342) ... * Studio: add audio.cpp as a native engine for speech, music and dictation audio.cpp (0xShug0/audio.cpp) runs 20+ audio model families natively on ggml. This adds it as a fourth native runtime beside llama.cpp, whisper.cpp and sd.cpp and makes its models usable on every audio surface: the Audio page (speech, music, transcribe), chat dictation and audio-upload transcription, read-aloud, Settings > Voice, and the OpenAI /v1/audio/speech, /v1/audio/transcriptions and /v1/models routes. - Runtime: studio/install_audio_cpp_prebuilt.py installs a self-contained audiocpp_server bundle (audio.cpp vendors its own patched ggml) from the pinned Etherll/audio.cpp release, verified against in-repo sha256 pins, with an upstream fallback, CPU fallback, offline/no-op reuse, a staged start check, and the prebuilt_core install lock. setup.sh/setup.ps1 run it fail-open; uninstall, the desktop app, Docker and CI know the new runtime. - Models: one curated catalog (audio_cpp_models.py, mirrored by audio-cpp-catalog.ts) of TTS, music and ASR packages from audio-cpp/audio.cpp-gguf, addressed by virtual ids and pinned to a tested revision. - Speech and music load into the main audio slot through the native-audio worker (AudioCppBackend); speech-to-text is a fourth dictation engine, "audiocpp". Both run one audiocpp_server child with Studio's usual lifecycle: loopback port, scrubbed env, per-launch readiness id, idle unload, cancel, training pre-emption. - A model the installed runtime cannot run (missing runtime, no eSpeak-ng build, a Windows path over MAX_PATH) is refused with a clear reason before any download or eviction. * Studio audio.cpp: fix Windows cancel, proxied readiness and link-farm cleanup - Cancel returns at once on Windows: the request runs on its own thread, since shutting the socket from another thread never wakes a blocked recv there. - The readiness probe uses http.client, so an ambient HTTP_PROXY cannot route loopback traffic away from the server. - An install from the upstream fallback asks for the fork again instead of matching offline; the fork lookup had only failed on that run. - Link-farm prune and materialize never follow or write through a symlink or junction planted in the farm, and prune keeps the empty folders of a downloaded model that a concurrent materialize has just made. - Clearing the hub cache and deleting the umbrella repo from a non-active cache prune the farm beside that cache, so the hardlinked bytes are freed. - The download plan of a fully downloaded model answers from disk, so it works offline. - The installer sweeps staging dirs a killed run left, normalises the accelerator setup forwards, and runs the staged server with secrets scrubbed like every other child. - Register audio_cpp_server.py in the shutdown-latch completeness test. * Studio audio.cpp: prune the farm of the hub root a repo delete targeted The Hub row sends the repo folder as cache_path, not a hub root, so the prune looked for a farm that does not exist and the deleted blobs stayed on disk. Resolve the hub root with scoped_delete_root, as the delete itself does. * Studio audio.cpp: parse the fake server's multipart without cgi (removed in Python 3.13) * Studio audio.cpp: refuse pinned Linux bundles on glibc < 2.35 before downloading; keep a user-chosen tag on the chosen repo * Studio audio.cpp: auto mode installs the CPU build when the Linux CUDA bundle has no NCCL (no-torch Studio) * Studio audio.cpp: report an unwritable link-farm location as runtime unavailable * Studio audio.cpp: keep the child's eSpeak-ng data dir within its path buffer under a long Studio root * Studio audio.cpp: stop the STT server when a transcription is cancelled mid-request * Studio audio.cpp: refuse deleting the umbrella repo while one of its models is loading * Studio audio.cpp: refuse an overlong Windows model path in the runtime preflight, before download or eviction * Uninstall: keep an unmarked ~/.unsloth/audio.cpp, as for sd.cpp * Tests: count audio.cpp's keep arm in the setup status contract setup.sh and setup.ps1 now carry one "keeping the existing complete install" arm per runtime: llama.cpp, whisper.cpp and audio.cpp. * Tests: stub the NCCL probe in the --print-asset resolution test On a Linux runner without NCCL, auto mode now resolves the CPU build only, so the test followed the runner rather than the CUDA host it describes. * Studio: treat audio.cpp models as ordinary audio GGUFs audio.cpp models were a curated catalog with virtual ids, their own download path and "audio.cpp" labels. They now go through the same GGUF plumbing as llama.cpp models, and the runtime is chosen at load time. - Detection by GGUF header: general.architecture "audiocpp" plus the embedded audiocpp.model_spec.family classify a repo as speech, music or speech-to-text, so any compatible Hugging Face repo works, not only a fixed list. llama-server refuses these files, and the Audio page refuses a GGUF that is not an audio model. - A family table (task, server task, eSpeak need, defaults, options, packages) replaces the model catalog. Families outside Studio's audio features are refused with a reason. - Umbrella folders keep their audio-cpp/audio.cpp-gguf/<Folder> ids; dedicated repos use their own ids. Quant variants come from /gguf-variants and download through the normal GGUF download, so rows read "<Folder> · <quant>". MiniMax Music 3 and YuE2 package repos list their component mixes; deleting one mix keeps files another still uses. - Per-model advanced options come from the model's spec and are rendered on the Audio page, saved per model and sent as audio_options. - YuE2 follows the requested length; missing required music fields are a 400. Speech-to-text accepts the /gguf-variants keys and keeps the loaded variant when none is named. - The UI shows -GGUF names with no audio.cpp branding. * Studio: keep saved dictation keys working with audio GGUF ids Settings > Voice and dictation store the built-in models by their short keys and compare them to /stt/status, which now reported folder ids, so a downloaded model showed as missing and never as ready. Status lists the saved keys whose files are downloaded and reports a model under the name the client loaded it with. User-facing errors say "the audio runtime" instead of naming the engine. * Studio: report the audio runtime fields as absent for llama.cpp loads The load response gained audio_family and audio_options, and the llama.cpp runtime-field builder requires every response field, so every chat GGUF load failed with "GGUF backend is missing runtime response fields". llama server never serves an audio GGUF, so it reports both as None. The worker source check now reads the mirrored-keys tuple instead of one spelling of it. * Studio audio.cpp: read per-major CUDA bundles and use them only when torch or the system has that CUDA runtime * Studio audio.cpp: install the unslothai/audio.cpp prebuilts, pinned to v0.8.2-audio8-perf-hotfix-unsloth.1 * Tests: stub the CUDA runtime probe in the --print-asset resolution test * Studio audio.cpp: pin v0.9.0-unsloth.1, fix MiniMax Music 3 requests, keep transducer ASR off CUDA graphs, cap GPU host threads, offer Qwen3-TTS CustomVoice speakers * Studio Audio: readable capability line, voice names and clip labels for GGUF audio models * Studio Audio: trim GGUF audio catalog comments * Studio audio.cpp: refuse repo file names that climb out of the link farm, keep music GGUF rows out of Chat, key resolution cache by credential * Studio dictation: skip the gc pass when releasing an empty Whisper sidecar and the link-farm prune on warm audio.cpp requests * Studio audio.cpp: prune the link farm beside every remembered cache after a delete * Studio dictation: move an audio.cpp server training put on the CPU back to the GPU once training ends * Studio audio.cpp: key remote GGUF headers by credential, and treat missing eSpeak data as an incomplete install * Studio audio.cpp: count missing model files in refs/main, where downloads land * Tests: scan the fixture cache in the music-row chat test * Studio audio.cpp: an Auto load on a CPU-only runtime is a CPU placement * Studio audio.cpp: record a CPU-only runtime's Auto load as resident in CPU RAM * Studio audio.cpp: hand a CPU-only runtime's Auto load to the worker as a CPU load * Studio audio.cpp: report variant STT downloads by row, and relist downloads after one finishes * Studio audio.cpp: launch a custom build on a backend it was compiled with * Studio audio: YuE2 generates without lyrics, and a custom GGUF dictation repo skips the Whisper validator * Studio audio: remember the picked STT quant with its repo across restarts * Studio audio.cpp: trim comments * Studio audio.cpp: name the --version probe's encoding; teach main's contract tests the audio fields and variant downloads --------- Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>	2026-10-02 09:32:09 -07:00
Daniel Han	922e62297e	Keep the regenerable caches inside the Studio root (#10597) ... * Keep the regenerable caches inside the Studio root Unsloth's own data already lives under one root, but the caches it creates along the way do not. Torch inductor, Triton, matplotlib, numba and the CUDA JIT cache all land in the home directory, and DataDesigner builds ~/.data-designer off Path.home() at import time. This pins the regenerable, process-scoped ones in every mode. Part of #8865, which asks for an install contained in a single directory. This half needs no new flags and helps default installs too. Added to the defaults dict in _setup_cache_env, all under the existing cache_root(): TORCHINDUCTOR_CACHE_DIR (was /tmp/torchinductor_<user>), TRITON_HOME and TRITON_CACHE_DIR (~/.triton), TORCH_EXTENSIONS_DIR (~/.cache/torch_extensions), CUDA_CACHE_PATH (~/.nv/ComputeCache), MPLCONFIGDIR (~/.config/matplotlib and ~/.cache/matplotlib), NUMBA_CACHE_DIR (~/.cache/numba), and DATA_DESIGNER_HOME with DATA_DESIGNER_MANAGED_ASSETS_PATH (~/.data-designer). DATA_DESIGNER_HOME is read at import time by data_designer's constants module, so it has to be set before the Data Recipes worker imports the package. The workers are spawned, so they copy this environment and need no change. diffusion_compile_cache.py also stops hardcoding ~/.cache/unsloth/diffusion_compile_cache. It resolves through cache_root() per call now, and still reads a bundle from the old path when one is there, so nobody pays a cold compile for the move alone. Fixes ~/unsloth_compiled_cache. unsloth train, inference and export import the backend in-process and never seeded the cache environment, so they inherited unsloth-zoo's bare relative UNSLOTH_COMPILE_LOCATION and resolved it against whatever directory the shell was in. Running from the home directory left an ~/unsloth_compiled_cache that cache_cleanup then refused to remove, because nothing marked it as ours. Seeding from ensure_studio_backend_path() covers every command that reaches the backend, without making `unsloth --help` pay for importing it. What deliberately does not move: the Hugging Face hub cache stays at ~/.cache/huggingface. It was taken out of the Unsloth tree on purpose in #4591 so that models downloaded before Unsloth was installed, or shared with LM Studio, Ollama and plain huggingface_hub, keep being found instead of downloaded a second time. HF_HOME never moves in any mode either: it owns the token path, and credentials should not follow a cache onto a removable volume. * Tighten the comments this branch adds * Correct the runtime-root docstrings and note the matplotlib version gap * Install the managed runtimes at the master root and contain the modules cache * Point the whisper.cpp builder at the master root and harden the legacy cache probe * Normalize the master root the same way in both installer scripts * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Assert ownership of the runtime children a master root moves * Remove the master root's runtime children on uninstall * Keep a shared staging directory and stop locked master-root runtimes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove only a file from an install-lock path in a master root prebuilt_core.install_lock creates these locks with os.open(O_CREAT | O_EXCL) and writes a pid, so an install lock is always a regular file. The master-root cleanup reached them through _remove_path, which is rm -rf: in a user-chosen root a directory that happens to be named .node.install.lock was deleted whole, with none of the owner-marker proof the runtime children beside it require. The names are fixed and nothing else routes here, so the shape test costs nothing. A symlink is unlinked rather than followed, since -f is true for a link to a file and rm takes the link. The test grew a case for each shape and now names every helper it extracts: an extraction that stops matching went command-not-found inside a subshell the harness swallows, which reads as "nothing was removed" and quietly passes the keep-assertions. * Lift install.ps1's new master-root helper into the preflight harness Two Windows checks in this file were failing on the branch. Get-ManagedLlamaCppDir now calls Get-MasterRootOverride, which the harness's list of functions to copy into the child did not name. A helper that is not lifted does not announce itself: the child dies on the first call, every Write-Host after it is lost, and the two assertions read as though the preflight had resolved the wrong directory. So the list is now checked against the commands the lifted bodies actually call, and a missing one is reported by name. The ownership guard's flag moved into a parameter so the runtime children beside studio\ under a master root are covered even when the Studio home is the legacy one. The shape assertion still names the old variable; it now names the new one, plus the line that defaults it back to the script-level flag, which is what made the old assertion worth making. * Do not pin the compiler caches into a root that contains a space Four fixes, all reproduced against origin/main before and after. 1. torch.compile stopped working when the Studio root contains a space. torch/_inductor/cpp_builder.py joins its arguments with spaces, interpolates the output path unquoted, and reparses the result with shlex.split, so a cache directory containing a space splits mid-path: g++: fatal error: input file .../my is the same as output file Nothing here can fix that quoting, so this stops creating the situation: the five variables a C++ or CUDA toolchain receives on a command line are left unset when the path has whitespace, and Inductor goes back to its own whitespace-free temporary directory, which is what shipped before these were pinned. Everything else still moves. An explicit value from the caller is still honoured; only a default we invented is ours to withhold. Measured on origin/main and on this branch, same fixture: main compiles, the branch before this commit raises CppCompileError, the branch after it compiles. Not a rare shape -- "C:\Users\First Last" is an ordinary Windows account name, so this is the DEFAULT root for a large share of installs. 2. A missing master root ended the whole uninstall under sh. uninstall.sh runs with set -e and a bare assignment takes the exit status of its command substitution, so the canonicalizing cd failed the run whenever the root was already gone: a second uninstall, or a portable install on a disconnected drive. Ordinary bash hides this by clearing errexit inside command substitution; dash, sh and bash --posix do not, and the advertised one-liner pipes into sh on Debian, Ubuntu and WSL. setup.sh already guarded the same call. 3. setup.ps1's Node ownership guard did not cover a master root. The guard and the owner marker both hung off $NodeOverride, which only the UNSLOTH_STUDIO_HOME branch sets. The master-root branch sets $NodeParent and leaves it null, so <master>\node reached the whole-directory os.replace() in install_node_prebuilt.py with no ownership evidence, and the tree the run then created stayed unmarked, which makes the uninstaller keep it as somebody else's. setup.sh had already moved both sites. 4. uninstall.ps1 resolved a relative master root differently from setup. [IO.Path]::GetFullPath anchors at [Environment]::CurrentDirectory, which PowerShell does not keep in step with its own location. After a Set-Location a relative UNSLOTH_HOME named one install to setup and another here, and the owner marker does not help: it spares trees that are not Unsloth's, and the wrongly resolved path is another Unsloth install. Resolved through the provider path now, as Get-CanonicalDir does. Also: the PowerShell uninstaller gets uninstall.sh's install-lock rule, its stale-lock glob keeps the leading dot the rename actually produces, and both uninstallers apply the rule to the default root as well as a master one. tests/sh/test_with_llama_cpp_dir_link_behavior.sh had been failing on this branch: it seeds the variables the lifted block reads and never learned the runtime ownership flag, so the block died on an unbound variable under set -u and printed nothing, which read as seven wrong answers instead of one missing seed. Seeded, and it now says which it is. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stub git and cmake for the extracted whisper block, so its tests do not need a toolchain Four of this branch's own tests failed on a runner without cmake: AssertionError: ERROR: cmake is required They lift the root selection out of scripts/build_whisper_cpp.sh as shipped text, and that slice carries the builder's `command -v` preflight along with it, so the block exits 1 before choosing any directory and the tests fail for a reason unrelated to what they assert. They passed here because cmake is at /usr/bin/cmake and the fixture hardcoded PATH=/usr/bin:/bin. Stubbed rather than cut out of the slice: excising the preflight would mean the tests no longer run the shipped text, which is the only reason to lift it instead of restating it. Verified by running the extracted block directly under a PATH with no cmake, which reports the preflight error and names no directory, and under the same PATH with the stubs prepended, which resolves <root>/whisper.cpp. * Fix six review findings about the master root All six reproduce; each has a test that fails without the fix. 1. UNSLOTH_HOME=$HOME/.unsloth named the root a default install already uses, and the runtime paths did not move, but the ownership flag was set from the variable being present rather than from where the runtimes land. A legacy source-built ~/.unsloth/llama.cpp carries neither the owner marker nor prebuilt metadata, so the assertion rejected an update that used to reuse that build. Customness now comes from comparing the resolved root against the legacy one, canonicalised the same way so a symlinked $HOME does not compare unequal to itself. 2. setup.ps1's Node ownership guard and marker hung off $NodeOverride, which only the UNSLOTH_STUDIO_HOME branch sets, so <master>\node reached install_node_prebuilt.py's whole-directory os.replace() with no ownership evidence and the tree it created stayed unmarked. Both sites now consult the runtime ownership state, as llama.cpp and whisper.cpp already did. 3. _assert_studio_owned_or_absent tested -d, so a dangling symlink and a regular file both read as "nothing is here" and the caller rm -rf'd the path or replaced it. A dangling link is the ordinary case, its target volume merely unmounted, and following it later installs onto somebody's other disk. Every existing entry is now occupied unless it carries ownership, and the refusal says which shape it found. 4. The uninstaller's master-root child gate tested -e, which is false for a dangling symlink, so an unmarked one fell through to _remove_path, which treats -L as present and unlinks it. 5. A whitespace-only UNSLOTH_STUDIO_HOME or STUDIO_HOME is unset to every resolver, but a bare truthiness test called it present and suppressed the master-root branch. The path was then discarded, so an uninstall carrying that environment removed the master root's runtime siblings and left <UNSLOTH_HOME>/studio installed. Both uninstallers trim first. 6. setup.ps1 persists TORCHINDUCTOR_CACHE_DIR to the USER environment, so every later Studio process inherits it and _setup_cache_env's fill-if-unset default never applied on Windows: the containment this branch is for did not happen there, and with long paths off the cache sat at C:\tc even in portable mode. The persisted value now names what the resolver would have chosen. Long paths off still keeps the short drive-root directory for MAX_PATH headroom, and a path containing a space is refused for the same reason storage_roots refuses one. Two more, from the same round: The uninstaller could not find a master root that was named for a single command: `UNSLOTH_HOME=/mnt/portable unsloth studio update` installs the runtimes there and leaves nothing behind, so a later uninstall removed the Studio tree and stranded them. setup.sh now records the root in the Studio tree and the uninstaller reads it. The deny list and the marker gate still apply to whatever the note names, so it cannot license a removal the environment could not. install.sh and install.ps1 do not read UNSLOTH_HOME yet, so a machine with it merely exported had the CLI resolve <master>/studio while the installers put everything at the legacy root, and `unsloth studio` reported that Studio was not set up. The master root now wins only when it has an install; otherwise the one that exists does. When the installers learn the flag this fallback stops being reachable. test_setup_staged_root.sh matched the old spelling of the branch that chooses the compiler cache rather than what it does. It now checks that a staged run takes the stage root and is asked first, which is what that assertion was for. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record the Windows master root, and clear the cache path uninstall leaves behind setup.sh writes a .unsloth-master-root note in the Studio tree and uninstall.sh reads it, so an uninstall run without UNSLOTH_HOME still finds the runtimes that `UNSLOTH_HOME=... unsloth studio update` installed under a root named for one command. setup.ps1 never wrote that note and uninstall.ps1's _MasterRoot consulted only the current environment, so on Windows the same run removed <master>\studio and stranded node, llama.cpp and whisper.cpp beside it. Both halves are here now, matching the POSIX rules: only a master root is noted, never a staging one, the note is staged then renamed, and the first line alone is read. setup.ps1 also persists TORCHINDUCTOR_CACHE_DIR to the User environment, and this branch made that path install-specific. The uninstaller deleted the tree and left the variable, so every later PyTorch process on the account, including ones with nothing to do with Unsloth, compiled into the removed directory and rebuilt part of it. It is cleared when it names a root this run owned. A directory the user chose is theirs, and the shared C:\\tc fallback is not install-specific and is not deleted here either. * Two ways the diffusion migration raised instead of missing Both are pre-existing weaknesses this branch made reachable, so it should carry the fix. json.loads returns [] for "[]" and None for "null", and manifest.get() on either raises AttributeError out of a function whose entire contract is that a bad cache entry is a miss. The manifest being validated now comes from the pre-relocation root, written by an older build on a disk this run has never checked. Path.exists() raises rather than answering False when a parent denies traversal. The write root moved, so it can land on a directory the process does not own, and the unguarded probe left begin() before the legacy fallback could run: exactly the fallback the relocation depends on for a warm start after an upgrade. * Keep a matplotlibrc written under the managed root The legacy probe re-runs on every launch, so on its own it hands a matplotlibrc written under the managed MPLCONFIGDIR to a ~/.config/matplotlib created later by some other tool. Measured across four launches: figure.dpi 177 from the managed config, 222 once a legacy rc appeared, 177 again when it was removed, with the managed rc present throughout. A style that changes on a later launch and changes back is worse than either choice made once. _data_designer_defaults already applies exactly this rule to its own home, and the pre-existing rule is unchanged: an empty managed directory still defers, since _setup_cache_env creates it on the first launch and existence alone would drop a real user matplotlibrc. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop the workspace guard pinning text the master-root fix moved Two stale assertions, both caught by a full local run rather than by CI. The setup.sh half sliced a fixed 600 characters from _assert_studio_owned_or_absent. Adding the dangling-symlink check pushed the second assertion past the window, so the test failed while the guard it is about was intact. The PowerShell half beside it already slices to the end of the function and says why; the sh half now does too. The other assertion named the exact line `[ -z "$_MASTER_ROOT" ] || _RUNTIME_ROOT_IS_CUSTOM=true`, which is no longer the rule: UNSLOTH_HOME can name ~/.unsloth, the root a default install already uses, and treating that as custom made an update reject a legacy source-built llama.cpp carrying no owner marker. tests/test_setup_master_root.py runs the shipped derivation for that case, so this one holds the two halves being present rather than their wording. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Four more ways a generated path could be published or taken wrongly A blank inherited toolchain variable is now dropped rather than left blank when the generated default is refused for whitespace. Inductor distinguishes an absent TORCHINDUCTOR_CACHE_DIR from a present one, so " " became a relative compiler path and was then split by the very unquoted command construction the refusal exists to avoid. "Blank counts as unset" now holds on a spaced root too. A generated toolchain path that could not be made is withheld instead of exported. The placement error is swallowed on purpose so startup survives, which is what let a regular file at <studio>/cache/torchinductor, or a parent with a restrictive ACL, stay published: torch treats the value as authoritative and fails every compile, where an unset variable would have used its own temporary cache. Only these keys. UNSLOTH_COMPILE_LOCATION unset falls back to a bare relative name resolving against the CWD, which is the bug that pin exists to fix. setup.ps1's Node guard asked -PathType Container, so a regular file or a symlink named node under a user-selected master root was invisible to it and then displaced by _swap_into_place. Anything at the path counts as occupied now, and a non-directory is refused outright since ownership evidence lives inside one. Neither uninstaller will take a Studio root that is also the master root. The resolver accepts that flat layout, and a Studio root is removed whole once it carries the ownership marker, so anything else kept in that directory went with the install. Every other master-root child is individually marker-gated for exactly that reason; this was the one hole in the rule. Kept rather than pruned: data left behind is recoverable and printed, a deleted file is not. * A link at managed-assets is use, not the empty layout we create _setup_cache_env only ever makes a plain directory there, so a symlink or a Windows junction is the user sending their assets to another disk. entry.is_dir() follows it, and an empty target therefore read as the untouched layout Unsloth creates: with a standalone ~/.data-designer also present, DATA_DESIGNER_HOME was dropped and the redirect went with it. os.path.isjunction is 3.12, and the Studio venv can be 3.11, so its absence means there are no junctions to worry about rather than an error. * Four ways the master-root cleanup could miss its target or take someone else's The master root is resolved once, before anything is deleted. _master_root can read its answer from a note INSIDE a Studio tree, and the custom-root loop removes that tree, so asking again afterwards returned nothing and stranded the marked llama.cpp, Node and whisper.cpp siblings: the exact failure the note was added to prevent, reintroduced by asking too late. Windows already computed the value early for the stop pass and then re-probed; it reuses it now. The note probe trims the Studio overrides, as _custom_studio_roots and setup.sh both do. UNSLOTH_STUDIO_HOME=" /mnt/studio " puts the note at /mnt/studio/share, and appending to the padded value looked somewhere that does not exist, so the note was never found while the loop still removed /mnt/studio. The stale-lock sweep matches the installer's own shape rather than any dotted name. prebuilt_core.py leaves <component>.install.lock.stale.<pid>; the glob also matched a user's .backup.install.lock.stale.copy, and in a root they chose that file is theirs. Both sweeps, both scripts. And _nothing_at looks at the link before following it. os.scandir follows a symlink or junction at stylelib, so one pointing at an empty directory, or a dangling one whose volume is not mounted, read as an empty directory and let MPLCONFIGDIR move away from a redirect the user created. The Data Designer probe already checked the link itself. * Refuse a non-directory at a managed runtime path on Windows * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the CLI, the backend and both setup halves on one master root Four defects found by simulating the branch rather than reading it. The CLI keeps an installed legacy root when UNSLOTH_HOME names a directory with no install in it, because install.sh and install.ps1 do not read the variable yet. That fallback is not custom, and _ensure_studio_env_exported returned early on non-custom, so nothing was exported at all while storage_roots.studio_root() went on honouring UNSLOTH_HOME with no install check. The CLI ran the legacy venv and the backend under it wrote studio.db, auth and the pid file to <master>/studio. The root is now exported without being marked custom, so the ownership guards are untouched. setup.ps1 derived the runtime ownership flag from whether UNSLOTH_HOME was set, where setup.sh derives it from where the runtimes land. UNSLOTH_HOME pointed at the legacy root moves nothing, and calling it custom there demanded an owner marker from a markerless source-built .unsloth\\llama.cpp that predates the marker, so Assert-StudioOwnedOrAbsent exited an update the shell half adopts. uninstall.ps1 removed the master root and its .staging only when empty, but asked with Get-ChildItem -ErrorAction SilentlyContinue and then called _RemovePath, which is recursive: a swallowed enumeration error read as an empty directory, and a file arriving between the check and the delete went with it. Directory.Delete(path, false) is one call that refuses a non-empty directory, which is what uninstall.sh's rmdir does and what this file's own TEMP prune already does. The bash and PowerShell harnesses in test_pr10648_offline_and_status.py stand in for the setup section that derives these flags, and the guards inside the block they extract now read the runtime flag, so both harnesses have to define it. Nine tests there failed under set -u with 'unbound variable'; the file passes 102 on the merge base and 102 here. * Hold the preflight lift check to install.ps1's own helpers The check resolved every command the lifted bodies call, so main's new Get-SecuritySoftwareNote, which reaches Get-CimInstance and Get-MpPreference, read as two unlifted helpers on a Linux runner and as nothing on Windows. Ask install.ps1 whether it defines the name instead: a helper the lift leaves behind is still caught, on either platform. * Stage the master-root note under an unpredictable name The note setup writes so the uninstaller can find a root that only one command named was staged at "<share>/.unsloth-master-root.$$" and written with a shell redirection. Anyone who can write share/ could precreate that name as a symlink and have setup truncate whatever it points at, which is the shape that matters here: a master root is normally a volume, and a volume is the thing two accounts share. setup.sh states the rule for the uv cache probe 2400 lines earlier, "mktemp (O_EXCL) not $$"; the note follows it now, and setup.ps1 opens its staging file CreateNew under a random name rather than writing through WriteAllText. The rename already replaced a link at the final path rather than following it, so only the staging name was open. tests/test_setup_master_root.py plants the link from the same shell that then runs the shipped block, so the guessed pid matches the way an attacker's would. The PowerShell half is held structurally: the Linux runners have no Windows filesystem to plant on. * Treat a whitespace-only Studio home as unset when exporting it _resolve_studio_home strips before deciding, and so does storage_roots, so " " means unset to both. The export guard read the raw value, where " " is truthy, and left it in place. With a master root exported and an install at the legacy root, that split the two: this process ran the legacy venv while the backend under it resolved <UNSLOTH_HOME>/studio and wrote studio.db, auth and the pid file there. Blank was already covered; whitespace was the shape that got through. uninstall.ps1 had the same bug and this branch already fixed it there. UNSLOTH_LLAMA_CPP_PATH gets the same treatment: llama_cpp.py's discovery takes a whitespace value as a directory to scan, so leaving it costs the master root's build and hands the scan a path that cannot exist. * Migrate the legacy diffusion bundle through a temp file Every other write in this module goes to a temp file in the destination directory and is renamed into place, because a save runs on a daemon thread that interpreter exit can kill anywhere. The migration out of the legacy root used shutil.copyfile straight onto the live names, so an interrupted copy published a partial manifest and a second backend migrating the same key interleaved its writes into one destination file. A torn manifest reads as a miss, which costs exactly the cold compile the migration exists to avoid. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Point the matplotlib containment tests at each platform's config dir The helper hardcoded _get_config_or_cache_dir's Linux branch, so on a mac runner these nine tests wrote the rc into ~/.config/matplotlib, which macOS matplotlib never reads. The guard correctly found nothing to strand, pinned MPLCONFIGDIR, and the assertions failed for a reason that was in the tests rather than in the code they cover; test_matplotlib_reads_the_config_the_pin _would_have_hidden proved it by watching real matplotlib fall back to its built-in matplotlibrc. The derivation is held against real matplotlib on each spoofed platform by the two tests above it, so it cannot drift into agreeing with storage_roots.py alone. * Give the master-root probes an environment Python can start in _resolve and _discover build the child's environment from scratch, and the list left out what the interpreter itself needs: Windows python exits 1 before running a line when SYSTEMROOT is absent. check = True then raised CalledProcessError with the child's stderr discarded, so all nine tests reported as though the resolver had answered wrongly on every Windows runner. The environment now carries SYSTEMROOT and its neighbours, and a non-zero exit asserts with the child's stderr in the message, since a dead interpreter passing for a wrong answer is the failure that hid this. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write the llama-server fixture under the name this platform looks for _find_llama_server_binary appends .exe on Windows, so the fixture's POSIX-only name meant discovery correctly found nothing and the three discovery tests failed on a Windows runner for a reason that was in the fixture. * Skip the setup.sh blocks on Windows rather than run them under Git Bash Sixteen tests here execute blocks of the POSIX installer and were guarded by "is bash on PATH". Git Bash puts it there on a Windows runner, so they ran against Windows path semantics and UTF-16 output and failed sixteen times about a script that platform never uses: `assert '\x00' == 'ALLOWED'` is the shape of it. setup.ps1 is held separately in this file and under tests/studio, so the guard now asks for bash on a POSIX filesystem and the Windows half of a cross-platform run reports only what it really covers. * Probe the cache dirs for writability, keep linked lock paths, harden the Node guard Three defects found in review. storage_roots._usable_dir accepted any existing directory. The mkdir above it passes exist_ok = False and swallows every OSError, so a directory that exists but cannot be written to arrived here having proven nothing, and the value was published anyway. torch treats these variables as authoritative: cache_dir() and triton_cache_dir() fall back to their own temporary directory only when the variable is UNSET, then call os.makedirs(exist_ok = True), which succeeds on an existing read-only directory and leaves every later compile write to fail (torch/_inductor/runtime/cache_dir_utils.py). Probe with a real create via tempfile.mkstemp and unlink it, so an unwritable directory is withheld and torch keeps its own fallback. scripts/uninstall.sh removed a symlink sitting at an install-lock path, so a lock name planted as a link to something outside the install took that link with it. uninstall.ps1 already excludes reparse points at the same names. Keep a link, and keep any other non-file, reporting both. studio/setup.ps1 probed the Node directory with Test-Path -PathType Container. Under Windows PowerShell 5.1 that answers for the link TARGET, so a dangling `node` link read as absent and install_node_prebuilt.py then installed straight through it. Probe with Get-Item -Force, as Assert-StudioOwnedOrAbsent already does, and treat a reparse point as not-a-directory however it resolves. Tests: a read-only managed cache is withheld while NUMBA_CACHE_DIR stays pinned, the write probe leaves nothing behind, a linked and a dangling-linked lock are both kept, and the two Node-guard structure tests now hold the stronger invariant. The sh suite's assert_link uses -L so a dangling link does not read as gone. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Recover the master root from the note setup already writes UNSLOTH_HOME is documented as settable for a single command, `UNSLOTH_HOME=/mnt/portable unsloth studio update`, and that command installs node, llama.cpp and whisper.cpp as SIBLINGS of studio/ under the root it names. Nothing persisted the variable, so once the command's environment went away unsloth_home() returned None and all three resolvers looked somewhere else. Measured against a real master-root tree: later launch, bare node -> ~/.unsloth/node (absent) later launch, UNSLOTH_STUDIO_HOME set node -> <master>/studio/node (absent) what setup installed node -> <master>/node The second row is the one that matters: even with the studio root known correctly, the runtimes resolved to its studio/ CHILD while the installer had put them beside it. whisper.cpp and the llama.cpp path pinned into every worker follow the same call. setup.sh and setup.ps1 already record the root at share/.unsloth-master-root, for the uninstallers. Read it here too, so the process agrees with the install on disk. One fallback in unsloth_home() covers all three resolvers, since each of them keys on it. Two guards keep a note from doing harm. The recorded root must exist, and its studio/ child must be the very directory the note was read from, so a note left behind by a tree that has since moved, or copied into an unrelated install, is ignored rather than adopted for caches and runtimes both. The candidate studio roots are derived without calling studio_root(), which asks unsloth_home() where the studio tree is and would recurse. Tests: the recorded root outlives the command that set it, a moved root and a copied note are both ignored, an empty note is not a root, and a default install still reads none. Each one fails with the fallback removed or a guard weakened. * Read the master-root note from this install's Studio root only Two installs on one box, one legacy and one named by UNSLOTH_STUDIO_HOME. Every note reader walked a list and took the first readable note, with the legacy tree first, so a note belonging to the OTHER install won while the rest of the run stayed on the named one. In storage_roots that split the process against itself: studio_root() returned the named tree while unsloth_home() returned the other install's master root, sending node, whisper.cpp, llama.cpp, the portable caches and orphan cleanup into a tree Studio was not running from. In both uninstallers it was worse than wrong, because a master root licenses removals: the run deleted the named Studio and then followed the other install's root and took its marker-owned runtime children with it. All three now consult exactly one Studio root, the one studio_root() itself would answer with the master root out of the picture: UNSLOTH_STUDIO_HOME, then STUDIO_HOME, then the legacy tree, and no falling through from one to the next. An explicit override names that exact directory, so its absence of a note is an answer, not a reason to keep looking. Also moves this file's three pwsh call sites onto run_pwsh, which tests/studio/ test_pwsh_calls_use_the_shared_runner.py requires: a direct spawn shares one startup cache with every other xdist worker and dies with `Stack overflow.` at a few percent per process. Tests: an unrelated install's note is not adopted and the named root's own note still is, in Python, in sh and in PowerShell, each running the shipped function. Restoring the legacy-first order fails all three. * Require a master-root note to describe the tree it was found in A Studio tree copied from master root A to B carries a note still naming A. Every reader took that at face value, and in the uninstallers it licenses removals: A's llama.cpp, node, whisper.cpp and sd.cpp carry exactly the owner markers B's would, so the marker gates cannot tell them apart and uninstalling the copy deleted the ORIGINAL install's runtimes. All three readers now require the Studio directory a note was read from to lie INSIDE the root it names. Containment rather than an exact <root>/studio match, including in storage_roots, which had the stricter rule: the flat layout, UNSLOTH_HOME and UNSLOTH_STUDIO_HOME naming one directory, is supported and an exact match refuses it. A Studio root deliberately placed outside its master root is not recovered from a note, which storage_roots already calls not self-contained and warns about on every resolve. An explicit UNSLOTH_HOME skips the check, being the user speaking rather than a file found on disk. Both uninstallers also read the default-mode studio.conf when no override names a Studio root. A master-root install puts Studio at <master>/studio, so with a bare environment that conf is the only thing naming the directory the note lives in, and the legacy path could never satisfy containment. Read inline rather than through _custom_studio_roots / _CustomStudioRoots, which call back into this function. unsloth_cli exports the recovered root rather than only reading it. setup runs as a subprocess, so an update with the root in a note but not the environment refreshed the runtimes at <master>/studio/ while the backend, which does read the note, kept launching the stale ones at <master>/. The CLI keeps its own aligned copy of the reader, as it does for studio_root(); tests/test_unsloth_home_root_agreement.py is what holds the two together. Tests: a copied note is refused in Python, in sh and in PowerShell, a flat recorded root is still honoured, and the CLI both recovers a real note and refuses a carried-in one. Four sh fixtures described a master root with no Studio tree under it, which cannot occur; they now describe the real layout. Every case fails with its guard removed. * Give the CLI the same note validation as the backend Relaxing the note check to containment in storage_roots and both uninstallers left the CLI's copy on the stricter <master>/studio match. UNSLOTH_HOME=/root with UNSLOTH_STUDIO_HOME=/root/custom/studio is a real pairing: setup records /root, the Studio directory is inside it, and it is not the default child. The backend accepts that note and the CLI declined it, so a later `unsloth studio update` passed setup no master root at all and refreshed the runtimes under the Studio directory while the backend went on loading the ones at /root. That is the same split this recovery exists to close, reopened one level up. Test: the CLI and the backend agree on a nested Studio root, in tests/test_unsloth_home_root_agreement.py. It fails with the strict rule restored. * Only record a master root a reader will honour Two shapes of the note were being written and then refused by every reader, so the feature recorded nothing while the file on disk said otherwise. Both were found by running the shipped setup.sh block against a fixture and handing the result to the real resolver, not by reading. Containment. setup.sh writes the note at $STUDIO_HOME/share, and STUDIO_HOME is derived only from UNSLOTH_STUDIO_HOME/STUDIO_HOME, never from UNSLOTH_HOME. Every reader requires the Studio directory to lie inside the root the note names. install.sh does not read UNSLOTH_HOME yet, so the documented case, UNSLOTH_HOME=/mnt/portable unsloth studio update on an ordinary install, leaves Studio at ~/.unsloth/studio and puts llama.cpp, node and whisper.cpp at /mnt/portable, which is exactly the shape all four readers reject. The note is now skipped there and the stranding is reported with the two ways out, instead of being recorded in a file nobody reads. The legacy default root. UNSLOTH_HOME=$HOME/.unsloth names the directory the install is already in. Both uninstallers refuse that value outright, but storage_roots honoured it, and honouring it made portable_mode() true on every later bare launch: HF_HUB_CACHE moved off ~/.cache/huggingface, which is the one cache this branch promises not to move, and every model already downloaded went missing. One command that named the default root converted the install permanently, with nothing set at the time. It is no longer recorded, and a note an earlier build left carrying it is cleared on the next run, since the writer only runs when UNSLOTH_HOME is set again and the user who hit this set it once. Also in storage_roots: - A note read that fails for a reason other than absence is no longer cached. _recorded_master_root memoises per studio path and the backend is long-lived, so an EACCES on share/ or an EIO off a mount that came back pinned that process to the legacy runtime paths for its whole lifetime, silently, with the trees it should have found beside it. Nothing in production calls forget_recorded_master_root. A genuine miss is still cached. - _usable_dir guards os.close like it already guards the unlink. Every other failure there is a False; an OSError escaping took _setup_cache_env, and the backend start that called it, down with it. - The UNSLOTH_PORTABLE warning no longer says 0/false/off/no will leave portable mode off. An off value is not a veto: a master root is what makes an install portable, as the comment three lines below already said. Tests: five new cases over the gate, the legacy-root sweep, the cached-note rule and the close guard, plus the setup.ps1 halves held structurally. 205 in test_setup_cache_env_containment and the cache suites, 356 across the master root, root agreement, workspace guard and system32 guard suites, 6710 in tests/studio/install, and all five tests/sh scripts pass; tests/studio/install matches the merge base exactly. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Prove a studio.db is absent before skipping the read, on Windows too The skip added here reads os.stat(studio_db_path()) and treats FileNotFoundError as "no database, nothing to read". On POSIX that is sound: a parent component that is a file raises NotADirectoryError and one that cannot be traversed raises PermissionError, each its own type. Windows collapses all of them into ERROR_PATH_NOT_FOUND, which Python surfaces as FileNotFoundError, so there the exception means "absent, OR something above it is not a directory, OR I could not look" and the guard took the first reading. The result was the cache home chosen in Settings being discarded on Windows and nowhere else. This is the branch's own new test failing, not a theory: test_uninspectable_studio_db_keeps_the_stored_cache_home[not_a_directory] is green on Linux and macOS and red on windows-latest. Measured as a differential on owned staging runners, base against head, same sixteen files, same three operating systems: ubuntu-latest has no real failure on either arm, macos-15 has the same five on both, and windows-latest has 82 on the base and 83 on the head. The one extra is that test. Nothing else the branch touches moved on any of the three. _absence_is_real answers the question the skip actually needs: the nearest ancestor that can be stat'ed decides. A directory means the tree is real and the file genuinely is not in it. Anything else means a component is a file or a reparse point, so the path could never have existed and the caller has learned nothing. An ancestor that cannot be inspected at all is not proof either way. Running out of ancestors means nothing on the path exists, which is the machine that has never opened Studio, and the skip has to hold there or the read builds the 250 KB database it was added to avoid. Checking only the immediate parent is not enough: account_path puts studio.db several levels under the Studio home, so the non-directory is usually further up. Held by a test that reproduces the Windows error shape rather than the platform, since the Linux runners cannot raise it, plus its opposite, an absent Studio home that must still skip. The first fails on the code as it was and passes after, so it measures the fix rather than restating it. 207 pass in the cache suites, 365 with 1 skipped across the master root, root agreement, workspace guard and system32 guard suites, 6754 with 32 skipped in tests/studio/install, 2 in the CLI export suite. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop the module-level errno import the probe does not need errno is used only inside _WINDOWS_SHAPED_GUARD_PROBE, which is a string run in a subprocess and imports it itself. The hoisted import was unused at module level and the source lint blocks on exactly that. * Decline a recorded legacy default root in the reader, not only the writer Found by enumerating the resolver's input space rather than reading it: 512 cells over UNSLOTH_STUDIO_HOME x STUDIO_HOME x UNSLOTH_HOME x UNSLOTH_PORTABLE x the recorded note, base against head, one subprocess per cell. In 12 of them the Hugging Face hub cache still moved off ~/.cache/huggingface with nothing set in the process at all. The write side was already gated: setup.sh and setup.ps1 refuse to record $HOME/.unsloth and sweep an old note away. But that only fires when setup runs again, and the user who hit this set UNSLOTH_HOME once, for one command. Until the reader declines it too the repair never reaches them, and both uninstallers were meanwhile refusing the same value outright, so the four readers the branch claims to keep aligned were not. _is_legacy_default_root declines it in storage_roots, and the CLI copy gets the identical rule, which is what test_unsloth_home_root_agreement.py holds. Recorded roots only: unsloth_home() returns an explicit UNSLOTH_HOME before it ever reads the note, and narrowing that would change what the variable means rather than what a stale file is allowed to claim. Re-run over the same 512 cells: root resolution 480 identical and 32 changed, which is the new UNSLOTH_HOME feature, with 0 violations; HF_HOME moves in 0 cells; of the 192 cells with no UNSLOTH_HOME and no UNSLOTH_PORTABLE, the caches move in 30 and all 30 carry a real recorded master root. Legacy-root notes: 12 before, 0 now. Foreign notes: 0 throughout. Also closes a gap in this file's own tests. Reverting the production hunks and keeping the tests left 25 of 113 containment tests green, because each asserts only that some variable is ABSENT, which is equally true when no pinning code exists: assertions, not negative controls. Three new cases pair each decline with the pins that must still happen in the same run, so the family fails when the resolver does nothing. All five new tests fail with the production hunks reverted and pass here. 119 pass in the containment suite, 147 across it plus the root agreement and managed tools suites. * Refuse a multi-line master-root note in both uninstallers The comment beside each read said "One line, first only: a note that grew a second line is not one we wrote", and neither did that. uninstall.sh took `head -n 1` and carried on; uninstall.ps1 took `Get-Content -TotalCount 1`. The second line was never looked for. That is the four readers disagreeing in the one place it costs something. storage_roots.py and unsloth_cli/commands/studio.py both read_text().strip() the WHOLE file, so a two-line note is not a directory to them and they decline it. The two uninstallers accepted line 1 and went on to authorise removing <master>/llama.cpp, node, whisper.cpp and stable-diffusion.cpp for a master root no runtime reader had agreed to. Measured on the fixture, all four readers, one note: storage_roots.unsloth_home() -> None CLI _recorded_master_root() -> None uninstall.sh _master_root() -> /<master> <- authorises the delete uninstall.ps1 _MasterRoot -> same shape Both now refuse when a second non-blank line is present, which is what the comments already promised. A trailing newline on an ordinary note is not a second line, and the one-line note keeps working -- both are asserted, so the refusal cannot pass by reading nothing. Found by the release-safety fan-out over the install and uninstall paths, then reproduced against the shipped helpers rather than taken on report. tests/sh/test_uninstall_master_root.sh grows three cases and runs the real extracted block: 45 pass here, and the two-line case fails against uninstall.sh exactly as submitted in this PR. The PowerShell half is held structurally, as the rest of that file's Windows coverage is. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not rewrite the master-root note when it already says this Move-Item -Force is not an atomic replace. PowerShell's filesystem provider deletes the destination and then moves the source, so an interruption between the two leaves NO note at all, while the complete staging file sits beside it unreferenced. A backend that looks next reads "no master root" and, since the miss is cached, keeps that answer for its whole lifetime. The window cannot be removed under Windows PowerShell 5.1, which has no atomic replace primitive this code can reach. What can be removed is re-entering it on every single update to write bytes that are already there, which is every run after the first. The note is compared first and the write skipped when it matches, and the staging file is now cleared in a finally rather than left behind when the move throws. Two tests, and the second one took two attempts to make honest: the first version compared LastWriteTime across two runs a millisecond apart, which can land on one filesystem tick, and it passed against the unfixed writer. It back-dates the note instead, so any rewrite is unmistakable, and then changes the value to prove the writer still writes when it should. Both fail against setup.ps1 exactly as submitted in this PR and pass here. Found by the release-safety fan-out over idempotency and interrupted runs, then reproduced against the shipped block under pwsh rather than taken on report. 43 pass in test_setup_master_root, 176 across it plus the containment, root agreement and managed tools suites, and 45 in tests/sh/test_uninstall_master_root.sh. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not read a Windows path error as proof the user's config is absent _nothing_at's docstring is explicit that Path.exists and is_dir must not be used because they report ENOTDIR and ELOOP as absence and the redirect would then hide the user's files. It then took FileNotFoundError at face value, which is the same reading restricted to one platform: POSIX distinguishes "not there" from "a parent component is a file" (NotADirectoryError) and "I may not traverse it" (PermissionError), while Windows collapses all three into ERROR_PATH_NOT_FOUND. So with a file where ~/.matplotlib belongs, POSIX declined the pin and Windows took it, moving MPLCONFIGDIR for the whole process and hiding whatever was underneath. Same defect and same cause as the studio.db guard fixed in 37f23c576, in a second probe, and _nothing_above is the same rule: the nearest ancestor that can be stat'ed decides, a directory means real absence, anything else means the path could never have existed, an uninspectable ancestor proves nothing, and no ancestors at all is the default install and is real absence. Found by the windows-latest staging leg, on a negative control written for the POSIX shape and added earlier in this review. base arm 82 real failures, head 83, the one extra being that test. It is the only new failure on any of the three operating systems. Held by a test that reproduces the error shape rather than the platform, since the Linux runners cannot raise it, with both opposite cases asserted so the fix cannot pass by declining everything: a genuinely empty present directory still reads as empty, and a path with nothing along it is still absence. It fails against storage_roots.py exactly as submitted in this PR. 584 pass with 1 skipped across the eleven suites this branch touches. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comment prose around the master-root resolvers * Give the cache-pin probes one table each, and the PowerShell probes one helper * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse a master-root note that names the legacy tree, and stop Node discovery seeding caches Three fixes found while auditing what happens to an existing install. A note left at <studio>/share/.unsloth-master-root that names the legacy tree itself passed every containment check, so a bare environment resolved a master root and the uninstaller then treated a second marked install under it as a root to remove. Both uninstallers and both setup scripts now decline a note whose directory is the legacy Studio dir. A staged update derived the runtime ownership flag from the master root, so a staging pass could mark runtimes custom when they were still legacy. The flag is now assigned only when no stage root is in play, and follows where the runtimes actually land. Node tool discovery imported the Studio backend path purely to find the bundled tools and built the whole compiler cache tree as a side effect. The import now takes a seed_cache_env flag and the discovery path opts out. Each fix has a regression test checked against the unfixed code. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <moonshotaisubstack@gmail.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-09-20 06:20:41 -07:00
Daniel Han	25c10a30dd	Trim comments in the studiobench harness and unsloth_cli (#10118) ... * Trim comments in the studiobench harness and unsloth_cli Comment-only pass over 170 changed files under tests/studio/studiobench, unsloth_cli and the root CLI entry points. 12,952 -> 7,893 comment lines (-39.1%) 962,064 -> 619,497 comment characters (-35.6%) Distinctive-fact retention against the base commit: 93.2%. A sample of 100 restoration and deletion decisions was judged by hand: 90 correct, and the 10 reversals were fixed rather than only reported. No comment line exceeds 120 physical characters. Nothing outside comments changes. Verified with an AST code-only signature of every file against the base commit, plus node --check on every JS and TS file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reattach comments that drifted onto unrelated code in the trimming pass Repair the relocated and truncated comments the review flagged: the 1,866 ms worst frame belongs to the reasoning_toggle clause in both from_payload.py and streamcost.js, the Hermes providers.* note belongs to the Hermes config test rather than the OpenCode one, and the detect_gguf_model conclusion belongs above the construct it concludes. Restore the per-constant notes for SURFACE_SWEEP_S and WATCHDOG_MARGIN, repair the v8 measurement sentence in streamcost.js, and drop the orphaned tails left behind when comments were reflowed. * Keep the phrase the A/B home-collision test asserts on tests/studio/studiobench/runtime/selftest/test_studiobench_ab_home_collision.py slices __main__.py from 'ONE HOME CANNOT HOLD TWO BUILDS' and asserts 'overwrites the first' appears in the next 1400 characters. The trim rewrote that clause away, so the assertion failed in Backend CI. Restored the clause, which also repairs the sentence: the trimmed version read 'two arms sharing one serve whichever build was installed last', dropping the home and the checkout it shares. 6 tests pass. * Put the section banners back The trim removed the rule-line comments that index these files. They are not decoration: tests locate a region by finding its banner and end it on the next one, so worker.py losing its whole numbered index broke test_rocm_support.py, test_grouped_mm_rdna4_fallback.py and test_audio_decode_load_order.py while the code-identity gate stayed green. Each banner is restored above the code line it preceded at 1d2679f. Where that anchor is not unique in the current file the banner is left out rather than guessed at, so this under-restores by design. * Repair the banner restore: drop duplicate headings, keep the blank line The scripted restore re-added each section banner without removing the de-bannered plain heading the trim had left behind, so a heading appeared twice, once ruled and once bare. It also inserted the banner flush against the next line of code where the base had a blank line after it, which turns a section separator into a doc comment for whichever def follows. Both are the same fault the trim itself was being screened for, introduced by the remediation rather than by the trim. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <michaelhan2050@gmail.com>	2026-09-02 03:21:16 -07:00
Michael Han	05cee801bb	Use Unsloth for bare Studio product references (#9732) ... * Studio: use Unsloth for product shorthand * Tests: restore case-folding fixture path * Complete Unsloth product reference rename * Tests: restore internal Studio references	2026-08-26 01:55:49 -07:00
Daniel Han	715535d1c5	Windows: start the backend from a usable folder on login autostart (#8575) ... * Windows: start the backend from a usable folder on login autostart "Run Unsloth at login" registers the desktop through an HKCU Run value, which cannot carry a working directory, so Windows starts the app in C:\Windows\system32. Every `unsloth` CLI child inherited that folder, and the CLI refuses to run there, so a reboot produced a tray icon and no server (issue #8510). Desktop side: pick the working directory explicitly for every CLI child (backend, both preflight probes, the install check, auth provisioning, update, installer) instead of passing on whatever the launcher gave us. The inherited folder is kept whenever it is usable, so ./models and other cwd-relative defaults resolve exactly where they used to; only a Windows system folder is replaced, with ~/.unsloth. A home that cannot be reached at all now reports working_directory_unavailable rather than looking like a broken install, which stops the pointless automatic repair and gets its own message in the UI. CLI side: move the System32 guard into unsloth_cli/_system_dir_guard.py and run it before the command modules import, since commands.studio resolves STUDIO_HOME at import time. The commands the desktop itself issues take no path from the user, so they move to a safe folder and carry on; everything else keeps the hard error, since relocating a command would silently rebase the relative paths its caller typed. This half fixes anyone whose installed desktop build predates the change, without waiting for a desktop release. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only trust a Windows directory that holds System32, and tell an absent profile from an absent install Two review findings on the working-directory work. WINDIR is an ordinary variable, so pointing it at the user's own profile made windows_roots() treat that profile as a Windows installation: an ordinary project folder underneath it then looked like a system folder, the fallback rejected the home for being "inside the Windows directory", and the backend could not start anywhere on that machine. Candidates are now checked rather than trusted, and a directory only counts if it actually contains System32. With nothing on the machine looking like Windows, the check falls back to SystemRoot or the default, never to the settable value. Same fix on both sides, since the CLI guard reads the same variables. The managed install lives under the user's profile, so a profile that is not mounted yet makes find_unsloth_binary() return None and preflight reported NotInstalled before the working-directory check could run. That is the exact case the check was added for, and it was sending those users to reinstall. Check whether the home is reachable before turning a failed lookup into "not installed", and report it as its own state with no binary path and no repair offered. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments added by this PR * Keep the preflight message choice testable The branch that tells an unreachable profile from an outdated install sat inline in use-tauri-backend.ts, which pulls in React and the Tauri APIs and so cannot be imported from a test. Move the choice into its own module and drive it directly, so a new backend reason cannot silently land in the stale-install bucket again. * Pin relative path overrides before relocating out of a system folder Studio resolves UNSLOTH_STUDIO_HOME and the cache overrides with Path.resolve(), which anchors a relative value to the working directory. So `UNSLOTH_STUDIO_HOME=.\custom unsloth studio update` from System32 moved first and then resolved the override against the new directory, silently targeting a folder the caller never named. That is the one thing this guard is supposed not to do to caller-supplied paths. Absolutise the relative overrides against the original directory before the move, so they keep meaning what they meant. A ~ value is left alone, since expanduser does not consult the working directory, and an environment that cannot be pinned is one we refuse to move underneath. Covers the Studio home pair plus the cache and llama.cpp overrides, which resolve the same way: UNSLOTH_LLAMA_CPP_PATH, UNSLOTH_COMPILE_LOCATION, HF_HOME, HF_HUB_CACHE, HUGGINGFACE_HUB_CACHE, HF_XET_CACHE. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin every relative path override and withhold repair on an unreachable profile - pin_relative_overrides() missed the Studio documents, projects and sandbox roots and the sd.cpp/whisper.cpp/llama.cpp engine paths, so a relative value would have been retargeted by the move, and the remaining single-path cache overrides are pinned for the same reason - an owned or ownerless-spawned backend that is stale no longer offers auto repair when the managed profile is unreachable: the repair runs through the same profile and stops a backend that still answers * Pin relative overrides on the desktop side too, and align the home checks - the desktop moved a CLI child out of a system folder without rewriting the relative path overrides it inherited, so the same install placed state in a different folder depending on whether the desktop or the CLI guard did the move; both layers now anchor those values to the directory being left, with a test that fails if the two lists drift apart - pin the diffusion cache dirs, OLLAMA_MODELS, DG_VISUAL_BIN and UNSLOTH_DG_SHIM, which are resolved against the working directory as well - home_dir_available() accepted a home the working directory resolver then rejected, so a SYSTEM account was offered an install that cannot start; both now go through usable_home_dir() * Report an unreachable profile as the reason a stale backend cannot repair Withholding auto repair was not enough: the result still carried the backend's own reason, so the frontend advised running the update, which needs the same profile the probe could not reach. Both stale paths now report working_directory_unavailable, and the backend's reason goes to the log. * Resolve drive-relative overrides through the OS before moving A value such as HF_HOME=D:cache names the current directory on drive D, so joining it to the folder being left hands it straight back and the move retargets it. Both layers now ask the OS to resolve it first, GetFullPathNameW through ntpath.abspath and std::path::absolute, which is what tracks each drive's own directory. The CLI guard refuses to move at all if that resolution fails, rather than moving and silently changing where the value points. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the stable-diffusion binary overrides and the rest of the cache family SD_CLI_PATH and SD_SERVER_PATH are the highest-priority binary locations for the sd.cpp engine, so a relative value pointed somewhere else after the move. Added alongside the llama.cpp and whisper.cpp equivalents, together with the HF and XDG names that belong to the same families as the ones already pinned. * Pin the GPU SDK roots, and do not relocate into a missing or shared profile - CUDA_PATH, HIP_PATH, HIP_PATH_57 and ROCM_PATH are joined with bin/ for DLL discovery, so a relative value pointed elsewhere after the move - a profile that has not mounted yet still has a writable parent, so makedirs built an empty second profile that would shadow the real one when it arrives; the guard now requires the home to exist, as the Rust resolver does - the public profile is refused whichever variable named it: allow_public only kept PUBLIC out of the candidate list, so a USERPROFILE or ~ that resolves there still put one account's state in a folder shared by every account * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments added since the last pass * Cover running the working-directory fix more than once Both halves run repeatedly over state an earlier run touched: the desktop resolves the directory on every spawn, the guard runs in every CLI process, and a child's environment reaches its own grandchildren. Two tests pin the fixpoint, one for a command configured twice and one for a value that has already been anchored. * Pin STUDIO_LOCAL_REPO, and blame a lost profile for the probe that failed - "studio update" is one of the commands that relocates, and it resolves a relative STUDIO_LOCAL_REPO against the working directory, so the move retargeted the checkout the user meant to install from - a profile can drop between the working directory check and the probes that follow, which reported cli_unusable or desktop_capability_probe_failed and offered a repair needing that same profile; both now ask again and report the profile when it is what went missing * Narrow the desktop marker, and anchor the values a move could still retarget Five independent reviews of the branch agreed on the same three gaps: - the marker is inherited by the backend and everything below it, so treating it as authorisation for any "studio" subcommand let a marked descendant relocate "studio run --model .\local.gguf", rebasing a path the caller chose. It now authorises only invocations that carry no path, which is what the desktop actually runs - a value like "\cache" is rooted to the drive of the current directory, not to a drive, so a profile on another drive moved it. Root-relative values now go through the OS with the drive-relative ones, and the extended prefix is matched case-insensitively, since the object manager accepts \\?\unc\ too - four more single-path overrides are pinned, and two path lists are anchored entry by entry, so one relative entry cannot change what a whole search or allowlist means The desktop also relocated a child out of any folder under the Windows tree, while the CLI only ever refused System32 and SysWOW64. It now uses the same definition, so a child running from somewhere like C:\Windows\Temp keeps the directory it had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Offer the roaming-profile cause only where roaming profiles exist A platform-isolation audit found this PR's working-directory-unavailable message asserting a Windows cause on every platform: Unsloth cannot reach your user folder, so it has nowhere to run from. This usually means a network or roaming profile is not available yet. The Rust half of this PR is deliberately not #[cfg]-gated; isolation comes from windows_roots() returning empty off Windows, which does hold for every env-var rewrite. But home_dir_available() is called ungated from preflight/managed.rs, so ManagedProbe::Unavailable{working_directory_unavailable} is reachable on Linux and macOS, where the same symptom means an unmounted home or a permissions problem. A roaming profile is a Windows concept and naming it there sends the reader looking for something that is not present. The symptom and the remedy are unchanged everywhere. Only the CAUSE moves behind a platform check, so it is offered where it applies instead of asserted everywhere. Test covers both directions: Windows still gets the sentence, Linux and macOS do not, and both keep the symptom and "Reconnect and try again". studio/frontend/tests/backend-preflight-message.test.ts 4 passed * Do not let the interpreter decide which folder a value names The Windows cross-platform CI caught this: ntpath.isabs answered True for a leading separator until Python 3.13 and False after it, so on 3.12 a root-relative "\cache" was treated as fully qualified and left to move with the working directory, which is the retargeting the pinning exists to prevent. Both halves now spell the test out, a drive plus a separator or a UNC share, so the same value names the same folder on every interpreter and on the Linux runner that tests the Rust half. * Run the update smoke tests when the CLI guard changes The guard runs on every CLI invocation, "unsloth studio update" included, but neither update workflow listed it, so the two suites that install Unsloth and then update it twice never ran for this change. * Do not rewrite an override the working directory never resolved Three of the pinned names are read as something other than a path by the code that consumes them, so anchoring one changed its meaning instead of preserving it. MLX_HOSTFILE holds either a filename or the host list itself, huggingface_hub expands %VAR% in HF_HOME and its neighbours after the guard has run, and the pre-quant allowlist ignores a bare on/off token precisely so that there is no allow-all mode: anchoring the "1" would have turned it into a real allowlisted directory. All three are now left alone, in the CLI guard and in the desktop twin. Also in this pass: - studio --frontend=.\dist carries a path inside the option token, which the marker gate missed because it only looked for a bare argument after the subcommand. A marked child running it is refused rather than moved. - \Windows\System32\config\systemprofile names SYSTEM's profile without a drive, so it compared equal to no drive-qualified Windows root and was accepted as a home. The drive-less spelling of each Windows root is compared too. - The child is only told where to run when that differs from where the parent already is. Reopening an inherited directory by name can fail if an ancestor turned unreadable after launch, where inheriting the open handle would have worked, so the no-move case stays exactly as it was. - An override the OS declines to resolve now refuses the whole move on the desktop side rather than being dropped, which is what the CLI guard already did: moving with that value still relative would retarget it. - The tests that read the ambient environment take the crate-wide env lock; XDG_DATA_HOME is one of the pinned names now, and the test that swaps it was documented as the only reader. * Scope the non-path exemptions, and anchor the import roots too Three follow-ups on the pinning: The exemptions for inline JSON, %VAR% / $VAR and bare on/off tokens now name the variables whose reader proves them, rather than applying to every pinned name. A directory really called [llama] or %data% is legal on Windows and UNSLOTH_LLAMA_CPP_PATH is read as exactly that, so the blanket form left it unpinned and the move retargeted it. PYTHONPATH joins the anchored search lists, and the guard anchors the relative entries this interpreter is already carrying in sys.path: those are resolved on every import, not at startup, so a move would let whatever sits in ~/.unsloth shadow a managed import. sys.path is best effort, unlike the environment: an import root is not worth refusing the move over, and refusing is how the login start broke to begin with. PATH is deliberately left out, being mostly other people's absolute entries. Preflight asks for the whole managed context, not just the directory. An override the OS declines to resolve fails the same spawn, and a probe that returned false for it was read as a broken CLI, which started an automatic repair that needed the same context and failed the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expand a cache override before deciding, and read the real invocation Three more from the review: huggingface_hub expands %VAR% in HF_HOME and its neighbours, but Studio's own hf_cache_settings._canonical() does not, so leaving such a value as written sent the two readers to different folders once the process moved. The guard now expands those names before deciding, writes the expanded form back when expanding is what made it name a folder, and anchors it when it does not. A name the machine does not set stays as written, which is what expandvars does too. The desktop got the same, with a small %VAR% expander since std has none. The updater removes PYTHONPATH on Windows, and pinning an inherited relative one put it straight back. The removal now happens after the managed context is applied: -I only covers the first interpreter, and the update's PowerShell and setup descendants start further Python processes that do not clear it. The Typer callback classified sys.argv even when the app was called as a library, so a host whose own argv looked like a desktop command could move the process out from under the caller's relative paths. It reads the invocation Click is running instead, and where Click keeps the tail to itself the invocation is refused rather than relocated. The console script is unaffected: it is classified at import, from the real argv. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Take the whole invocation, and the spellings that follow the process Four more from the review: os.path.expandvars takes %NAME%, $NAME and ${NAME} on a Windows path; the desktop expander only took the first, so a value written $LOCALAPPDATA\hf was anchored under System32 by the desktop while the CLI guard would have kept it. All three forms now, with a test that walks each one. The Typer group records the tokens it was handed. Click keeps the tail on the child context, so the callback saw only the subcommand name and refused a library `studio --api-only` that it should have relocated. Reading the recorded list classifies the invocation in full, whether it arrives through app(args = [...]) or a runner. HF_TOKEN_PATH is pinned like the caches beside it: huggingface_hub reads the credential file from there, and a relative value would follow the child and lose access to gated repos while everything still looked healthy. PYTHONPATH has two spellings that follow the process rather than the caller: an empty component means the working directory itself, and `~` is never expanded there, so Python reads ~\plugins as an ordinary relative folder. Both are anchored to the directory being left, in the environment, in sys.path and in the desktop twin. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only anchor a sys.path entry that is a directory, and name the setting Two findings from a fresh review round. sys.path holds other people's strings as well as folders. setuptools registers a relative sentinel for an editable namespace install and its own path hook accepts that sentinel by exact equality, and a relative .zip keeps the spelling its already imported packages hold in their loaders; rewriting either breaks the import the pinning was meant to protect. Both were reproduced: an editable namespace stopped importing after the rewrite, and a package loaded from a relative archive lost its submodules. Only an entry that is a directory right now is anchored, plus the empty entry, which is the working directory by definition. A value that cannot be pinned now says so. Windows caps an environment variable at 32767 characters, so a long enough list can stop fitting once every entry names its folder in full, and a drive with no current directory of its own cannot be resolved either. Both were reported as "check that the user profile for this account exists and is writable", which sends the reader looking in the wrong place. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expand exactly as ntpath does, and say which context failed The desktop expander read $NAME as far as the first non-word character, where ntpath counts a hyphen as part of the name. With CACHE and CACHE-ROOT both set, HF_HOME=$CACHE-ROOT\hf resolved to C:\right\hf in the CLI and C:\wrong-ROOT\hf in the desktop: one install, two folders. It now mirrors ntpath's pattern outright, including the single-quoted run that is copied through unexpanded, %% and $$ standing for one character, and anything unterminated staying as written. The guard's own tests call os.path.expandvars against the environment they describe rather than a stand-in that only knew %NAME%. Preflight tells its two context failures apart. A probe that could not be configured returns that instead of false, so a context which recovers between the failed apply and the check afterwards can no longer make an untested CLI look broken and start a repair. And an override the OS cannot resolve is reported as path_setting_unresolvable rather than as an unreachable user folder: the profile is fine, the value is not, and the frontend now says so instead of advising a reconnect. * Never panic on the spawn path, and leave a host's state as it was Five more from the review. The backend spawn unwrapped the managed context, so a drive that went between the preflight check and the spawn took the desktop down with it. It reports through the same diagnostics path as every other start failure now. UNSLOTH_STUDIO_HOME and STUDIO_HOME are removed for every managed child, because Tauri uses the legacy root whatever the environment says. Trying to resolve them could only invent a failure for a value the child never sees, so they are skipped when pinning and removed by the context helper itself rather than only at the call sites. A child was moved with its relative overrides untouched when the original directory could not be read at all. That silently retargets each of them at the new directory, so it is refused unless there is nothing relative left to preserve. The Typer callback no longer relocates. It runs after the command modules are imported, and commands.studio resolves STUDIO_HOME at import time, so a host that reached that point cannot be moved without leaving the cached root behind. The console script is unaffected: it is checked before any command module loads. The environment and sys.path are put back if the move does not happen. Inside a host process both belong to the caller, and a chdir that fails after pinning left them rewritten as though it had succeeded. * Write a ~ value out, and pin the uv cache Two more from the review round. A `~` value was left alone on the grounds that expanduser does not consult the working directory. That is true of expanduser and false of the readers: llama_cpp.py hands UNSLOTH_LLAMA_CPP_PATH and LLAMA_SERVER_PATH straight to Path(), as the whisper and stable diffusion overrides do with theirs, so `~\llama.cpp` was an ordinary relative path for them and followed the child to the new directory. It is written out now, on both sides, which is what the caller meant and what the readers that do call expanduser would have computed for themselves. That also covers the PYTHONPATH case more honestly than anchoring it did: `~` names the profile rather than a folder called "~" beside the old directory. UV_CACHE_DIR joins the pinned names. uv reads it as written, Studio treats a non-blank value as authoritative, and `unsloth studio update` runs uv through setup.ps1, so a relative one moved the install cache. The guard's tests and the simulations now call the real expanduser against the environment each case describes; the stand-in returned the profile for every input, which is what hid this. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Judge a list entry by entry, and keep the checkout out of the children Three more from the review. The lost-directory check read a whole path list as one value, so PYTHONPATH=C:\vendor;plugins looked qualified because of the drive at the front while `plugins` still depended on the directory that was gone. Each entry is judged on its own now, with the empty PYTHONPATH component counting as the directory itself. MLX_IBV_DEVICES is pinned beside MLX_HOSTFILE and exempted from anchoring the same way. `_json_rank_count_from_env` reads the two identically: either the device list inline as JSON, or a filename. STUDIO_LOCAL_REPO is read by the update and installer path alone (install_python_stack.py), so a stale drive-relative value was failing preflight, backend startup, capability probes and auth provisioning over a setting none of them look at. It stays pinned for the update child and is dropped for the rest. * Anchor an import root that is really there, and read /var/cache as absolute Two more from the review. The lost-directory check judged every value by Windows rules, and it is the one part of the pinning that runs off Windows: a desktop on Linux or macOS whose launch directory was deleted read XDG_CACHE_HOME=/var/cache as relative and failed every managed spawn with path_setting_unresolvable over a value that depends on no directory at all. The native spelling counts there too now. sys.path entries are anchored when they name something on disk rather than only a directory. Skipping every non-directory kept setuptools' editable sentinel safe, which is what it was for, but it also left a relative importable archive behind: os.environ is not sys.path, so after the move the next import from that archive looked for it beside the new working directory. A folder or an archive is anchored; a string that names nothing on disk is still left exactly as written. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read a POSIX path list with POSIX rules, separator included The lost-directory fallback is the one part of the pinning that also runs off Windows, and absoluteness there was already fixed. The separator was not: the list was still split on ';', so "/opt/vendor:plugins" stayed a single entry, started with '/', and passed as absolute. The relative entry behind it was never seen, and the child got it resolved against the wrong directory. Deciding both from a parameter rather than cfg! is what makes this testable. The existing entry-by-entry test feeds a Windows list and runs on every platform, so a compile-time cfg would either break it off Windows or leave the POSIX path untested. Now each test names the rules it means, and the POSIX one also pins that a Windows-shaped value stays Windows-judged. Supersedes the narrower POSIX test added alongside the absoluteness fix: same fixture, and the new one additionally covers the separator and the Windows-stays-Windows direction. * Normalize overrides before refusing a lost-directory move When the launch directory is gone, the pin scan judged each override exactly as written, without the expansion and non-path exemptions the moving path applies. HF_HOME=%LOCALAPPDATA%\hf, an inline JSON MLX_HOSTFILE and UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1 were all read as relative, so preflight and every managed spawn failed with path_setting_unresolvable over values that never depended on a directory. The same normalization now runs before the check. * Shorten the comments added by this branch Same intent, fewer lines: the reasons that are not obvious from the code stay, the restatements of it go. No code or behaviour change. * Pin the model paths llama-server reads for itself llama-server takes LLAMA_ARG_MODEL, LLAMA_ARG_MMPROJ and the two draft-model spellings straight from the environment and resolves a relative one against its own working directory, and Studio reads them back when it sizes a launch (llama_cpp.py). A managed child moved out of System32 with one of those still relative would look for the model or the projector beneath ~/.unsloth. Both mirrored lists now carry them. The URL and HF-repo spellings stay out: they name no local file. * Settle an expansion before anchoring it, and restore the real sys.path Two defects found by a fresh round of idempotency review. A value whose expansion needs a second pass, LOCALAPPDATA holding %USERPROFILE% and HF_HOME holding %LOCALAPPDATA%, was anchored while still half expanded, so the reader that expands saw a folder name with a second drive in the middle of it. Expansion now runs to a fixpoint, and a value that never settles, HF_HOME holding itself, is left exactly as written rather than anchored or grown. Both layers, with a test each. The console script passes no list, so the guard pinned the real sys.path with no snapshot to put back: a chdir that then failed left the process carrying import roots it never agreed to, while the environment was restored. The snapshot is now taken from the list actually being pinned. The existing rollback test passed its own list and so missed the one path production takes; the harness can now leave it out. * Never let a lost directory fail a spawn, and let the installer build its profile Two success-to-failure changes found by the same review round. A process whose working directory has been deleted or unmounted can still spawn children from the handle it holds. Pinning cannot anchor anything to a directory it cannot name, and the answer was to refuse, which took the capability probe, the backend start, the auth provision and the update down over a setting the command may never read. The pins still report what a move would lose; that report now decides whether the child moves rather than whether it runs, so it stays where it is, exactly as it did before this file learned about working directories. The installer shared the managed resolver, which requires the home to exist so that a child never builds an empty folder shadowing a roaming profile that has not mounted yet. install.ps1 and install.sh detect a SYSTEM profile themselves, and before they shared the resolver a home that did not exist was simply created along with ~/.unsloth, so they get that policy back. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the pinning decision as a table * Expand a value the way its reader does, and never leave the process elsewhere Four findings from a fresh round of review, all in code this branch added. Expansion now runs exactly one pass, which is what every reader runs, and the result is written back only when expanding it again would change nothing. The fixpoint loop that was here consumed an escaped %%NAME%% on the first pass and then expanded what the escape was protecting, and a nested reference came out half expanded. Both are now left exactly as written, alongside the self-referencing case they were added for. Python never expands ~ in PYTHONPATH, so neither does the pinning: `~\plugins` is anchored as the relative folder the interpreter actually reads rather than turned into a profile folder it was never reading, which also stops an inert entry from becoming an importable one. Click reads `-f.\dist` as a value exactly as it reads `--frontend=.\dist`, so an invocation carrying one is refused rather than rebased under the new folder. A chdir can succeed and still land somewhere the guard refuses, through a junction or a profile inside the Windows tree. The process now goes back where it started, so the values written for the move are put back with it instead of being restored under a directory nobody chose. * Resolve a tilde the way ntpath does, and pin two more paths More from the same review round, all cross-layer. The two layers expanded a tilde and a variable in opposite orders, so a value that names a folder through both reached different folders. Rust now does the tilde first and the variables second, as the CLI guard does. Rust also resolved ~someone-else as a sibling of this profile unconditionally. ntpath declines to guess unless the profile is named after the current user, because C:\Users\me.DOMAIN is not me's sibling, and now so does this. AMDGPU_ASIC_ID_TABLE_PATH and VLLM_CACHE_ROOT are read straight from the environment as file paths, by import_fixes.py and by Studio when the caller set one themselves, so both lists carry them. The update no longer pins PYTHONPATH on Windows, where build_update_command drops it anyway: pinning it could only refuse an update over a value the child never receives. * Never refuse an update over a setting it drops, and name the one that blocks Two review items, both about what a failure costs. STUDIO_LOCAL_REPO is now pinned best effort in both layers. A bare `unsloth studio update` drops it before anything reads it, and `update --local` is refused rather than relocated, so a stale drive-relative value could only refuse the one update form the System32 fallback exists for. It is still written out whenever it can be, so a reader that appears later still finds the folder the caller meant; every other setting still stops the move, because something does read those. A context failure now carries the setting that caused it, as `path_setting_unresolvable:HF_HOME`, and the window names it. "One of Unsloth's folder settings" is not something anyone can act on, and every pin failure already names the setting it could not preserve. The name only, never the value, since this reaches the screen. * Refuse the move when one pass leaves the folder up to the caller Leaving an unsettled expansion as written and moving anyway takes the value with the process: the reader resolves what one pass gives it against whatever directory it is standing in, so a nested %NESTED%, an escaped %%NAME%% or a self-reference quietly followed the child to ~/.unsloth. One pass is still what the reader does, so it is still what decides. If that result names a folder on its own, C:\\cache\\%UNSET%\\assets, the value means the same thing from anywhere and is left alone. If it does not, the setting cannot be preserved across a move and the move is refused, naming it. Both layers, a test each. * Read a path list the way the host does, and report what will not fit Four more from review, all in the moving path. The list loop split and rejoined on ';' and judged absoluteness by Windows rules, while the lost-directory branch had already learned the native ones. A POSIX PYTHONPATH of plugins:/opt/vendor was therefore one entry, and both import roots left with it. Both loops now read the host's separator and the host's idea of absolute. A pinned list that no longer fits in a Windows variable is now reported as the setting that did not fit, rather than discovered by CreateProcess, where the window would have offered a repair that hits the same wall. The tilde now follows USERPROFILE, which is what ntpath.expanduser answers and what the CLI guard uses. dirs::home_dir() reads the known folder, which a portable or overridden environment moves, and the two layers have to name the same folder. GGML_BACKEND_PATH joins both lists: llama_cpp.py preserves it into the llama.cpp child, which resolves it against wherever it is standing. * Expand a POSIX value the POSIX way, and check a scalar against the limit too The lost-directory branch is the one part of the pinning that runs off Windows, and it still read %HOME%/hf as an expandable reference. posixpath.expandvars leaves that literal and expands $HOME instead, so a value that depends on where the process is standing was read as one that does not, and the child moved out from under it. Expansion now follows the host: $NAME and ${NAME} off Windows, %NAME% and the rest on it. The oversized check covered the joined list but not a scalar, which crosses the same limit once it names its folder in full. Same check, same message, both layers. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-13 06:32:58 -07:00