* fix: add AutoRound format detection to prevent misclassification as GGUF
- Add ModelFormat::Autoround enum variant in models.rs
- Update is_prequantized() to include AutoRound
- Fix bit-per-parameter calculation for AutoRound-4bit
- Update scrape_hf_models.py to detect auto-round quant_method
- Correct 3 misclassified AutoRound models in hf_models.json
- Add AutoRound-4bit to SUPPORTED_QUANTS and parse_quant() in plan.rs
- Update pre-quantization comments in fit.rs
- Update test_model_format_prequantized() assertion
Fixes#526
* style: apply cargo fmt formatting
* ci: use npx for web build/test steps to fix Windows PATH issue
* fix: add AutoRound-8bit to quant metrics in models.rs
Add explicit entries for AutoRound-8bit in quant_speed_multiplier,
quant_bytes_per_param, and quant_quality_penalty. Without these,
an 8-bit AutoRound model would incorrectly fall through to the
4-bit defaults (0.5 bpp, quality penalty -5.0).
* chore: Add Qwen3.5 Claude Opus 4.6 reasoning models
* feat: Support parallel models update
* fix: Fix cursor navigate in tui
* feat: Support parallel models enriching
* feat: Support marquee animation for model name column auto if text too long
* chore: Ignore __pycache__
* chore: Remove hf_models backup
* chore: Remove duplicate Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled-GGUF in scrape hf models
Add four RWKV7 G1f series models from shoumenchougou GGUF repos.
These are pure RNN/SSM models with no KV cache, 8192 context,
and full llama.cpp compatibility. Architecture "rwkv" is already
handled in models.rs as pure SSM.
Closes#515
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- New scripts/scrape_benchmarks.py scrapes all 27 hardware presets from
localmaxxing.com and writes data/benchmark_cache.json (+ llmfit-core copy)
- Cache is embedded at compile time via include_str! and parsed lazily
- TUI falls back to cached data when the API is unreachable, showing
"Using cached data (API unreachable)" instead of an empty error
- Initial cache contains 115 results across RTX 5090/4090/3090,
RX 7900 XTX/XT, and Apple M4 Max
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add DeepSeek-V4-Pro, V4-Pro-Base, V4-Flash, and V4-Flash-Base to the
model database. Also fix MoE detection to handle the n_routed_experts
config key used by DeepSeek V4.
Closes#500
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(python): scaffold in-repo PyPI packaging
Copied hatch_build.py, src/__init__.py, src/__main__.py, and tests/ from
the external llmfit-pypi repo into llmfit-python/. Created pyproject.toml
at the repo root (adapted from llmfit-pypi) with updated author, paths, and
non-standard source mapping for the gitignore-safe llmfit-python/src/ layout.
Created .github/workflows/pypi.yml to build and publish all 8 platform wheels
on tag push, mirroring the docker.yml trigger pattern.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(python): rewrite hatch_build.py to use in-repo binaries
hatch_build.py previously downloaded release binaries from GitHub at
wheel-build time, verifying SHA256 and fetching license via the GitHub
API (~267 lines). It now expects the pre-compiled binary to already
exist at target/<rust-target>/release/ and reads version and license
directly from Cargo.toml (~174 lines, no network calls).
This enables the new CI split in release.yml: each build matrix job
uploads its binary as a binary-<target> artifact, and a dedicated
publish-python job (ubuntu-latest only) downloads all of them and runs
uv build in a loop over platform tags before publishing to PyPI in a
single step. Wheel failures no longer block binary release assets.
Also removes the now-redundant standalone pypi.yml workflow, adds
license = "MIT" to Cargo.toml workspace.package, and relocates the
Python package source into src/llmfit/ to match hatch layout.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: add Python gitignore entries
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: add Python code quality Makefile targets
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(python): respect LLMFIT_PYTHON_PLATFORM_TAG in editable installs
When the env var is set, look in target/{rust_target}/release/ (via the
renamed _find_binary_for_target) rather than the default host-native
paths (_find_local_binary). Also splits py_target_from_env / py_target
in initialize so the editable branch reuses the already-read value.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* ci: add Python CI workflow and tighten Rust CI path filters
- Add python-ci.yml with cross-platform pytest job (gated on llmfit-python/
or pyproject.toml changes via dorny/paths-filter) and a lint/typecheck job
that also triggers on scripts/**/*.py changes
- Exclude llmfit-python/** from Rust CI path triggers
- Replace cryptic Cargo.*o* glob with explicit Cargo.{lock,toml}
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: remove committed .pyc file from tracking
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: normalise ruff lint exception comment formatting
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Ruff formatted these lines into one line
* Fix import ordering
* Move pyproject.toml and uv.lock into llmfit-python/
* Correct path to tested code
* Run checks without installing llmfit itself (which fails)
* chore: bump setup-python and setup-uv to latest versions
Upgrades actions/setup-python v5 → v6 and astral-sh/setup-uv v5 → v8.1.0
to resolve Node.js 20 deprecation warnings in CI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Disallow setting LLMFIT_PYTHON_PLATFORM_TAG on an editable build
* Improve inline comments in binary selection logic
* Move binary version check out of selection logic
* Always check that binary file exists
* Check all binaries before building wheels
* Set the execute bit that was lost by GitHub Actions
* Python CI downloads latest release binary instead of rebuilding Rust
Replaces the cargo build steps in the test job with a download of the
latest release archive from GitHub Releases, verified via SHA256.
LLMFIT_VERSION is set to the downloaded release version so the wheel
build does not conflict with an unreleased Cargo.toml bump.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* publish-python: use release file downloads instead of GHA artifacts
Removes the binary artifact upload/download mechanism. publish-python
now depends on the release job (not build) and downloads each archive
directly from GitHub Releases, verifying its checksum inline via pipe
before extracting the binary and building the wheel.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix typo in ci.yml comment: worfklows -> workflows
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Accept optional v prefix in binary --version output
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Simplify BinaryNotFoundError to inherit only from LlmfitError
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Add fmt-check target; use in check instead of fmt
Avoids `make check` unexpectedly modifying files on disk.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Split up checksum file to work around * in sha file on Windows
* Fix Windows zip extraction in Python CI workflow
Use unzip instead of tar to extract .zip archives on Windows, since
Git Bash's tar does not support the zip format.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Make llmfit binary version visible in CI tests
* Prevent multiline echo-on-Windows weirdness
* Consolidate CI workflows and add path-based job gating
Merges python-ci.yml into ci.yml. Adds a changes job using
dorny/paths-filter to detect whether Rust or Python files changed,
then gates fmt/clippy/check/check-python jobs accordingly via
job-level if conditions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Tell uv where the python project is
* Remove outdated sentence about LLMFIT_VERSION from docstring
* Run Python tests with debug binary in editable mode for max reuse with cargo test
* Control the working dir to control test discovery
* Run Python build system integration test when there are no Python changes
* Update comment
* Always run cargo build on macos
* Show contents of target/ (for debugging)
* Debug production of debug builds by cargo test
* Always run cargo build to get a debug build
* Only build debug binary for subsequent pytest
* Revert "Debug production of debug builds by cargo test"
This reverts commit c88158e5ae.
* Revert "Show contents of target/ (for debugging)"
This reverts commit 947cb39988.
* Update comment with corrected description of work reuse
* Move prerequisite debug build step above uv sync step
* fix(ci): use picomatch extglob to exclude llmfit-python from rust filter
The previous config used a leading-! entry (`!llmfit-python/**`) to try
to exclude the Python directory from the rust filter. In dorny/paths-filter's
default 'some' mode, each entry is OR-ed: a leading ! is passed to picomatch
as a pattern that matches anything *not* in llmfit-python/, but the positive
`llmfit-*/**` entry already matched, so the negation had no effect and
rust-changed was always true for Python-only PRs.
Replace the two entries with a single picomatch extglob pattern
`llmfit-!(python)/**`. The !(python) extglob is evaluated against the
segment between `llmfit-` and the path separator, so llmfit-python/ never
matches while all other llmfit-* directories do.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(python): add rust_integration pytest mark for Rust-triggered CI runs
Marks test_binary_runs with @pytest.mark.rust_integration so it runs
in CI when only Rust code has changed, replacing the uv sync --no-dev
integration test step.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Add Qwen3.6-27B (dense, 27.8B) and Qwen3.6-35B-A3B (MoE, 36B/3B active)
to the curated model list with GGUF sources from unsloth and ggml-org.
Add hybrid attention layout inference for Qwen3.5/3.6 architectures.
Full re-scrape with --discover -n 2000 brings total to 962 models with
422 GGUF-sourced entries.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fixes#445
- Maverick num_experts was 16 (from generic llama4 fallback), now 128
matching the model name (17B-128E)
- Maverick context_length was 4096 (bare max_position_embeddings),
now 1048576 after applying RoPE scaling factor
- MODELS.md listed both Scout/Maverick as 128k; corrected to 10M/1M
- Scraper detect_moe() now checks text_config for nested MoE fields
(Llama 4 multimodal models store num_local_experts there)
- Scraper infer_context_length() now applies rope_scaling factor when
present, so models using RoPE extension get correct context values
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements all five phases of the roadmap in #417, requested in #411.
Phase 1 (#418): replace the linear KV approximation in models.rs with
a precise per layer formula
kv_bytes = 2 * n_layers * n_kv_heads * head_dim * ctx * dtype_bytes
when num_hidden_layers and head_dim are known on LlmModel. Falls back
to the old linear formula when metadata is missing so the existing
catalog doesn't regress. New fields num_hidden_layers, head_dim, and
attention_layout added to LlmModel.
Phase 1b (#418): scrape num_hidden_layers, num_attention_heads,
num_key_value_heads, head_dim, and hidden_size from each model's
config.json during llmfit update. Cache version bumped to 2 so old
caches re-fetch. Python scraper updated to write the same fields.
Phase 2 (#419): new KvQuant enum (fp16 / fp8 / q8_0 / q4_0 / tq) with
parse() and bytes_per_element(). New --kv-quant flag on plan, new
PlanRequest.kv_quant, new PlanEstimate.kv_quant, plumbed through
evaluate_current and build_path_estimate. CLI prints the active KV in
the plan header. JSON output exposes kv_quant and kv_alternatives with
snake_case labels matching the flag values.
Phase 3 (#420): new AttentionLayout { full, linear } struct with
compressible_fraction(). infer_attention_layout_from_name covers
Qwen3-Next, Jamba, Zamba, Mamba/RWKV; defaults to all-full for plain
dense transformers. effective_attention_layout prefers explicit
metadata, falls back to the heuristic.
Phase 4 (#421): TurboQuant variant gated to GpuBackend::Cuda in
estimate_model_plan; clear error message points at 0xSero/turboquant
and suggests fp8/q8_0/q4_0 for llama.cpp users. CLI prints a warning
on every --kv-quant tq invocation. KvQuantAlternative.note for the TQ
row always says "Experimental: not in upstream vLLM" plus the layer
split when known. The compression only applies to the full attention
slice; linear layers stay at fp16, so hybrid models see honest savings
(~21% on a 10/40 model, not the headline 83%).
Phase 5 (#422): new KvQuantAlternative rows on PlanEstimate showing
per-quant kv_cache_gb, memory_required_gb, savings_fraction, note,
and supported. Rendered as a "KV Cache Alternatives" table in the CLI
plan output and in the TUI plan view (highlighted active row, n/a
marker for unsupported options, per-row notes).
TUI: PlanField gains a KvQuant variant; new plan_kv_quant_input state,
input handling, and Tab cycle position; KV Quant input row in the
plan editor; Active KV line and KV Cache Alternatives section in the
rendered plan output.
Tests: 16 new unit tests covering KvQuant::parse, the precise formula
matching a hand calc on Llama-3.1-8B, KV scaling across fp16/q8/q4,
fallback when metadata is missing, TurboQuant only compressing full
attention layers, honest hybrid savings landing around 21% (not the
83% headline), AttentionLayout::compressible_fraction, name based
hybrid detection, plan errors on TQ + non-CUDA, plan succeeds on TQ
+ CUDA, and TQ row marked unsupported in alternatives off CUDA. All
301 workspace tests pass; no new clippy warnings.
Closes#418#419#420#421#422
Refs #417#411
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add MiniMaxAI/MiniMax-M2.7 to curated models list and manual entries
- Add M2.7 active parameter count for MoE calculations
- Add M2.7 entries to both data/hf_models.json and llmfit-core/data/hf_models.json
- M2.7 is placed before M2.5 as the latest flagship model
- All previous MiniMax models (M2, M2.1, M2.5) retained as alternatives
Add support for Docker Desktop's built-in Model Runner as a fourth
runtime provider alongside Ollama, llama.cpp, and MLX. Detection probes
the OpenAI-compatible /v1/models endpoint on localhost:12434 (configurable
via DOCKER_MODEL_RUNNER_HOST). Downloads use `docker model pull`.
A new scraper (scripts/scrape_docker_models.py) queries Docker Hub's ai/
namespace and cross-references against the HF model database to produce
an embedded catalog (docker_models.json) of confirmed available models.
Only models verified in the catalog appear as downloadable via Docker.
- Provider: detect, list installed, pull via docker CLI
- TUI: status bar shows Docker availability, 'D' in Inst column,
provider picker includes Docker Model Runner
- Inst column refactored from enum to bitfield for extensibility
- Makefile: `make update-catalogs` refreshes all scrapers and rebuilds
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_gguf_source_deserialization — GgufSource JSON round-trips correctly
- test_gguf_sources_default_to_empty — models without gguf_sources in JSON default to []
- test_catalog_popular_models_have_gguf_sources — 5 well-known models (Llama-3.3-70B, Qwen2.5-7B, etc.)
have non-empty gguf_sources in the catalog
- test_catalog_gguf_sources_have_valid_repos — every gguf_source in the catalog has owner/repo format,
non-empty provider, and contains GGUF
- test_catalog_has_significant_gguf_coverage — at least 25% of catalog models have GGUF sources (currently
30%)
providers.rs (7 tests):
- test_hf_name_to_gguf_candidates_generates_common_patterns — heuristic generates bartowski, ggml-org,
TheBloke candidates
- test_hf_name_to_gguf_candidates_strips_owner — strips the Org/ prefix correctly
- test_lookup_gguf_repo_known_mappings — hardcoded mappings resolve for known models
- test_lookup_gguf_repo_unknown_returns_none — unknown models return None
- test_has_gguf_mapping_matches_known_models — boolean check works
- test_gguf_candidates_fallback_covers_major_providers — fallback covers all 3 providers and all end in
-GGUF
- test_gguf_candidates_known_mapping_returns_single — hardcoded mapping returns exactly 1 result
Signed-off-by: AlexsJones <alexsimonjones@gmail.com>
Add ModelFormat enum (Gguf/Awq/Gptq/Mlx/Safetensors) and Vllm inference
runtime. Pre-quantized models are detected from config.json by the scraper
and filtered to CUDA/ROCm only (no Apple Silicon support yet). Dynamic
re-quantization is skipped for fixed-precision formats.
Co-Authored-By: Claude <noreply@anthropic.com>
Incorporates PR #97 by @lichengzhe with merge conflicts resolved.
Adds: DeepSeek V3.2/Speciale, GLM-5, Kimi-K2.5, MiniMax-M2.5,
MiMo-V2-Flash, MiMo-7B-RL, Nemotron-3-Nano, Nemotron-Nano-9B-v2,
Phi-4-reasoning/mini/multimodal, EXAONE-4.0 (32B/1.2B), SmolLM3-3B,
Gemma-3n (E4B/E2B), Qwen3-Coder-Next.
Includes MoE configs, active parameter mappings, Ollama tag mappings,
and comprehensive fallback entries. Resolves conflicts with existing
Qwen 3.5 entries on main.
Co-authored-by: lichengzhe <lichengzhe@users.noreply.github.com>
Signed-off-by: Three Foxes (in a Trenchcoat) <threefoxes53235@gmail.com>
Signed-off-by: Three Foxes (in a Trenchcoat) <threefoxesyes3inatrenchcoat@gmail.com>
Three improvements to GGUF source enrichment:
1. TUI detail view: Added '── GGUF Downloads ──' section showing
providers with 📦 icons, HuggingFace URLs, and download command
hints. Appears between Memory and Notes sections.
2. Default enrichment: --gguf-sources is now enabled by default when
running the scraper. Use --no-gguf-sources to skip (faster scrape).
3. Persistent cache: GGUF source lookups are cached in
data/gguf_sources_cache.json with a 7-day TTL. Subsequent scrapes
hit the cache instead of the HuggingFace API (72 cache hits, 0 API
calls on second run). Cache file is .gitignored.
Trade-off for default enrichment: adds ~30s to first scrape (194 API
calls at 150ms rate limit), but subsequent runs are instant via cache.
The --no-gguf-sources flag preserves the fast path when needed.
Signed-off-by: Three Foxes (in a Trenchcoat) <threefoxesyes3inatrenchcoat@gmail.com>
Add a --gguf-sources flag to the scraper that checks known GGUF providers
(unsloth, bartowski) on HuggingFace for pre-quantized versions of each
model in the database.
Why not add Unsloth as a model provider?
Unsloth publishes repacks/quantizations of existing models, not original
architectures. Adding them as separate model entries would create duplicates
with identical RAM/VRAM requirements. Instead, we link existing models to
their GGUF download sources — this is more actionable for users who want
to actually download and run models locally.
Changes:
- scraper: --gguf-sources flag checks unsloth/ and bartowski/ repos
- models.rs: GgufSource struct + gguf_sources field on LlmModel
- display.rs: show GGUF Downloads section in model info with direct
HuggingFace links and llmfit download command hint
- JSON output: gguf_sources included in fit results
38 models enriched with GGUF sources across both providers.
Signed-off-by: Three Foxes (in a Trenchcoat) <threefoxesyes3inatrenchcoat@gmail.com>
Add recently released LLM models including DeepSeek V3.2,
GLM-5, Kimi K2.5, MiniMax M2.5, MiMo V2 Flash, Nemotron 3,
Phi-4 reasoning family, EXAONE 4.0, SmolLM3, and Gemma 3n.
Update MOE_CONFIGS and MOE_ACTIVE_PARAMS for new MoE models.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Capture createdAt from HuggingFace API in the scraper, propagate
release_date through the Rust data model, and expose it as a new
Date column in the TUI table with a ReleaseDate sort option (newest
first, models without dates sorted last). Also shown in the detail
view, CLI output, and JSON export.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Expands model database from 33 to 48 models by adding highly popular
models from HuggingFace based on downloads, likes, and community usage.
New models added (15 total):
**Code Models (4):**
- meta-llama/CodeLlama-7b-Instruct-hf (6.7B) - Popular code generation
- meta-llama/CodeLlama-13b-Instruct-hf (13.0B) - Larger code model
- meta-llama/CodeLlama-34b-Instruct-hf (33.7B) - Professional code generation
- WizardLMTeam/WizardCoder-15B-V1.0 (15B) - Code specialist (fallback)
**Small/Edge Models (3):**
- google/gemma-2-2b-it (2.6B) - Smallest Gemma, very popular for edge devices
- microsoft/Phi-3.5-mini-instruct (3.8B) - Newer Phi variant, long context (131K)
- mistralai/Ministral-8B-Instruct-2410 (8.0B) - Smaller Mistral variant
**Reasoning & Chat Models (3):**
- microsoft/Orca-2-7b (7.0B) - Reasoning model, step-by-step solutions
- microsoft/Orca-2-13b (13.0B) - Larger reasoning model
- HuggingFaceH4/zephyr-7b-beta (7.2B) - Very popular fine-tune
**Multilingual Models (2):**
- 01-ai/Yi-6B-Chat (6.1B) - Popular Chinese/English bilingual
- 01-ai/Yi-34B-Chat (34.4B) - Larger Yi variant
**Mid-size Models (2):**
- mistralai/Mistral-Nemo-Instruct-2407 (12.2B) - Mid-size Mistral
- tiiuae/falcon-7b-instruct (7.2B) - TII UAE model
**Multimodal (1):**
- meta-llama/Llama-3.2-11B-Vision-Instruct (10.7B) - Vision + text
**Community Fine-tunes (1):**
- NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO (46.7B) - Popular community model
Changes:
- Updated TARGET_MODELS list with 21 new model repo IDs
- Added fallback entries for 12 gated/auth-required models
- Extended provider mapping for new organizations (01.ai, Upstage, TII,
HuggingFace, OpenChat, LMSYS, NousResearch, WizardLM)
- Regenerated data/hf_models.json with 48 total models
Note: Some models couldn't be scraped due to metadata issues or repository
changes (SOLAR-10.7B, falcon-40b, openchat-3.5, vicuna variants, WizardLM-13B).
These can be added with manual fallback entries if needed.
Addresses community requests for more model coverage, especially:
- Code-specialized models (CodeLlama series)
- Smaller edge models (Gemma 2B)
- Multilingual support (Yi series)
- Reasoning models (Orca 2)
- Popular community fine-tunes (Zephyr, Nous-Hermes)
Signed-off-by: Three Foxes (in a Trenchcoat) <threefoxes53235@gmail.com>
