mirror of
https://github.com/AlexsJones/llmfit.git
synced 2026-10-20 11:13:01 +00:00
a4b669c814
* fix(scraper): revalidate a budgeted slice of retained catalog entries The merge is additive: a model that drops out of discovery is retained exactly as last scraped, so later scraper fixes never reach it. That is why #1045 persists. Discovery has fetched config.json (and applied the packed-parameter correction) for a while, but three of the four repos in the report are retained entries scraped before that, still listed at 7.8B/9.1B/2.3B with null architecture fields. Each run now re-fetches up to --revalidate (default 250) retained entries through scrape_model and the rate-limited _hf_urlopen door, most suspect first: 0. pre-quantized repos with no architecture metadata (#1045) 1. context windows above 2M (YaRN double-scaling leftovers) 2. missing release_date (#176) ordered by downloads within a tier. A repo that is gone, gated or unparseable returns nothing and stays retained. Refs #1045, #176 * fix(scraper): only correct packed parameter counts on quantized repos safetensors.total under-reports only when weights are packed, so the architecture estimate may override it only for a quantized repo (config declares quantization, or the name does). The 2x margin alone is not enough: once the estimator learned n_routed_experts (#1055 follow-up) it priced every Nemotron-H hybrid layer as a MoE transformer layer and reported 101.6B for the 31.6B BF16 Nemotron-3-Nano and 3.5-Lightning checkpoints. main's shipped data predates that commit and is correct; the next weekly run would have written the wrong value. Revalidation also keeps the retained entry when a re-fetch reports under 2/3 of its parameters, which is more likely an uncorrectable packed count (gpt-oss-20b 3-bit: 22.3B -> 2.9B) than a real correction. Catalog regenerated with --no-discover --revalidate 1000: 943 of 1000 suspect retained entries re-fetched. * fix(scraper): cool down attempted revalidations, ignore 'unquantized' names Review follow-ups on #1057: - The ranking is deterministic and failed re-fetches change nothing, so repos that are gone or gated would hold the top of the budget every run (92 already do) and eventually starve the lower tiers. Every attempt is now stamped with _revalidated and sits out a 90-day cooldown. The 1000 entries this run attempted are stamped from its log. - google/gemma-3-1b-it-qat-int4-unquantized matched 'int4' but ships full-precision weights; names containing 'unquantized' are no longer treated as pre-quantized. schema.json allows the new optional _revalidated date. --------- Co-authored-by: Alex Jones <axjns@example.com> |
||
|---|---|---|
| .. | ||
| install-openclaw-skill.sh | ||
| scrape_benchmarks.py | ||
| scrape_docker_models.py | ||
| scrape_hf_models.py | ||
| test_api.py | ||
| test_preserve_catalog_metadata.py | ||
| update_models.sh | ||
| validate_community_benchmarks.py | ||
| validate_generation_scoring.py | ||
| verify_models.py | ||