llmfit/scripts
Alex Jones 84007d5d68
fix(scraper): coerce list-valued expert counts; a failed estimate cannot abort a scrape (#1064)
* fix(scraper): coerce list-valued expert counts; a failed estimate cannot abort a scrape

The weekly run on 2dd0628 crashed two minutes in:
  TypeError: can only concatenate list (not "int") to list
baidu/ERNIE-4.5-VL-28B-A3B-PT declares moe_num_experts as a per-modality
list ([64, 64]). The shared expert-key helper from #1055 started reading
that key; list * int is silently a list, so the failure surfaced a line
later in estimate_params_from_arch. The previous run missed it only
because the quantized-only gate (removed in #1063) skipped the estimator
for that repo. Left alone the list would also have reached the catalog's
integer num_experts field and failed the embedded Rust parse.

- _config_num_experts / _config_active_experts return the first positive
  int, taking the head of a list.
- correct_packed_param_count catches an estimator failure, logs it and
  keeps the reported count: one odd config.json must not discard a
  6,000-model scrape.

* fix(scraper): per-model discovery boundary with a mass-failure ceiling

Review follow-up on #1064: a null text_config raises AttributeError, which
the estimator guard did not catch, and discovery had no per-model boundary
at all. One malformed repo is now logged and skipped (its retained entry
stands). Skips are counted: more than 25 means the builder is broken, and
the run fails rather than shipping a quietly thinned catalog.

---------

Co-authored-by: Alex Jones <axjns@example.com>
2026-09-19 13:02:14 +01:00
..
install-openclaw-skill.sh docs: brew official formula 2026-03-02 11:58:15 +07:00
scrape_benchmarks.py chore: remove duplicated root data/ copies of catalog and benchmark cache 2026-07-02 21:49:07 +01:00
scrape_docker_models.py feat: add Docker Model Runner as a runtime provider 2026-03-14 17:42:58 +00:00
scrape_hf_models.py fix(scraper): coerce list-valued expert counts; a failed estimate cannot abort a scrape (#1064) 2026-09-19 13:02:14 +01:00
test_api.py feat: adding serve capabilities 2026-03-03 14:16:31 +00:00
test_preserve_catalog_metadata.py fix(scraper): coerce list-valued expert counts; a failed estimate cannot abort a scrape (#1064) 2026-09-19 13:02:14 +01:00
update_models.sh chore: remove duplicated root data/ copies of catalog and benchmark cache 2026-07-02 21:49:07 +01:00
validate_community_benchmarks.py ci: validate community benchmark submissions on PR 2026-07-10 11:33:44 +01:00
validate_generation_scoring.py feat(models): add the Qwen3.8 family and fix Qwen generation parsing (#893) 2026-08-17 07:57:27 +01:00
verify_models.py feat: caught some unavailable models on ollama 2026-02-19 10:40:18 +00:00