mirror of
https://github.com/AlexsJones/llmfit.git
synced 2026-10-20 11:13:01 +00:00
31a90a4465
* feat(scraper): backfill release_date for catalog entries without one 6,978 of the 15,007 catalog entries have release_date null, all but 12 of them outside the curated list: _build_first_page_url never expanded createdAt, so the listing carried no date, and the additive merge never revisits retained entries, so they never got one. The --sort date order sinks half of the catalog to the bottom. - Expand createdAt on the discovery listing. Same request, one more field, nothing else in the response changes. - backfill_release_dates() after the merge guard: entries with no date, most-downloaded first, through the model endpoint with expand[]=createdAt (id and date only). Bounded by --date-backfill-limit (default 2000, 0 disables), cached in data/release_date_cache.json so a killed run resumes and a repo with no date is not asked again every week. Only 401, 403, 404 and a response without createdAt count as "no date": a 429 that survives the retries, a 5xx or a network error leaves the entry uncached. No createdAt means None, never a guess. - Hermetic tests: listing expand, order, limit, cache hits, stale negatives, no-date repos, rate-limited and failed lookups. Script only, no catalog change: the regenerated hf_models.json comes in its own PR, as agreed on #176. Refs #176 * fix(scraper): a malformed release-date response cannot abort the scrape A 200 whose body is not an object, or whose createdAt is not a string, raised AttributeError or TypeError inside the worker; _lookup only catches OSError and ValueError, so the exception escaped pool.map and took the whole refresh down for one repository. A truncated body (http.client.IncompleteRead) escaped the same way. fetch_release_date now accepts only the documented shape, an object whose createdAt is absent, null or empty (no date) or an ISO timestamp, and raises ValueError for anything else; _lookup maps that, and HTTPException, to "unsettled": nothing cached, asked again next run. A createdAt that is not a real date no longer reaches the catalog. Test walks a null body, a list body, a numeric, zero and garbage createdAt and a truncated read, then a healthy repo, to show the run carries on and caches only the healthy one. Refs #176 * fix(scraper): default the release-date backfill to 500 and drop the priority-2 revalidation clause Review on #1060. The backfill runs inside the weekly job ahead of the GGUF pass, so its default is what CI actually spends: 500 lookups is one api window, about half an hour beside GGUF_PROBE_BUDGET, and the backlog still drains, over more weeks. The flag stays for manual runs. revalidation_priority no longer ranks a missing release_date: that path re-fetches the whole record at two requests per entry, its cooldown stamp and the date cache do not know about each other, and the backfill fills the same gap at one request. Revalidation keeps its budget for packed counts and suspect contexts. Refs #176 * ci: persist the release-date cache between weekly runs Same restore and save steps as the GGUF source cache, one more path on each, so the negatives recorded by backfill_release_dates() survive to the next run instead of being re-asked every week. Keys unchanged, so the existing GGUF cache is still restored. Refs #176 |
||
|---|---|---|
| .. | ||
| install-openclaw-skill.sh | ||
| record_demo.sh | ||
| scrape_benchmarks.py | ||
| scrape_docker_models.py | ||
| scrape_hf_models.py | ||
| test_api.py | ||
| test_preserve_catalog_metadata.py | ||
| update_models.sh | ||
| validate_community_benchmarks.py | ||
| validate_generation_scoring.py | ||
| verify_models.py | ||