An existing post's page answered 404 on one run in about 30, and the call
returned no items without an error. Reddit answers a missing post, user or
subreddit with a 200 page holding none, so a 404 is one exit's bad answer: it
is asked once more on a fresh exit, then read as nothing.
A call that stopped at max_items left its generator to the garbage
collector, which closed it in another context: releasing the subreddit
call's held session raised a ValueError, logged on every such call (live,
2026-10-23). scrape_reddit now closes the stream itself, as Instagram's does,
and so do the fan_out loops inside it, so their workers' sessions close
before the call returns too.
A feed read the posts the embed lacks, its pinned ones and those past the
embed's six, one after another on one exit: six pages took 16.8s there and
7.8s on three exits (2026-10-23), which also keeps each under its ~14 reads.
The feed walks its grid in windows of only the posts still wanted, so a full
feed reads no more pages and order stays the grid's; one blocked exit
cancels the others and is the target's refusal, as before.
The warm fetched instagram.com for a csrftoken, which only the JSON API
needs. Every logged-out page the plugin reads, profile, embed, reels tab,
post and reel, works as an exit's first request and leaves the next page
working (probe 2026-10-23, three countries), so fetch_html no longer warms.
fetch_json still does. Every Instagram call makes one request fewer.
A call naming only a community ran outside fan_out, so each page opened its
own exit and paid its own warm-up: 6 requests where 3 do. It now holds one
session, as fan_out does for every other target, and still streams, so it
stops at max_items rather than reading every post's comments first. Live, a
subreddit with its details went from 10.7s to 8.8s median.
An apostrophe followed by & or = stays inside the address, so
?q=patients'&location=clinic no longer logs its second field. HTTPS:// and
WSS:// are cut like their lower-case forms, and only one sentence-ending
punctuation mark after an address is left outside the redaction, so a query
ending in !!! is hidden too.
Without a uuid PostHog assigns a fresh one per send, so a manual --day rerun, or a send whose mark failed, counted the day twice. The uuid is uuid5 over the event name, day, plugin, tool and outcome.
A posts feed's newest posts come from the embed, which has no video URL, and
reading each post's own page would double a video-heavy feed's requests
against an exit's ~14 reads. scrape's description now says to scrape the
post's URL for a video's file instead. Its URL is signed and expires within
hours to days, so fetching it when needed also hands back one that works.
A post taken from the embed had no product type, alt text or owner's name,
so a reel in the posts feed did not say it was one. The profile grid carries
the first two and the profile page the name. Its video URL stays missing
rather than cost the post's own page; Known gaps says so, and that the reels
feed and a post URL carry it.
Each reel is read from its own page for its video URL, which the embed's
copy lacks, so the embed went unused: a second per profile and one of the
about 14 reads an exit gets before Instagram rate-limits it.
Newest first, the old default for every call, answered "fastapi" with job
posts that only mention it. sort is now optional: a search asks Reddit for
relevance, or for the newest under a post date limit, which stops paging at
the first older post; a subreddit's listing still orders by newest.
Ordered by relevance, Reddit wraps most results in
search-post-with-content-preview rather than search-post-unit, around the
same unit, so a search for fastapi read 1 of its 7 posts.
web_profile_info answers require_login to every logged-out session, so
pages/ reads what a visitor's browser is served instead. Details join the
profile's page (bio, following, link) with its embed (exact follower and post
counts, its newest posts in full), into the web_profile_info shape the parsers
read. The posts feed is the page's grid, each post taken from the embed or
else its own /p/<code>/ page; pinned posts are the ones the grid lists before
the embed's first. The reels feed is the /reels/ tab, which marks pinned reels
and counts plays.
A feed stops at the 12 a logged-out visitor sees, as web_profile_info's did,
and the tool says so. The login-wall message no longer claims profiles need
an account.
After about 14 page reads on one exit, Instagram redirects to its login page
with is_from_rle and a 429. That is the exit asking too often, not a page that
needs a login, so it rotates like a 401/403 instead of failing the target.
Each tab is its own feed under /svc/shreddit/profiles/. Comments there are
<shreddit-profile-comment> elements, laid out unlike a post page's; one on
the user's own profile is in their u_<name> subreddit. The posts feed links
no next page, so it pages on from its last post's more-posts-cursor.
A subreddit's own pages, /r/<name>/ and /about/, often serve a JavaScript
check, so its header is read from the page of the top post in its hot
listing, which carries the name, id, description and member count. Whether
it is NSFW comes from the listing.
A post comes from /comments/<id>/, whose comment tree streams in after
</html>, so the page is parsed whole. Later top-level comments load by the
page's own POST to /svc/shreddit/more-comments/ until max_comments, which the
scraper now passes down as the .json read's limit was. Nested "more replies"
stay unfollowed, as the .json read stopped at its more stubs.
Reddit closed logged-out .json on 2026-10-13 but still serves the HTML it
renders for a logged-out visitor. Each exit now warms with one listing sent
with the over18 cookie, which sets the loid jar; asked with no cookie, Reddit
answers a reCAPTCHA page and then a JavaScript check on that exit's full pages.
A challenge page rotates the exit rather than being solved, and no browser is
launched.
The scraper reads through pages/, which turns each page into the .json shape
it always paged through. This change reads subreddit listings and search,
site-wide and in one subreddit. Routes are asked for as Reddit spells them,
and a form POST carries Reddit's own referer, since either mistake is a 403.
Adding a plugin by an address ending in /sse, as providers' docs still
give theirs, is refused before anything is contacted, and the Add dialog
says so as the user types it, with Add off. An address that is not /sse
but streams SSE is refused as before.
A plugin recorded on SSE by an earlier version is no longer moved to its
server's /mcp at a check: it is never contacted, and its status, in the
list and in its details, says SSE is not supported and how to replace it.
The registry bot names a submitted /sse address as SSE whatever it
answers, since a sign-in there read as a key to take, and gives the /mcp
beside it when MCP answers there. The submit form says Streamable HTTP
only.
Only SurfSense's own plugins carried example requests and no registry entry
does, so they did nothing while the tools fit and, past the share, ranked
SurfSense's plugins ahead for words the others lack. Tools are now ranked
by what every plugin has: name, description, parameters and its plugin's
line. The list entries' `examples` field and its registry check go too.
While every ready plugin tool fits a quarter of the model's window and the
provider's limit on tools, an opencode turn holds them all. Past that the
agent sees every ready plugin in SurfSense's MCP instructions and holds its
mentioned plugins, the best tools of plugins its message names, the tools
the thread used, and what keyword and embedding search ranks highest,
example requests included, until the share is spent.
A turn's holding only grows, by whole plugin, with the session's rules
rewritten before the step that needs them: after a plugin call, inside the
gateway before the result goes back; when the agent calls a tool it does not
hold, which reads in the thread as "Loaded X's tools"; and through
surfsense_open_plugin and surfsense_find_tools, offered to opencode only and
only when the tools do not all fit. Claude Code still holds every tool.
The window budget and the "Not all of a plugin's tools fit this turn" notice
go. List entries may carry example requests per tool, bounded by the
registry's check, and SurfSense's own plugins now do.
Each MCP server is added off and then connected, so one that hangs
holds no other, waited on until its tools are offered, with Code Mode
off and a hard call limit. The gateway takes elicitation out of the
runtime's offer, and turns a runtime's cancel of a call on SurfSense's
plugin host into the host's own stop once no other call of the thread
runs there; a saved result and a step's whole result read
state.content.
SurfSense no longer speaks the older HTTP+SSE transport, on either
engine: connecting a server that answers only there is refused as one,
the gateway's SSE endpoint goes, and the registry refuses an SSE entry.
A plugin recorded on SSE is tried at its server's /mcp at its next
check, moving there, asking to sign in again, or staying, never
offered, with a status that says why.
It cuts http(s) and ws(s) addresses up to the first space. An address without
its scheme, with escaped slashes or with an unencoded space keeps the rest, so
the docs no longer say a page or search never reaches a log.
Reddit's and Instagram's refusals quote the subreddit, post id or shortcode,
and the fan-out workers logged the whole error at WARNING, which reaches
stderr in production. They now log the class; the tool's error still tells
the user which target was refused.
Every earlier test read through the meter, whose reads write first, so a host that never ran the meter's jobs still passed. These read the file with a store of their own and watch a stand-in PostHog while the host serves.
Perf and warning lines printed the pages, searches, ASINs, video, place and
job ids users asked plugins for. They now name a host or a count. The log
redaction also cuts every web address in a record to its scheme and host, so
a URL quoted in a library's error never reaches a log either.
The meter read a whole request body before the check refused it, so an unlicensed caller could make the host hold a body of any size; before the meter, a refusal came after the headers alone. Past 1 MiB the meter stops reading, hands the server what it read and the rest, and leaves the call uncounted. An agent's tool call is a few kilobytes.
The image makes /srv/remote/usage/, owned by the host's user, where USAGE_DB_PATH points by default; production mounts a persistent volume there. A local run's file stays out of git and the image. .env.example lists USAGE_DB_PATH, USAGE_ADMIN_TOKEN, POSTHOG_API_KEY and POSTHOG_HOST.
host/usage_meter/ counts only tools/call requests to /<id>/mcp: per UTC day, Keygen license id (or unlicensed), plugin, tool and outcome (ok, tool_error, refused, cancelled, error), with calls and total milliseconds, and each license's first day with a call. It sits outside the license check so refusals count, reads only the tool's name, keeps a name the plugin lacks as unknown, and never keeps the arguments, the key or the thread's token.
Counts are written to SQLite (USAGE_DB_PATH) every minute and at shutdown, off the event loop; a failing meter is logged once and never fails a call. Once a day, rows older than 13 months are forgotten and, with POSTHOG_API_KEY, yesterday's totals go to PostHog as personless plugin_usage_daily and plugin_licences_daily events with no license id; python -m host.usage_meter.rollup --day sends one by hand. With USAGE_ADMIN_TOKEN, /admin/usage/first-seen and /admin/usage/daily serve the records to a bearer token; admin is reserved as a plugin id.
validate-key already answers with the license as data.id. The check keeps it per key from the same answer, with no extra Keygen call, and sets checked_license for the request, allowed or refused, never the key.
Say what servers the registry supports, drop the Inspector paragraph,
shorten the no-secrets box, and word the access choices as Free and Paid,
so a free account no longer reads as needing one sold by the publisher.
The intake's FREE and PAID match the new choices.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A registry entry can name the headers a key goes in, say `auth.user_client`
for a provider that takes only registered apps, and record `transport`. The
submission form and the bot check them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A publisher or an agent fills in the Submit a plugin form; no git. The bot
contacts the server as the app does, reads how it signs in and its hosts,
checks the declared tool names against what it lists without sign-in, fetches
the icon, and keeps one comment with the entry or what to fix. A maintainer's
registry: accept label has it open the pull request into dev.
Tool names are required, since the plugin's page shows them; a token entry's
key wording is optional, the app's own being used without it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Google Search: Google's sorry wall is recognised by host and path, not a
"/sorry" substring, so a result linking to a page with "sorry" in its URL
no longer burns the wave's exit (goto and the captcha check share
captcha.is_sorry_url).
- Walmart: a redirect still pending after the last allowed hop is a refused
try, so the next exit is tried instead of reading an empty page.
- Amazon: a hedge whose sticky lease is unavailable falls back to the
gateway instead of spending the call's hedge budget on nothing.
- Fixtures: live tracking and session tokens scrubbed from the captured
pages (Indeed, Amazon, Google, Google Maps, TikTok, YouTube).
- plugin-host.md: shared_access() everywhere, and the Reddit and Instagram
gaps now describe their fail-fast latches.
Reddit refuses logged-out .json and Instagram's profiles sit behind a login.
Reddit now gives up after one rotation instead of about 2m50s and latches the
refusal, so calls answer at once with an error naming the countries tried.
Instagram no longer rotates on its require_login 401 and points to /p/ and
/reel/ URLs, which still work. Both use the shared license cache, progress and
text results.
- Pages are crawled level by level, several at once, with an exact page budget.
- HTML is capped at 1 MB before trafilatura and parsed once: a 7.3 MB page
took 17-25 s to extract, now about 1.5 s. Extractors warm up at startup.
- The tier ladder follows the HTTP status and challenge markers: a 404 fails
in about 1 s instead of trying the browser tiers. The dynamic tier runs on
the shared browser pool with a bounded settle.
- Output is bounded: links and contacts are capped with totals, and every
page gets its share of the agent's text. A 3-page crawl reply went from
1.85 MB to 49 KB.
- Fixes: GitHub and GitLab URLs filed as social, 404s and Cloudflare pages
returned as success, loadedUrl ignoring redirects, links resolved against
the pre-redirect URL, the oldest date on the page used as its date,
permalink anchors in titles, and raw HTML from the stealth tier.
A 3-page docs crawl went from about 28 s to 4-8 s.
- Comments and trending read TikTok's unsigned JSON endpoints, with the
browser as fallback: comments went from 25-96 s to about 2 s.
- Listings run on the shared browser pool, with event-driven waits, a JS
click, no home-page warm-up, media blocked, a hedged second country, and
concurrent targets; profile and video pages go over HTTP.
- Geo-blocked and login-bounced exits end the attempt at once.
- user_search recovers an empty first body inside the page instead of
starting a new browser, which made it intermittent.
- Fixes: textLanguage, isAd, isPinned, isSlideshow, mentions and slideshow
media dropped by parse_video, business accounts marked verified, withheld
comment pages, and the profile item counting toward max_items.
- Review pages after the first are fetched in parallel and deduplicated;
start URLs and search terms run together, in order.
- Fail-fast transport with hedging; 444, 403, 5xx and a /blocked redirect are
blocks that rotate, not pages; raw bytes and orjson instead of the lxml
round trip.
- A failed detail returns the listing card with detailsError instead of an
error item and an extra listing fetch.
- Fixes: variant products with 0 stars and 0 reviews, the sponsored flag and
search provenance lost on details, duplicate reviews at page boundaries,
reviews of other variants unlabelled, specifications and fulfillment always
null, and lone surrogates in review text.
200 reviews went from 34-99 s to about 10-13 s. WALMART_PARALLEL=0 restores
sequential fetches; WALMART_CANONICAL_URL=1 (off) skips the /ip redirect.
- Fail-fast transport with one hedge on slow fetches; sources run together
in order; the offers panel runs alongside the product page and sellers as
soon as they are known; one detail wave for the default call.
- No Google referer on search, best-seller and offers requests: in an A/B it
cut blocked search pages from 10 of 10 to 3 of 10.
- Sponsored cards are unwrapped instead of failing as product_not_found, and
max_items fills from the current page before paginating.
- Fixes: listPrice read the per-unit price, search positions and
reviewsCount ('89.7K' read as 897), isAmazonChoice/isBestSeller always true,
duplicate ASINs, null prices, stock and delivery on Amazon's fallback product
page (about 70% of pages), offer prices, the reviews link, and zip_code
silently ignored (now reported, and failures cached).
A default 10-product search went from 31-74 s, with null prices, to 8-11 s.
AMAZON_NO_GOOGLE_REFERER=0 and AMAZON_HEDGE=0 restore the old behaviour.
