- Google Search: Google's sorry wall is recognised by host and path, not a
"/sorry" substring, so a result linking to a page with "sorry" in its URL
no longer burns the wave's exit (goto and the captcha check share
captcha.is_sorry_url).
- Walmart: a redirect still pending after the last allowed hop is a refused
try, so the next exit is tried instead of reading an empty page.
- Amazon: a hedge whose sticky lease is unavailable falls back to the
gateway instead of spending the call's hedge budget on nothing.
- Fixtures: live tracking and session tokens scrubbed from the captured
pages (Indeed, Amazon, Google, Google Maps, TikTok, YouTube).
- plugin-host.md: shared_access() everywhere, and the Reddit and Instagram
gaps now describe their fail-fast latches.
Reddit refuses logged-out .json and Instagram's profiles sit behind a login.
Reddit now gives up after one rotation instead of about 2m50s and latches the
refusal, so calls answer at once with an error naming the countries tried.
Instagram no longer rotates on its require_login 401 and points to /p/ and
/reel/ URLs, which still work. Both use the shared license cache, progress and
text results.
- Pages are crawled level by level, several at once, with an exact page budget.
- HTML is capped at 1 MB before trafilatura and parsed once: a 7.3 MB page
took 17-25 s to extract, now about 1.5 s. Extractors warm up at startup.
- The tier ladder follows the HTTP status and challenge markers: a 404 fails
in about 1 s instead of trying the browser tiers. The dynamic tier runs on
the shared browser pool with a bounded settle.
- Output is bounded: links and contacts are capped with totals, and every
page gets its share of the agent's text. A 3-page crawl reply went from
1.85 MB to 49 KB.
- Fixes: GitHub and GitLab URLs filed as social, 404s and Cloudflare pages
returned as success, loadedUrl ignoring redirects, links resolved against
the pre-redirect URL, the oldest date on the page used as its date,
permalink anchors in titles, and raw HTML from the stealth tier.
A 3-page docs crawl went from about 28 s to 4-8 s.
- Comments and trending read TikTok's unsigned JSON endpoints, with the
browser as fallback: comments went from 25-96 s to about 2 s.
- Listings run on the shared browser pool, with event-driven waits, a JS
click, no home-page warm-up, media blocked, a hedged second country, and
concurrent targets; profile and video pages go over HTTP.
- Geo-blocked and login-bounced exits end the attempt at once.
- user_search recovers an empty first body inside the page instead of
starting a new browser, which made it intermittent.
- Fixes: textLanguage, isAd, isPinned, isSlideshow, mentions and slideshow
media dropped by parse_video, business accounts marked verified, withheld
comment pages, and the profile item counting toward max_items.
- Review pages after the first are fetched in parallel and deduplicated;
start URLs and search terms run together, in order.
- Fail-fast transport with hedging; 444, 403, 5xx and a /blocked redirect are
blocks that rotate, not pages; raw bytes and orjson instead of the lxml
round trip.
- A failed detail returns the listing card with detailsError instead of an
error item and an extra listing fetch.
- Fixes: variant products with 0 stars and 0 reviews, the sponsored flag and
search provenance lost on details, duplicate reviews at page boundaries,
reviews of other variants unlabelled, specifications and fulfillment always
null, and lone surrogates in review text.
200 reviews went from 34-99 s to about 10-13 s. WALMART_PARALLEL=0 restores
sequential fetches; WALMART_CANONICAL_URL=1 (off) skips the /ip redirect.
- Fail-fast transport with one hedge on slow fetches; sources run together
in order; the offers panel runs alongside the product page and sellers as
soon as they are known; one detail wave for the default call.
- No Google referer on search, best-seller and offers requests: in an A/B it
cut blocked search pages from 10 of 10 to 3 of 10.
- Sponsored cards are unwrapped instead of failing as product_not_found, and
max_items fills from the current page before paginating.
- Fixes: listPrice read the per-unit price, search positions and
reviewsCount ('89.7K' read as 897), isAmazonChoice/isBestSeller always true,
duplicate ASINs, null prices, stock and delivery on Amazon's fallback product
page (about 70% of pages), offer prices, the reviews link, and zip_code
silently ignored (now reported, and failures cached).
A default 10-product search went from 31-74 s, with null prices, to 8-11 s.
AMAZON_NO_GOOGLE_REFERER=0 and AMAZON_HEDGE=0 restore the old behaviour.
- Fail-fast transport with one keep-alive session per place job, rotated on a
block and hedged when slow; no Chromium fallback on a 403 or 429.
- The place header is fetched with the first reviews page; sources and places
run concurrently; no NID cookie mint; name-only URLs resolve with one search
RPC; place_id to fid is cached.
- Fixes: textTranslated was the truncated preview, raw <br> and entities in
reviews and replies, '' for rating-only reviews, the address repeating the
name, phoneUnformatted, kgmid on reviews, @lat,lng URLs resolving to the
wrong place, and a refused request silently returning nothing (it now
rotates, then names the input in an error item).
Reviews for 2 places went from about 11-22 s to 2-6 s. GOOGLE_MAPS_KEEP_IDLE=1
keeps warm sessions across calls (off by default).
- Fail fast and rotate on Google's sorry redirect instead of following it;
hedge slow requests onto a second exit (1 rotation per refused exit).
- Subtitles from YouTube's own player on the warm session, alongside the
watch fetch and across videos: a video with subtitles went from 13 s to 1 s.
- Channel tabs fetched together without the redundant sort chip; the
playlist stops at its limit; comments read reply pages and use /next.
- JSON read with raw_decode and orjson, one parse per page.
- Fixes: search terms dropped next to urls, truncated search text, rounded
views and likes, SEO keywords as hashtags, duration formats, verified
badges, reply pagination, None counts, a translation request without
subtitles, and relative times.
- The agent reads every video in the text; transcripts share the cap.
YOUTUBE_KEEP_IDLE=1 keeps warm sessions across calls (off by default).
- No per-call home-page warm-up: go straight to the page and solve Cloudflare
only when a challenge is there.
- One kept, country-pinned browser context holds the clearance across calls,
so a warm call is about 2 s instead of 30-125 s.
- Job details come in batches from Indeed's own spa=1 JSON inside the page;
a call's targets run concurrently; the inline JSON is read with raw_decode.
- Fixes: jobKey on direct /viewjob URLs, detail salaries, company rating,
review count, job types and country on details, clean companyUrl, 0 ratings
read as unknown, cross-target dedupe, a 403 page no longer read as empty,
and a failed warm-up no longer recorded as done.
INDEED_KEPT_CONTEXT=0, INDEED_SPA_DETAILS=0 and INDEED_TARGET_CONCURRENCY=1
turn the new paths off.
- No network-idle wait or fixed sleeps: the page is read once results are on
it, and People also ask and the AI Overview expand on events.
- A call's queries and pages render concurrently, in order; /goto links
resolve during the render on one session per wave, fail fast and hedge.
- The browser launches at startup; the DOM is trimmed before it is read.
- Fixes: resultsTotal on page 2 and in ja/zh, dates ending in a dash, star
ratings read as dates, 'Read more' in descriptions, AI Overview sources
(parsed and resolved), displayedUrl on forum cards, countryCode stamped as
sent, site: results that Google ignored (dropped, re-rendered, then SearXNG),
and PAA answers after a captcha.
- The agent reads each page's results, PAA and AI Overview in the text.
Warm, one query went from about 16 s to 6 s, two queries from 36 s to 10 s.
GOOGLE_SEARCH_FAST_SETTLE=0 restores the old waits.
- A plugin's server.py may expose a lifespan: the host enters it at startup,
warm-ups run in the background, and scraping/lifecycle.py closes the shared
sessions, browsers and API clients once at shutdown.
- One license cache per process, stale-while-revalidate, instead of one per
plugin: the app's 5-minute check no longer pays a Keygen round trip per
plugin. Refusals and the 24 h grace work as before.
- Kept API clients for Keygen, SearXNG and the captcha solver, which now polls
every second.
- mcp_server/results.py: tool_result puts each item in the text as one
compact JSON line, within 32 KB. opencode reads only a result's text when
there is any, so the agent never saw prices, page text or reviews. A request
with x-surfsense-text-only: 1 gets no structured copy.
- mcp_server/progress.py: stage progress, so a long call never sits silent.
- GET and DELETE on /<id>/mcp answer 405: the stateless host has no stream to
offer, and each GET held an idle SSE connection open per plugin.
scraping/browser_pool.py generalises Google Search's kept browser: at most 3
browsers on the browser loop, a fresh proxied context per page, a capped set
of kept contexts (for Indeed's Cloudflare clearance) with a TTL, and close or
relaunch on a crashed browser without taking sibling pages down. Cancelling a
call closes only its page.
scraping/settle.py navigates to DOMContentLoaded or a ready selector with a
bounded idle wait, and solves Cloudflare only when a challenge is on the page.
A contract test fails if Scrapling's private helpers move.
Shared by the scraper plugins:
- scraping/http.py: one attempt per request with a split (connect, read)
timeout, no hidden Scrapling retries or sleeps on the same exit, raw bytes
back (no lxml round trip on the event loop), and one perf line per request.
FAIL_FAST gives the same timeouts to call sites still on Scrapling.
- scraping/sessions.py: a per-job keep-alive SessionHolder that rotates or
burns its exit on a block, sticky exits leased so no two jobs share one,
and an optional, capped pool of idle sessions kept across calls.
- scraping/concurrency.py: ordered_map, a bounded fan-out that yields in
input order and cancels what is left when the caller stops, and hedged,
a second attempt on a fresh exit when the first is slow.
curl_cffi and orjson, already installed through Scrapling, are declared, and
Scrapling is pinned to 0.4.15, whose private browser API the pool uses.
Removing, importing or replacing a license, opening Plugins with a stale
verdict, and a turn after one all ask the plugin host's new GET /license,
once for every paid plugin, and apply its answer: a refused license holds
the plugin out of @ and the agent, and a valid one brings it back. The host
stays the only judge; the license file only says when to ask, and no answer
keeps every verdict. A paid plugin's refusal in a turn is named as its
license, not a sign-in. Contract 2 gains the route.
The core/ and remote/ proposals described work that has shipped. What they
said that the architecture docs lacked now lives there: the MCP client's
rules, what the gateway does today, what a plugin can reach, adding a plugin
to the host, who changes proprietary/, and the backend's copy of the
scrapers. proposals/plugins/README.md now holds only what is not built: the
chat router step, jobs and scaling on the host, organisation policy, later
MCP features and the open questions. Bundles stay deferred, with their
registry entry moved in. ADRs 0051, 0053 and 0054 are accepted, and links
into the deleted files point at the architecture docs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Redaction formatted every record and dropped its arguments. uvicorn's access
log unpacks its five arguments, so on the deployed host every request logged
"TypeError: cannot unpack non-iterable NoneType object" in place of its access
line. Each argument is now redacted in place, keeping its shape and type, and a
template with arguments is left whole so its placeholders still match. A record
whose credential shows only once formatted, split across the template and its
arguments, still keeps just its redacted text.
plugin-host.md describes the Keygen settings and answers, log redaction and the
new tests, and replaces the empty-account-id gaps with Reddit's and Instagram's.
.env.example gains the KEYGEN_* settings and names the eight plugins in PLUGINS.
Scrapling logs a failed proxy with its user and password, so a deployed host
would write the proxy vendor's credentials into its logs. python -m host now
redacts each record as it is made, its message, arguments and traceback: a URL's
user:password@, the captcha solvers' user:password@host:port form, and a password
or a solver's key as a field.
keygen.py held Keygen Cloud's address and an empty account id, so the host could
not start with a paid plugin, and could not reach the self-hosted Keygen CE that
holds production's licenses. It now reads KEYGEN_API_URL (the accounts endpoint,
Keygen Cloud's when unset), KEYGEN_ACCOUNT_ID and KEYGEN_HOST, with the backend's
names and meanings, so a deployment gives both the same values. With KEYGEN_HOST
set it sends Host and X-Forwarded-Proto: https, as CE needs.
A 4xx or a redirect from Keygen refuses the key as invalid, with a warning in the
log, instead of failing the request with a 500. A reply with no code is no
answer, so the 24-hour grace still applies.
Each plugin in Browse opens one shared public page, built from its list
entry and the same whether it is connected or not: its publisher, about,
listed tools as chips, authentication, the hosts it connects to, the
month it was added, and its homepage and privacy policy. Yours keeps a
connected plugin's settings and live tools alone.
Entries gain `about`, `tools` and `added`, required in both lists and
checked in review for the registry; the first-party list's tools are
checked against each server by the plugin host's tests. One Connect
serves every listed plugin. Browse lists them in two columns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
plugins/registry/ holds the list publishers add to by pull request, starting
with DeepWiki, and its icons. registry_check checks every entry and icon rule;
on dev, CI signs the list with Ed25519 into published/, which the release pull
request carries to main, so nothing pushes to main and the source list is
never rewritten. A small Caddy container serves the signed list, its icons and
SurfSense's OAuth client document and callback page on plugins.surfsense.com,
apart from the plugin host so a listing never restarts the scrapers. CI does
not contact listed servers: most need a sign-in, so a maintainer connects each
listing on dev. plugins/README.md is the guide for publishers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The docs no longer name a platform: the plugin host needs a container, Redis,
object storage and a proxy in front that routes by path, and where it runs is
the deployment's choice. The signed registry, its icons and the OAuth files are
served by a small static container beside the plugin host, on the same
hostname, so updating the list never restarts the plugins. CI commits the
signature back to main, and the app refuses a list older than its cached copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/web_crawl/ is the backend's web crawler behind one tool, crawl, at
/web-crawl/mcp, licensed like Reddit: pages read as markdown, or a site
followed by its links. It refuses private addresses before any fetch, and its
browser tiers await the async captcha solver. The first-party list renames the
plugin from web-crawler to web-crawl, and Google Search's refusal of a page to
read now points the agent at it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/indeed/ is the backend's Indeed scraper behind one tool, scrape, at
/indeed/mcp, licensed like Reddit. Its Cloudflare-solving session is Scrapling's
stealth Chromium, so the image needs no Camoufox, and each session now holds
one sticky IP, since its clearance belongs to that IP.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/tiktok/ is the backend's TikTok scraper behind four tools, scrape,
comments, trending and user_search, at /tiktok/mcp, licensed like Reddit.
TikTok withholds profile feeds from a headless browser, so the image installs
Xvfb and host/entrypoint.sh starts it when CRAWL_HEADED_XVFB_ENABLED is true,
then execs the host so it still gets the container's signals.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/instagram/ is the backend's Instagram scraper behind two tools,
scrape and details, at /instagram/mcp, licensed like Reddit. A keyword finds
profiles through the Google Search plugin's scraper; Instagram refusing every
address is the tool's error. Instagram's anonymous profile API now answers 401
on every address, so both tools fail on public profiles until they read the
profile page instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An agent wanting an article's text passed its URL as a query, and Google
Search paid a render and a captcha solve to search for the URL. Another site's
URL is now the tool's error, which its description warns of; Google Search
URLs are still scraped as they are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DataImpulse ignores a session id in the username and rotates the exit IP per
request, measured live: five requests on one session went out on five IPs.
Each port from 10000 to 20000 holds one IP, with or without a country, so a
session now picks its port from its id. Reddit's warm-up, Amazon, Walmart and
Google Search were all asking for sticky IPs and not getting them; Google
Search paid a captcha solve on every page for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Redis runs as a container that restarts with the platform. A copy of the host
now starts without it and resubscribes when it returns, instead of failing to
start; a Stop sent while Redis is down still ends the calls on the copy that
took it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/google_search/ is the backend's Google Search scraper behind one
tool, scrape, at /google-search/mcp, licensed like Reddit. Google walling every
address, with no SearXNG page to fall back on, is the tool's error; a SearXNG
page says it is not Google's ranking.
The captcha solvers move to proprietary/scraping/captcha/, async on httpx2,
with 2captcha over HTTPS, set by SCRAPING_CAPTCHA_PROVIDER and
SCRAPING_CAPTCHA_API_KEY. The shared warm-IP pool uses async Redis at
REDIS_URL, and sticky IPs come from the proxy provider, so NodeMaven holds
them too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An expired key that was valid earlier in the day passed while Keygen was
unreachable. Any refusal now ends the 24-hour grace.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/google_maps/ is the backend's Google Maps scraper behind two
tools at /google-maps/mcp, scrape and reviews, licensed like Reddit; a
sign-in wall is the tool's error. Its summary shows a review count only when
known. The backend's tests came with it, those reading uncaptured responses
skipped as there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/walmart/ is the backend's Walmart scraper behind two tools at
/walmart/mcp, scrape and reviews, licensed like Reddit, plain HTTP on US
exits. A block is an error item per input, as the backend's. The backend's
tests came with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/amazon/ is the backend's Amazon scraper behind one tool, scrape,
at /amazon/mcp, licensed like Reddit, plain HTTP with the exit country
following the domain. A block is an error item per input, as the backend's.
The backend's tests came with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/youtube/ is the backend's YouTube scraper behind two tools at
/youtube/mcp, scrape and comments, licensed like Reddit, with progress per
item from an on_item callback. youtube-transcript-api, fake-useragent and
requests join the host for its subtitles. The backend's tests came with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The custom provider passed a proxy URL through unchanged, so it could not
hold a sticky session or change country, which Reddit needs, and a vendor's
gateway set up as custom failed quietly. Only SurfSense runs these scrapers,
on one vendor, so the provider is nodemaven or dataimpulse; unset, scrapes go
direct, and an unknown name is an error rather than a silent fall back. The
pool of comma-separated URLs went with it. The proxy messages name the
SCRAPING_* variables, and host/.env.example keeps one line per setting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One image from python:3.12-slim with the locked packages, Chromium through
scrapling install and patchright under /ms-playwright, and fonts, running
python -m host as an unprivileged user with a health check. It holds what
Reddit needs; each later plugin adds its own. host/.env.example lists every
setting the container reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/reddit/ is the backend's anonymous Reddit scraper behind one MCP
tool, scrape, at /reddit/mcp, licensed with LicenseAccess(): a summary as
text, the items as structured content, and progress after each item. A run
whose every target Reddit refused is now that refusal, not an empty result.
proprietary/scraping/ holds what the scraper plugins share, the proxy
providers and the browser loop, reading SCRAPING_* settings. The backend's
Reddit and proxy tests came with them; Keygen reads its address when made,
so a test can point a plugin at a stand-in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The plugin host serves MCP without sessions, so any copy answers any request
and Azure can add copies on load. opencode cancels with a message no such
copy can match, so each turn gives plugins on the plugin host a token for
the thread, and the app's Stop calls the host's POST /cancel with it and the
license; Redis passes it to the copy running the call, whose response ends
with Request cancelled. PLUGINS limits a container to some plugins, as one
split out of the shared container runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first-party list holds ten scraper plugins, from Google Search to Indeed,
each its own MCP server at https://plugins.surfsense.com/<id>/mcp. The host
serves a folder under its id with underscores as hyphens, since a package
cannot hold a hyphen, reserves scraping/ for the code they will share, and
lints plugins/remote/ with ruff's ASYNC rules, since its plugins share each
process's event loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
plugins/remote/host mounts each plugin folder's MCP server at /<id>/mcp and
answers /health; without proprietary/ it serves only the Apache-2.0 plugins.
A plugin's access hook runs before its server sees a request, and
proprietary/license_check applies contract 2 with Keygen's validate-key: a
five-minute cache, a day's grace while Keygen is down, and 401, 403 or 503
otherwise. The check is license_check/, since license/ beside LICENSE is one
path on macOS and Windows. CI checks plugins/remote/ on its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The root LICENSE names the folder, which holds its own copy of the Business
Source License. CODEOWNERS gives it to the maintainers, and code-quality.yml
fails a pull request from anyone else that touches either proprietary/ folder,
so SurfSense holds all of that code's copyright and needs no contributor
agreement. The docs and ADRs say so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
