diff --git a/docs/developers/tools/web-fetch.md b/docs/developers/tools/web-fetch.md index 88f0b6daa9..3d6a00f716 100644 --- a/docs/developers/tools/web-fetch.md +++ b/docs/developers/tools/web-fetch.md @@ -13,10 +13,10 @@ Use `web_fetch` to fetch content from a specified URL and process it using an AI - `url` (string, required): The URL to fetch content from. Must be a fully-formed valid URL starting with `http://` or `https://`. - `prompt` (string, required): The prompt describing what information you want to extract from the page content. - `format` (string, optional): Controls only the `Accept` header sent to the server, indicating your content preference. **All fetched content is normalized to plain text for LLM processing**, regardless of the format specified. Defaults to `"auto"` if not specified. - - `"auto"` (default): Prefers markdown via content negotiation (`Accept: text/markdown, text/html`), accepts HTML as fallback. **Recommended for most use cases** as it can reduce token usage by up to 80% for servers that support markdown. - - `"markdown"`: Sends `Accept: text/markdown`. Use when you explicitly need markdown content. - - `"html"`: Sends `Accept: text/html`. Use when the server requires HTML in the Accept header. Content is still converted to plain text for LLM processing. - - `"text"`: Sends `Accept: text/plain`. Use when you specifically need plain text content. + - `"auto"` (default): Prefers markdown via content negotiation (`Accept: text/markdown, text/html;q=0.9, text/plain;q=0.8, */*;q=0.1`), then falls back to HTML, plain text, or other content types. **Recommended for most use cases** as it can reduce token usage by up to 80% for servers that support markdown while still working with JSON-only APIs. + - `"markdown"`: Prefers `Accept: text/markdown, */*;q=0.1`. Use when you explicitly need markdown content. + - `"html"`: Prefers `Accept: text/html, */*;q=0.1`. Use when the server requires HTML in the Accept header. Content is still converted to plain text for LLM processing. + - `"text"`: Prefers `Accept: text/plain, */*;q=0.1`. Use when you specifically need plain text content. ## How to use `web_fetch` with Qwen Code @@ -71,8 +71,8 @@ web_fetch(url="https://developers.cloudflare.com/fundamentals/reference/markdown - **Single URL processing:** `web_fetch` processes one URL at a time. To analyze multiple URLs, make separate calls to the tool. - **URL format:** The tool automatically upgrades HTTP URLs to HTTPS and converts GitHub blob URLs to raw format for better content access. -- **Content negotiation:** The tool supports "Markdown for Agents" content negotiation. When using `format="auto"` (default), it sends `Accept: text/markdown, text/html` headers, allowing servers that support markdown to return it directly instead of HTML. This can reduce token usage by up to 80%. -- **Content processing:** The tool fetches content directly and processes it using an AI model. When the server returns HTML, it converts it to readable text format. When the server returns markdown or plain text, it uses the content as-is. +- **Content negotiation:** The tool supports "Markdown for Agents" content negotiation. When using `format="auto"` (default), it sends `Accept: text/markdown, text/html;q=0.9, text/plain;q=0.8, */*;q=0.1`, allowing servers that support markdown to return it directly instead of HTML. The low-priority `*/*` fallback keeps JSON-only APIs and other non-text endpoints fetchable. This can reduce token usage by up to 80%. +- **Content processing:** The tool fetches content directly and processes it using an AI model. When the server returns HTML, it converts it to readable text format. When the server returns markdown, plain text, or another fallback content type such as JSON, it uses the content as-is. - **Output quality:** The quality of the output will depend on the clarity of the instructions in the prompt. - **MCP tools:** If an MCP-provided web fetch tool is available (starting with "mcp\_\_"), prefer using that tool as it may have fewer restrictions. @@ -83,13 +83,13 @@ Qwen Code's `web_fetch` tool implements support for [Cloudflare's Markdown for A ### How it works 1. The `format` parameter controls **only** the `Accept` header sent to the server (it does not affect the output format): - - `format="auto"`: sends `Accept: text/markdown, text/html` - - `format="markdown"`: sends `Accept: text/markdown` - - `format="html"`: sends `Accept: text/html` - - `format="text"`: sends `Accept: text/plain` + - `format="auto"`: sends `Accept: text/markdown, text/html;q=0.9, text/plain;q=0.8, */*;q=0.1` + - `format="markdown"`: sends `Accept: text/markdown, */*;q=0.1` + - `format="html"`: sends `Accept: text/html, */*;q=0.1` + - `format="text"`: sends `Accept: text/plain, */*;q=0.1` 2. If the server supports markdown, it returns `Content-Type: text/markdown` 3. The tool uses markdown or plain text content directly without conversion -4. If the server returns HTML, it converts to readable text format for LLM processing +4. If the server returns HTML, it converts to readable text format for LLM processing; markdown, plain text, and fallback content types such as JSON are used as-is 5. All content is normalized to text before being processed by the AI model ### Benefits diff --git a/packages/core/src/tools/web-fetch.test.ts b/packages/core/src/tools/web-fetch.test.ts index 6508423d08..768493ede1 100644 --- a/packages/core/src/tools/web-fetch.test.ts +++ b/packages/core/src/tools/web-fetch.test.ts @@ -124,11 +124,13 @@ describe('WebFetchTool', () => { expect(fetchSpy).toHaveBeenCalledWith( 'https://example.com', expect.any(Number), - { Accept: 'text/markdown, text/html, text/plain' }, + { + Accept: 'text/markdown, text/html;q=0.9, text/plain;q=0.8, */*;q=0.1', + }, ); }); - it('should request only markdown when format is markdown', async () => { + it('should prefer markdown when format is markdown', async () => { const fetchSpy = vi .spyOn(fetchUtils, 'fetchWithTimeout') .mockResolvedValue({ @@ -153,11 +155,11 @@ describe('WebFetchTool', () => { expect(fetchSpy).toHaveBeenCalledWith( 'https://example.com', expect.any(Number), - { Accept: 'text/markdown' }, + { Accept: 'text/markdown, */*;q=0.1' }, ); }); - it('should request only HTML when format is html', async () => { + it('should prefer HTML when format is html', async () => { const fetchSpy = vi .spyOn(fetchUtils, 'fetchWithTimeout') .mockResolvedValue({ @@ -182,11 +184,11 @@ describe('WebFetchTool', () => { expect(fetchSpy).toHaveBeenCalledWith( 'https://example.com', expect.any(Number), - { Accept: 'text/html' }, + { Accept: 'text/html, */*;q=0.1' }, ); }); - it('should request plain text when format is text', async () => { + it('should prefer plain text when format is text', async () => { const fetchSpy = vi .spyOn(fetchUtils, 'fetchWithTimeout') .mockResolvedValue({ @@ -211,10 +213,44 @@ describe('WebFetchTool', () => { expect(fetchSpy).toHaveBeenCalledWith( 'https://example.com', expect.any(Number), - { Accept: 'text/plain' }, + { Accept: 'text/plain, */*;q=0.1' }, ); }); + it('should process JSON content returned by fallback content negotiation', async () => { + let receivedContent = ''; + vi.spyOn(fetchUtils, 'fetchWithTimeout').mockResolvedValue({ + ok: true, + headers: new Headers({ 'content-type': 'application/json' }), + text: () => + Promise.resolve( + JSON.stringify({ + published_at: '2026-01-27T11:50:52Z', + body: '

Release notes

', + desc: 'Use & for ampersand', + }), + ), + } as Response); + + mockGenerateContent.mockImplementation((options) => { + receivedContent = options.contents[0].parts[0].text; + return Promise.resolve({ text: 'Processed', usage: undefined }); + }); + + const tool = new WebFetchTool(mockConfig); + const params = { + url: 'https://api.github.com/repos/openai/codex/releases/tags/rust-v0.92.0', + prompt: 'report the published date', + }; + const invocation = tool.build(params); + await invocation.execute(new AbortController().signal); + + expect(receivedContent).toContain('published_at'); + expect(receivedContent).toContain('2026-01-27T11:50:52Z'); + expect(receivedContent).toContain('

Release notes

'); + expect(receivedContent).toContain('Use & for ampersand'); + }); + it('should include markdown content in prompt when server returns markdown', async () => { let receivedContent = ''; vi.spyOn(fetchUtils, 'fetchWithTimeout').mockResolvedValue({ diff --git a/packages/core/src/tools/web-fetch.ts b/packages/core/src/tools/web-fetch.ts index bba4469d1e..562c23654b 100644 --- a/packages/core/src/tools/web-fetch.ts +++ b/packages/core/src/tools/web-fetch.ts @@ -40,9 +40,9 @@ export interface WebFetchToolParams { * Preferred content format (controls only the Accept header) * All content is normalized to plain text for LLM processing * - auto: Prefers markdown via content negotiation (default) - * - markdown: Request markdown format only - * - html: Request HTML format only (still converted to text) - * - text: Request plain text format + * - markdown: Prefer markdown format + * - html: Prefer HTML format (still converted to text) + * - text: Prefer plain text format */ format?: 'auto' | 'markdown' | 'html' | 'text'; } @@ -68,14 +68,14 @@ class WebFetchToolInvocation extends BaseToolInvocation< const format = this.params.format ?? 'auto'; switch (format) { case 'markdown': - return 'text/markdown'; + return 'text/markdown, */*;q=0.1'; case 'html': - return 'text/html'; + return 'text/html, */*;q=0.1'; case 'text': - return 'text/plain'; + return 'text/plain, */*;q=0.1'; case 'auto': default: - return 'text/markdown, text/html, text/plain'; + return 'text/markdown, text/html;q=0.9, text/plain;q=0.8, */*;q=0.1'; } } @@ -124,15 +124,20 @@ class WebFetchToolInvocation extends BaseToolInvocation< } else if (contentType.includes('text/plain')) { this.debugLogger.debug('[WebFetchTool] Received plain text content'); textContent = responseText.substring(0, MAX_CONTENT_LENGTH); - } else { + } else if (contentType.includes('text/html')) { this.debugLogger.debug('[WebFetchTool] Converting HTML to text'); - textContent = convert(responseText, { + textContent = convert(responseText.substring(0, MAX_CONTENT_LENGTH), { wordwrap: false, selectors: [ { selector: 'a', options: { ignoreHref: true } }, { selector: 'img', format: 'skip' }, ], - }).substring(0, MAX_CONTENT_LENGTH); + }); + } else { + this.debugLogger.debug( + `[WebFetchTool] Passing through ${contentType || 'unknown'} content as text`, + ); + textContent = responseText.substring(0, MAX_CONTENT_LENGTH); } this.debugLogger.debug( @@ -269,7 +274,7 @@ export class WebFetchTool extends BaseDeclarativeTool< super( WebFetchTool.Name, ToolDisplayNames.WEB_FETCH, - 'Fetches content from a specified URL and processes it using an AI model\n- Takes a URL and a prompt as input\n- Supports content negotiation for markdown (reduces tokens by ~80%)\n- Fetches the URL content, converts HTML to text if needed\n- Processes the content with the prompt using a small, fast model\n- Returns the model\'s response about the content\n- Use this tool when you need to retrieve and analyze web content\n\nUsage notes:\n - IMPORTANT: If an MCP-provided web fetch tool is available, prefer using that tool instead of this one, as it may have fewer restrictions. All MCP-provided tools start with "mcp__".\n - The URL must be a fully-formed valid URL\n - The prompt should describe what information you want to extract from the page\n - format parameter (optional): controls only the Accept header sent to the server. All content is normalized to plain text for LLM processing, regardless of format.\n - "auto" (default): Prefers markdown via content negotiation, accepts HTML as fallback. Use when user does NOT specify a format.\n - "markdown": Sends Accept: text/markdown. Use when user explicitly asks for markdown content.\n - "html": Sends Accept: text/html. Content is still converted to plain text for LLM processing.\n - "text": Sends Accept: text/plain. Use when user explicitly asks for plain text.\n - This tool is read-only and does not modify any files\n - Results may be summarized if the content is very large\n - Supports both public and private/localhost URLs using direct fetch', + 'Fetches content from a specified URL and processes it using an AI model\n- Takes a URL and a prompt as input\n- Supports content negotiation for markdown (reduces tokens by ~80%)\n- Fetches the URL content, converts HTML to text if needed\n- Processes the content with the prompt using a small, fast model\n- Returns the model\'s response about the content\n- Use this tool when you need to retrieve and analyze web content\n\nUsage notes:\n - IMPORTANT: If an MCP-provided web fetch tool is available, prefer using that tool instead of this one, as it may have fewer restrictions. All MCP-provided tools start with "mcp__".\n - The URL must be a fully-formed valid URL\n - The prompt should describe what information you want to extract from the page\n - format parameter (optional): controls only the Accept header sent to the server. All content is normalized to plain text for LLM processing, regardless of format.\n - "auto" (default): Prefers markdown via content negotiation, accepts HTML, text, or other content as fallback. Use when user does NOT specify a format.\n - "markdown": Prefers text/markdown. Use when user explicitly asks for markdown content.\n - "html": Prefers text/html. Content is still converted to plain text for LLM processing.\n - "text": Prefers text/plain. Use when user explicitly asks for plain text.\n - This tool is read-only and does not modify any files\n - Results may be summarized if the content is very large\n - Supports both public and private/localhost URLs using direct fetch', Kind.Fetch, { properties: {