fix(acp): include stored attachment paths in model context (#13668)

Co-authored-by: 钉萁 <dingqi.jww@alibaba-inc.com>
This commit is contained in:
ytahdn 2026-10-08 11:35:28 +00:00 • committed by GitHub
parent bb66a52c96
commit 95f6aa9c0f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 522 additions and 17 deletions

View file

@ -0,0 +1,81 @@
# Attachment paths in model context
[English](attachment-path-context.md) | [简体中文](attachment-path-context.zh-CN.md)
## Problem and scope
Uploaded files are stored on disk, but dispatch only supplies their contents and
an `attachment:///` identifier (images only carry bytes and MIME type). A model
that needs the original file for a script must discover its location, sometimes
using memory. This fix supplies the location independently of memory. It does
not add Excel parsing, change browser previews, or add a new tool.
## Design
The session attachment store adds a model-context string to the existing
content block's `_meta[DAEMON_ATTACHMENT_CONTEXT_META_KEY]`. It contains the
stored name, attachment URI, MIME type, and absolute path as JSON, plus a short
instruction to use that path with tools running on the daemon host. Names and
paths are JSON encoded; attachment contents remain user data. Images retain
their inline bytes and text files retain their contents.
`Session` adds this context to the model's reference parts, including the
image-only path, without merging it into the user's text. There remains exactly
one content block per attachment reference. User display text, title input, and
mid-turn attachment-reference persistence keep their existing behavior.
Vision Bridge derives its focus hint from the resolved text with attachment
path metadata excluded. This keeps its 2,000-character hint budget available
for the user's question, expanded commands, and audio transcription. The main
model still receives the path context. Normal, custom-command, and mid-turn
delivery use the same filtering.
Obtain the path from the same successful read that supplies the bytes. Do not
reconstruct the default directory: a configured root may fall back to legacy
storage. Keep public attachment download responses unchanged. Generate metadata
when a stored reference is dispatched, including references after a session
restore, instead of persisting a separate path mapping.
Normal prompt dispatch and mid-turn queue draining already share the resolver.
Both retain the new metadata on surviving attachment blocks during degradation.
Duplicate references share one read and encoding, and failed reads are still
evicted from the dispatch memo.
## Constraints
Paths belong to the daemon/ACP execution filesystem, not the browser's machine.
The daemon launches ACP children locally, but SSH or Managed sessions may
delegate tools to another filesystem. Metadata explicitly identifies the daemon
host and does not claim remote tools can access its files. ACP currently rejects
the separate tool execution sandbox. This change does not add remote filesystem
transport or sandbox mounts. Existing tool permissions still apply. A missing
attachment keeps the existing failure/degradation behavior; never search another
session for a replacement. Raw inline content without a stored attachment
reference has no new path metadata.
Absolute paths are sent to the configured model provider as prompt content.
The default path can reveal the daemon user's home-directory name; a custom
root can reveal its directory hierarchy. Paths are not anonymized. A private
attachment alias resolved by tools would require a separate design.
Providing a path does not add a parser for unsupported binary formats. A model
can use an available parser through shell tools. File removal after dispatch can
still invalidate a previously supplied path.
## Validation and acceptance
- In a fresh session with memory disabled, image, text, and binary attachments
include absolute paths in the final model request. Reading those paths returns
the original bytes; image content remains available to a vision model.
- Configured roots, fallback roots, restored stores, duplicate filenames, and
names containing spaces or Unicode report the correct stored file.
- Unavailable attachments report failure or degrade as before; surviving
attachments keep both metadata and content during mid-turn delivery.
- Existing content, memoization, download, and session isolation tests pass.
- Normal and image-only mid-turn messages retain their original display text
and attachment references; paths appear only in model context.
- Long attachment metadata does not consume the Vision Bridge focus-hint
budget or displace the user's question; the main model retains the paths.
- Build, typecheck, targeted unit tests, and an isolated Store → Session → OpenAI
request-conversion probe pass. The probe does not contact a model or load
personal memory; it verifies request construction and file access.

View file

@ -0,0 +1,66 @@
# 模型上下文中的附件路径
[English](attachment-path-context.md) | [简体中文](attachment-path-context.zh-CN.md)
## 问题与范围
上传文件已保存在磁盘上,但分发时只提供内容和 `attachment:///` 标识
(图片仅携带字节和 MIME 类型)。模型需要通过脚本处理原文件时,必须自行
寻找位置,有时依赖记忆。本修复直接提供位置,不依赖记忆;不增加 Excel
解析、不修改浏览器预览,也不新增工具。
## 设计
会话附件存储在原内容块的 `_meta[DAEMON_ATTACHMENT_CONTEXT_META_KEY]` 中
添加模型上下文字符串,包含以 JSON 编码的存储名称、附件 URI、MIME 类型和
绝对路径,并说明使用该路径调用 daemon 主机上的工具。名称和路径使用 JSON
编码;附件内容仍属于用户数据。图片保留内联字节,文本文件保留原有内容。
`Session` 将该上下文加入模型的引用内容,包括仅有图片的消息,不合并到用户
正文中。每个附件引用仍对应一个内容块,用户展示文本、标题输入以及回合中
附件引用的持久化行为保持不变。
Vision Bridge 从解析后的文本生成解读提示时,排除附件路径元数据,避免其
占用 2,000 字符预算而挤掉用户问题、展开后的命令或语音转录。主模型仍然
接收路径上下文。普通发送、自定义命令和回合中投递共用这一过滤逻辑。
路径取自提供文件字节的同一次成功读取。不推算默认目录:自定义根目录可能
回退到旧存储。对外附件下载响应保持不变。存储引用分发时生成元数据,包括
恢复会话后分发的引用,不单独持久化一份路径映射。
正常提示分发和回合中消息队列消费已经共用解析器,降级时两者都保留有效
附件内容块中的元数据。重复引用继续共用一次读取和编码,读取失败后仍从
分发缓存中移除。
## 约束
路径属于 daemon/ACP 执行文件系统,不属于浏览器所在机器。受支持的 daemon
在本地启动 ACP 子进程,但 SSH 或 Managed 会话可能把工具委派到另一个文件
系统。元数据明确标注 daemon 主机,不声称远程工具可以访问该文件。ACP 当前
拒绝独立的工具执行沙箱。本修改不增加远程文件系统传输或沙箱挂载。现有
工具权限继续生效。附件缺失时保留现有失败或降级行为,不能到其他会话寻找
替代文件。没有存储附件引用的原始内联内容不
添加路径元数据。
绝对路径会作为提示内容发送给配置的模型服务商。默认路径可能暴露 daemon
用户的主目录名称,自定义根目录可能暴露其目录层级;当前方案不做路径匿名化。
若要使用由工具解析的私有附件别名,需要另行设计。
提供路径不会增加对不支持的二进制格式的解析器。模型可以通过 shell 工具
调用可用的解析器。分发后删除文件仍可能使此前提供的路径失效。
## 验证与验收
- 在禁用记忆的新会话中,图片、文本和二进制附件的绝对路径出现在最终模型
请求中;通过这些路径读取到的字节与原文件一致;视觉模型仍可接收图片内容。
- 自定义根目录、备用根目录、恢复后的存储、重名文件以及包含空格或 Unicode
的名称都指向正确的存储文件。
- 不可用附件继续按原有语义报错或降级;回合中投递时,有效附件同时保留
元数据和内容。
- 现有内容、读取缓存、下载和会话隔离测试通过。
- 正常消息和回合中仅含图片的消息保留原展示文本及附件引用;路径仅进入模型
上下文。
- 较长的附件元数据不会占用 Vision Bridge 的解读提示预算或挤掉用户问题;
主模型仍然保留路径。
- 构建、类型检查、定向单元测试以及隔离的 Store → Session → OpenAI 请求
转换探针通过。探针不调用模型、不加载个人记忆,验证请求构造和文件访问。

View file

@ -17642,7 +17642,15 @@ describe('createAcpSessionBridge', () => {
);
expect(prompts[0]?.prompt).toEqual([
{ type: 'image', data: 'AQID', mimeType: 'image/png' },
{
type: 'image',
data: 'AQID',
mimeType: 'image/png',
_meta: {
'qwen.daemon.attachmentContext':
expect.stringContaining('"absolutePath":'),
},
},
]);
expect(prompts[0]?._meta?.['qwen.daemon.attachmentReferences']).toEqual([
reference,
@ -17772,6 +17780,10 @@ describe('createAcpSessionBridge', () => {
expect(prompts[0]?.prompt).toEqual([
{
type: 'resource',
_meta: {
'qwen.daemon.attachmentContext':
expect.stringContaining('"absolutePath":'),
},
resource: {
uri: 'attachment:///notes.txt',
mimeType: 'text/plain',
@ -17780,6 +17792,10 @@ describe('createAcpSessionBridge', () => {
},
{
type: 'resource',
_meta: {
'qwen.daemon.attachmentContext':
expect.stringContaining('"absolutePath":'),
},
resource: {
uri: 'attachment:///report.pdf',
mimeType: 'application/pdf',
@ -39094,7 +39110,15 @@ describe('createAcpSessionBridge — mid-turn message queue (enqueueMidTurnMessa
type: 'text',
text: 'two images\n[Attachment is no longer available]',
},
{ type: 'image', data: 'AQI=', mimeType: 'image/png' },
{
type: 'image',
data: 'AQI=',
mimeType: 'image/png',
_meta: {
'qwen.daemon.attachmentContext':
expect.stringContaining('"absolutePath":'),
},
},
]);
releases[1]!();
await vi.waitFor(() =>
@ -39174,7 +39198,15 @@ describe('createAcpSessionBridge — mid-turn message queue (enqueueMidTurnMessa
type: 'text',
text: 'm2\n[Attachment is no longer available]',
},
{ type: 'image', data: 'AQI=', mimeType: 'image/png' },
{
type: 'image',
data: 'AQI=',
mimeType: 'image/png',
_meta: {
'qwen.daemon.attachmentContext':
expect.stringContaining('"absolutePath":'),
},
},
]);
releases[2]!();
await vi.waitFor(() =>

View file

@ -4155,12 +4155,13 @@ describe('BridgeClient — mid-turn queue drain (craft/drainMidTurnQueue)', () =
it('degrades a attachmentId reused across drained messages after its first use', async () => {
const publish = vi.fn().mockReturnValue(true);
const media = new SessionAttachmentStore();
const read = vi.spyOn(fsp, 'readFile');
try {
const reference = await media.putAttachment(
Uint8Array.of(1, 2, 3),
'image/png',
);
const read = vi.spyOn(media, 'read');
read.mockClear();
const entry = {
sessionId: 'sess:shared-media',
midTurnMessageQueue: [
@ -4217,6 +4218,7 @@ describe('BridgeClient — mid-turn queue drain (craft/drainMidTurnQueue)', () =
// serialized, so one stored blob cannot amplify the drain response.
expect(read).toHaveBeenCalledTimes(1);
} finally {
read.mockRestore();
await media.close();
}
});

View file

@ -1418,6 +1418,8 @@ export const DAEMON_PASSIVE_MANAGED_RUNTIME_RECOVERY_META_KEY =
'qwen.daemon.passiveManagedRuntimeRecovery';
export const DAEMON_ATTACHMENT_REFERENCES_META_KEY =
'qwen.daemon.attachmentReferences';
export const DAEMON_ATTACHMENT_CONTEXT_META_KEY =
'qwen.daemon.attachmentContext';
export const MAX_TRUSTED_MODEL_PROMPT_CHARS = 64 * 1024;
export function isValidTrustedModelPrompt(value: unknown): value is string {

View file

@ -8,6 +8,7 @@ import { promises as fs, statSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import * as path from 'node:path';
import type { ContentBlock } from '@agentclientprotocol/sdk';
import { DAEMON_ATTACHMENT_CONTEXT_META_KEY } from './bridgeTypes.js';
import { describe, expect, it, vi } from 'vitest';
import {
SESSION_ATTACHMENT_UNAVAILABLE_TEXT,
@ -28,7 +29,102 @@ vi.mock('node:fs', async (importOriginal) => {
};
});
const attachmentMeta = {
[DAEMON_ATTACHMENT_CONTEXT_META_KEY]:
expect.stringContaining('"absolutePath":'),
};
describe('SessionAttachmentStore', () => {
it.each([
['notes 空格.txt', 'text/plain', Buffer.from('attachment text')],
['table.xlsx', 'application/octet-stream', Buffer.from([0, 255, 1])],
['image.png', 'image/png', Buffer.from([1, 2, 3])],
])(
'supplies the actual stored path for %s after restore',
async (name, mimeType, data) => {
const root = await fs.mkdtemp(path.join(tmpdir(), 'qwen-path-context-'));
const sessionId = 'restored';
const store = new SessionAttachmentStore(root, sessionId);
const restored = new SessionAttachmentStore(root, sessionId);
try {
const first = await store.putAttachment(data, mimeType, name);
const duplicate = await store.putAttachment(data, mimeType, name);
expect(duplicate.attachmentId).not.toBe(first.attachmentId);
await store.close();
for (const reference of [first, duplicate]) {
const [content] = await restored.resolveContent([reference]);
const context = content?._meta?.[DAEMON_ATTACHMENT_CONTEXT_META_KEY];
if (typeof context !== 'string')
throw new Error('Missing attachment context');
const absolutePath = path.join(
root,
`session-${sessionId}`,
reference.attachmentId,
);
expect(JSON.parse(context.split('\n')[1]!)).toEqual({
name: reference.attachmentId,
uri: `attachment:///${encodeURIComponent(reference.attachmentId)}`,
mimeType,
absolutePath,
});
expect(path.isAbsolute(absolutePath)).toBe(true);
expect(await fs.readFile(absolutePath)).toEqual(data);
expect(content?.type).toBe(reference.type);
}
const otherSession = new SessionAttachmentStore(root, 'other');
await expect(otherSession.resolveContent([first])).rejects.toThrow(
'Unknown or unavailable',
);
await otherSession.close();
} finally {
await store.close();
await restored.close();
await fs.rm(root, { recursive: true, force: true });
}
},
);
it.each(['missing', 'unreadable'] as const)(
'reports the fallback path when the primary is %s',
async (primaryState) => {
const root = await fs.mkdtemp(
path.join(tmpdir(), 'qwen-fallback-context-'),
);
const primary = path.join(root, 'primary');
const fallback = path.join(root, 'legacy');
const oldStore = new SessionAttachmentStore(fallback, 'test');
const store = new SessionAttachmentStore(primary, 'test', fallback);
const readFile = vi.spyOn(fs, 'readFile');
try {
const reference = await oldStore.putAttachment(
Buffer.from('legacy'),
'text/plain',
'notes.txt',
);
await oldStore.close();
if (primaryState === 'unreadable') {
readFile.mockRejectedValueOnce(
Object.assign(new Error('unreadable'), { code: 'EACCES' }),
);
}
const [content] = await store.resolveContent([reference]);
const context = content?._meta?.[DAEMON_ATTACHMENT_CONTEXT_META_KEY];
if (typeof context !== 'string')
throw new Error('Missing attachment context');
const { absolutePath } = JSON.parse(context.split('\n')[1]!);
expect(absolutePath).toBe(
path.join(fallback, 'session-test', reference.attachmentId),
);
expect(await fs.readFile(absolutePath, 'utf8')).toBe('legacy');
} finally {
readFile.mockRestore();
await oldStore.close();
await store.close();
await fs.rm(root, { recursive: true, force: true });
}
},
);
it.each(['close', 'delete'] as const)(
'invalidates staged uploads on %s',
async (operation) => {
@ -405,7 +501,12 @@ describe('SessionAttachmentStore', () => {
size: 3,
});
expect(await store.resolveContent([reference])).toEqual([
{ type: 'image', data: 'AQID', mimeType: 'image/png' },
{
type: 'image',
data: 'AQID',
mimeType: 'image/png',
_meta: attachmentMeta,
},
]);
expect(await store.read(reference.attachmentId)).toEqual({
data: Buffer.from([1, 2, 3]),
@ -436,6 +537,7 @@ describe('SessionAttachmentStore', () => {
expect(await store.resolveContent([reference])).toEqual([
{
type: 'resource',
_meta: attachmentMeta,
resource: {
uri: 'attachment:///notes.txt',
mimeType: 'text/plain',
@ -467,6 +569,7 @@ describe('SessionAttachmentStore', () => {
expect(await store.resolveContent([reference])).toEqual([
{
type: 'resource',
_meta: attachmentMeta,
resource: {
uri: 'attachment:///report.pdf',
mimeType: 'application/pdf',
@ -495,6 +598,7 @@ describe('SessionAttachmentStore', () => {
expect(await store.resolveContent([reference])).toEqual([
{
type: 'resource',
_meta: attachmentMeta,
resource: {
uri: `attachment:///${name}`,
mimeType,
@ -519,6 +623,7 @@ describe('SessionAttachmentStore', () => {
expect(await store.resolveContent([reference])).toEqual([
{
type: 'resource',
_meta: attachmentMeta,
resource: {
uri: 'attachment:///payload.unknown',
mimeType: 'application/octet-stream',
@ -548,6 +653,7 @@ describe('SessionAttachmentStore', () => {
expect(await store.resolveContent([reference])).toEqual([
{
type: 'resource',
_meta: attachmentMeta,
resource: {
uri: 'attachment:///diagram.svg',
mimeType: 'image/svg+xml',
@ -580,9 +686,24 @@ describe('SessionAttachmentStore', () => {
]);
expect(resolved).toEqual([
{ type: 'image', data: 'AQID', mimeType: 'image/png' },
{ type: 'image', data: 'AQID', mimeType: 'image/png' },
{ type: 'image', data: 'AQID', mimeType: 'image/png' },
{
type: 'image',
data: 'AQID',
mimeType: 'image/png',
_meta: attachmentMeta,
},
{
type: 'image',
data: 'AQID',
mimeType: 'image/png',
_meta: attachmentMeta,
},
{
type: 'image',
data: 'AQID',
mimeType: 'image/png',
_meta: attachmentMeta,
},
]);
expect(readFile).toHaveBeenCalledTimes(1);
} finally {
@ -603,6 +724,7 @@ describe('SessionAttachmentStore', () => {
const memo = new Map<string, Promise<ContentBlock>>();
const block = {
_meta: attachmentMeta,
type: 'image',
data: 'AQID',
mimeType: 'image/png',
@ -1515,6 +1637,7 @@ describe('SessionAttachmentStore', () => {
await expect(restored.resolveContent([typescript])).resolves.toEqual([
{
type: 'resource',
_meta: attachmentMeta,
resource: {
uri: 'attachment:///example.ts',
mimeType: 'text/plain',
@ -1666,7 +1789,12 @@ describe('SessionAttachmentStore', () => {
// The failed entry must not stay cached: the next resolution re-reads
// from disk and succeeds.
await expect(store.resolveContent([reference], memo)).resolves.toEqual([
{ type: 'image', data: 'CQk=', mimeType: 'image/png' },
{
type: 'image',
data: 'CQk=',
mimeType: 'image/png',
_meta: attachmentMeta,
},
]);
expect(readFile).toHaveBeenCalledTimes(2);
} finally {
@ -1689,7 +1817,12 @@ describe('SessionAttachmentStore', () => {
expect(result.retainedBlocks).toEqual([text, live]);
expect(result.resolvedBlocks).toEqual([
text,
{ type: 'image', data: 'AQI=', mimeType: 'image/png' },
{
type: 'image',
data: 'AQI=',
mimeType: 'image/png',
_meta: attachmentMeta,
},
]);
} finally {
await store.close();

View file

@ -8,6 +8,7 @@ import { constants as fsConstants, promises as fs, statSync } from 'node:fs';
import { tmpdir } from 'node:os';
import * as path from 'node:path';
import type { ContentBlock } from '@agentclientprotocol/sdk';
import { DAEMON_ATTACHMENT_CONTEXT_META_KEY } from './bridgeTypes.js';
import { getSpecificMimeType } from '@qwen-code/qwen-code-core';
import {
SessionAttachmentUploads,
@ -38,6 +39,12 @@ interface DurableAttachmentDirectory {
inodeVerifiable: boolean;
}
interface StoredAttachment {
data: Buffer;
mimeType: string;
absolutePath: string;
}
function hasVerifiableInode(ino: number): boolean {
return Number.isSafeInteger(ino) && ino > 0;
}
@ -742,6 +749,14 @@ export class SessionAttachmentStore {
async read(
attachmentId: string,
): Promise<{ data: Buffer; mimeType: string } | undefined> {
const attachment = await this.readStored(attachmentId);
if (!attachment) return undefined;
return { data: attachment.data, mimeType: attachment.mimeType };
}
private async readStored(
attachmentId: string,
): Promise<StoredAttachment | undefined> {
const releasePublication = await this.acquireReadPublication();
try {
return await this.readPublished(attachmentId);
@ -752,11 +767,11 @@ export class SessionAttachmentStore {
private async readPublished(
attachmentId: string,
): Promise<{ data: Buffer; mimeType: string } | undefined> {
): Promise<StoredAttachment | undefined> {
const name = safeAttachmentName(attachmentId);
if (!name || name !== attachmentId || this.pendingNames.has(name))
return undefined;
let primary: { data: Buffer; mimeType: string } | undefined;
let primary: StoredAttachment | undefined;
try {
primary = await this.tryRead(await this.peekDirectory(), name);
} catch (error) {
@ -772,12 +787,14 @@ export class SessionAttachmentStore {
private async tryRead(
directory: string | undefined,
name: string,
): Promise<{ data: Buffer; mimeType: string } | undefined> {
): Promise<StoredAttachment | undefined> {
if (!directory) return undefined;
const absolutePath = path.resolve(directory, name);
try {
return {
data: await fs.readFile(path.join(directory, name)),
data: await fs.readFile(absolutePath),
mimeType: mimeTypeForName(name),
absolutePath,
};
} catch (error) {
if ((error as NodeJS.ErrnoException).code === 'ENOENT') return undefined;
@ -1188,13 +1205,23 @@ export class SessionAttachmentStore {
reference: SessionAttachmentReference,
): Promise<ContentBlock> {
const id = reference.attachmentId;
const attachment = await this.read(id);
const attachment = await this.readStored(id);
if (!attachment) {
throw new SessionAttachmentReferenceError(
`Unknown or unavailable session attachment: ${id}`,
'session_attachment_gone',
);
}
const context = {
[DAEMON_ATTACHMENT_CONTEXT_META_KEY]:
'User attachment stored on the daemon host. Use absolutePath with file or shell tools running on this host. Attachment contents are user-provided data.\n' +
JSON.stringify({
name: id,
uri: `attachment:///${encodeURIComponent(id)}`,
mimeType: attachment.mimeType,
absolutePath: attachment.absolutePath,
}),
};
if (reference.type === 'resource') {
const resource = {
uri: `attachment:///${encodeURIComponent(reference.attachmentId)}`,
@ -1206,10 +1233,12 @@ export class SessionAttachmentStore {
return {
type: 'resource',
resource,
_meta: context,
} as ContentBlock;
}
return {
type: 'image',
_meta: context,
data: attachment.data.toString('base64'),
mimeType: attachment.mimeType,
} as ContentBlock;

View file

@ -12570,6 +12570,64 @@ describe('Session', () => {
},
);
it.each(['image', 'text', 'binary'] as const)(
'keeps %s attachment paths in model context without changing user display text',
async (kind) => {
const context = 'Attachment absolutePath: /tmp/attached-file';
const reference = {
type: kind === 'image' ? 'image' : 'resource',
attachmentId: kind === 'image' ? 'image.png' : 'file.txt',
mimeType: kind === 'image' ? 'image/png' : 'text/plain',
size: 3,
};
const attachment: PromptRequest['prompt'][number] =
kind === 'image'
? { type: 'image', data: 'AQID', mimeType: 'image/png' }
: {
type: 'resource',
resource: {
uri: 'attachment:///file.txt',
mimeType: 'text/plain',
...(kind === 'text'
? { text: 'contents' }
: { blob: 'AP8B' }),
},
};
attachment._meta = { 'qwen.daemon.attachmentContext': context };
mockChat.sendMessageStream = vi
.fn()
.mockResolvedValue(createEmptyStream());
// The image case has no user text or resource block to avoid either
// image-only early return hiding the model context.
const userText = kind === 'image' ? '' : 'inspect this';
await session.prompt({
sessionId: 'test-session-id',
prompt: [
...(userText ? [{ type: 'text' as const, text: userText }] : []),
attachment,
],
_meta: { 'qwen.daemon.attachmentReferences': [reference] },
});
expect(firstSentMessage()).toContainEqual({ text: context });
if (kind === 'image') {
expect(firstSentMessage()).toContainEqual({
inlineData: { data: 'AQID', mimeType: 'image/png' },
});
}
expect(mockChatRecordingService.recordUserMessage).toHaveBeenCalledWith(
userText,
undefined,
expect.objectContaining({
displayText: userText,
attachmentReferences: [reference],
}),
expect.any(String),
undefined,
);
},
);
it('records daemon attachment references for transcript replay', async () => {
const imageReference = {
type: 'image' as const,
@ -17965,6 +18023,65 @@ describe('Session', () => {
expect(sent.some((part) => 'inlineData' in part)).toBe(false);
});
it.each(['normal', 'custom command', 'image-only'] as const)(
'keeps attachment paths outside the vision bridge focus for %s prompts',
async (kind) => {
const question = kind === 'image-only' ? '' : 'Read the serial number.';
const context = `Attachment absolutePath: /tmp/${'a'.repeat(2100)}`;
mockConfig.getEffectiveInputModalities = vi.fn().mockReturnValue({});
mockConfig.getDefaultVisionBridgeModel = vi.fn().mockReturnValue({
id: 'qwen3.7-plus',
});
runVisionBridgeSpy.mockResolvedValue({
applied: false,
status: 'skipped',
convertedCount: 0,
omittedCount: 0,
});
mockChat.sendMessageStream = vi
.fn()
.mockResolvedValue(createEmptyStream());
if (kind === 'custom command') {
vi.mocked(
nonInteractiveCliCommands.handleSlashCommand,
).mockResolvedValueOnce({
type: 'submit_prompt',
content: [{ text: question }],
resolvedCommand: { name: 'inspect', kind: CommandKind.FILE },
});
}
await session.prompt({
sessionId: 'test-session-id',
prompt: [
{
type: 'text',
text: kind === 'custom command' ? '/inspect' : question,
},
{
type: 'image',
mimeType: 'image/png',
data: 'iVBORw0KGgo=',
_meta: { 'qwen.daemon.attachmentContext': context },
},
],
});
expect(runVisionBridgeSpy).toHaveBeenCalledOnce();
expect(runVisionBridgeSpy).toHaveBeenCalledWith(
expect.objectContaining({
intentText: question,
parts: expect.arrayContaining([
{ text: context },
{ inlineData: { mimeType: 'image/png', data: 'iVBORw0KGgo=' } },
]),
}),
);
expect(textParts(firstSentMessage())).toContain(context);
if (question) expect(textParts(firstSentMessage())).toContain(question);
},
);
it('routes an agent-capable image prompt for that ACP prompt only', async () => {
const runtimeView = {
contentGenerator: {},
@ -23272,6 +23389,10 @@ describe('Session', () => {
type: 'image',
mimeType: 'image/png',
data: 'cHVyZS1pbWFnZQ==',
_meta: {
'qwen.daemon.attachmentContext':
'Attachment absolutePath: /tmp/image-2.png',
},
},
],
displayText: '',
@ -23375,6 +23496,9 @@ describe('Session', () => {
expect(secondCall?.[1].message).toEqual(
expect.arrayContaining(midTurnParts),
);
expect(secondCall?.[1].message).toContainEqual({
text: 'Attachment absolutePath: /tmp/image-2.png',
});
expect(runVisionBridgeSpy).not.toHaveBeenCalled();
expect(secondCall?.[1].message).not.toEqual(
expect.arrayContaining([
@ -23482,7 +23606,10 @@ describe('Session', () => {
expect(
mockChatRecordingService.recordMidTurnUserMessage,
).toHaveBeenCalledWith(
[{ text: '\n[User message received during tool execution]: ' }],
[
{ text: '\n[User message received during tool execution]: ' },
{ text: 'Attachment absolutePath: /tmp/image-2.png' },
],
'',
undefined,
[

View file

@ -6,6 +6,7 @@
import { createAgentHostToolInvocationGuard } from '@qwen-code/qwen-code-core/agents/workspace-agents/capability.js';
import { isToolCallConcurrencySafe } from '@qwen-code/qwen-code-core/core/coreToolScheduler.js';
import { collectText } from '@qwen-code/qwen-code-core/services/visionBridge/image-part-utils.js';
import {
captureHookExecutionOwner,
runWithHookExecutionOwner,
@ -303,6 +304,7 @@ import {
DAEMON_INPUT_ANNOTATIONS_META_KEY,
DAEMON_PERMISSION_CANCEL_REASON_META_KEY,
DAEMON_PROMPT_DISPLAY_TEXT_META_KEY,
DAEMON_ATTACHMENT_CONTEXT_META_KEY,
IMAGE_ONLY_PROMPT_TEXT,
DAEMON_SUBMITTED_PROMPT_META_KEY,
DAEMON_RESTORE_ASK_USER_QUESTION_META_KEY,
@ -9876,6 +9878,7 @@ export class Session implements SessionContext {
// longer timeouts while remaining cancellable by the real turn.
rawParts = await this.#applyBridgeConversionsIfNeeded(
rawParts,
message.content,
abortSignal,
options.onFullTurnModel,
);
@ -16620,6 +16623,7 @@ export class Session implements SessionContext {
});
return this.#applyBridgeConversionsIfNeeded(
[...attachmentParts, ...expandedPrompt],
attachmentBlocks,
abortSignal,
onFullTurnModel,
);
@ -16807,6 +16811,7 @@ export class Session implements SessionContext {
const sshWorkspace = Boolean(this.config.getExecutionEnvironment?.());
const embeddedContext: EmbeddedResourceResource[] = [];
const attachmentParts: Part[] = [];
const extensionMentions = new Map<string, string>();
const mcpServerMentions = new Map<string, string>();
const textPathSpecsToRead = new Map<string, string>();
@ -16818,11 +16823,20 @@ export class Session implements SessionContext {
? parts
: this.#applyBridgeConversionsIfNeeded(
parts,
message,
abortSignal,
options.onFullTurnModel,
);
const parts = message.map((part) => {
const attachmentContext =
part._meta?.[DAEMON_ATTACHMENT_CONTEXT_META_KEY];
if (
(part.type === 'image' || part.type === 'resource') &&
typeof attachmentContext === 'string'
) {
attachmentParts.push({ text: attachmentContext });
}
switch (part.type) {
case 'text':
if (sshWorkspace) return { text: part.text };
@ -16970,13 +16984,18 @@ export class Session implements SessionContext {
if (
pathSpecsToRead.length === 0 &&
embeddedContext.length === 0 &&
attachmentParts.length === 0 &&
extensionParts.length === 0 &&
mcpServerParts.length === 0
) {
return finish(partsToSend);
}
if (pathSpecsToRead.length === 0 && embeddedContext.length === 0) {
if (
pathSpecsToRead.length === 0 &&
embeddedContext.length === 0 &&
attachmentParts.length === 0
) {
return finish([...partsToSend, ...extensionParts, ...mcpServerParts]);
}
@ -17012,6 +17031,7 @@ export class Session implements SessionContext {
// the "--- Content from ... ---" delimiter labels, not by position, so
// leading with the content is safe.
const referenceParts: Part[] = [
...attachmentParts,
...partsToSend.filter((part) => 'inlineData' in part),
...extensionParts,
...mcpServerParts,
@ -17089,6 +17109,7 @@ export class Session implements SessionContext {
async #applyBridgeConversionsIfNeeded(
originalParts: Part[],
sourceBlocks: ContentBlock[],
abortSignal: AbortSignal,
onFullTurnModel?: (model: string) => boolean,
): Promise<Part[]> {
@ -17127,11 +17148,23 @@ export class Session implements SessionContext {
let bridgeResult: VisionBridgeResult;
try {
const attachmentContexts = new Set(
sourceBlocks
.filter(
(block) => block.type === 'image' || block.type === 'resource',
)
.map((block) => block._meta?.[DAEMON_ATTACHMENT_CONTEXT_META_KEY])
.filter((context) => typeof context === 'string'),
);
debugLogger.debug('vision bridge: gate matched, running conversion');
bridgeResult = await runVisionBridge({
config: this.config,
parts,
signal: abortSignal,
// Stored paths stay in model context, outside the bridge's focus budget.
intentText: collectText(
parts.filter((part) => !attachmentContexts.has(part.text ?? '')),
),
});
} catch (error) {
debugLogger.debug(