2 commits

Author	SHA1	Message	Date
Haozhe	ca5cc76211	fix(tree-sitter-bash): recognize heredocs in character-level balanced scanning (#3503) ... * fix(tree-sitter-bash): recognize heredocs in character-level balanced scanning scanBalancedStatements treated heredoc bodies as ordinary characters, so a stray quote, paren, or backtick inside a heredoc body (for example an apostrophe in a PR body passed through a command substitution) broke the scan and produced ERROR nodes; the resulting hasError made dangerous-command-ask judge the whole command unanalyzable and prompt for approval even in yolo mode. Parse << and <<- delimiters (excluding <<< herestrings) with the same unquoting rules as the token-level heredoc reader, queue pending bodies, and skip them line-wise at newlines. Skip arithmetic $(( ... )) regions via scanBalanced so a left-shift << is never mistaken for a heredoc operator. * fix(agent-core-v2): raise the bash parse wall-clock budget to 500ms A 20ms wall-clock budget could abort an otherwise fine parse under CPU contention, GC pauses, or cold-start JIT, flipping the permission verdict to unanalyzable (spurious approval prompts) or silently dropping AGENTS.md re-reminders. Normal commands parse in well under 1ms; maxNodes stays the deterministic cap, and 500ms remains a backstop against pathological parser loops. * fix(tree-sitter-bash): skip comments and legacy arithmetic during heredoc-aware scanning The heredoc-aware scan queued a heredoc for any << it encountered, including inside comments (echo $(printf x # <<EOF\n)) and legacy arithmetic expansions (echo $(echo $[x << 2]\n)); both are valid bash that parsed cleanly before, and the regression flipped them to hasError and an unanalyzable permission verdict. Skip # comments to end of line when the preceding character starts a new word, and skip ${ ... } / $[ ... ] expansions as balanced units (mirroring skipDollar), so << is only recognized where a redirection operator can actually appear. * fix(tree-sitter-bash): skip word-glued subscripts during heredoc-aware scanning Indexed assignments such as echo $(a[x<<2]=3\n) put arithmetic inside a word-glued [ ... ] subscript; the heredoc-aware scan read the << shift operator there as a heredoc start, regressing valid bash that parsed cleanly before to hasError and an unanalyzable permission verdict. Skip a [ ... ] region as a balanced unit when the bracket immediately follows a word character (subscripts and glued glob classes); a bracket at word start keeps the existing character scan, so real heredocs after words (cat foo[ab]<<EOF) and bare [ command arguments are unaffected. The subscript regression is covered by a unit case only: the reference parser splits subscript arithmetic into binary_expression while this parser keeps it an opaque word, a structural difference that predates this change and has no differential fixture yet. * fix(tree-sitter-bash): skip conditional regions and look through continuations in heredoc-aware scanning Two more character-scan contexts queued bogus heredocs after the heredoc-aware scan: a [[ ... ]] conditional region (echo $( [[ x == @(<<EOF) ]]\n) and regex right-hand sides such as [[ x =~ <<a ]]), and a # preceded by a backslash-newline continuation (echo $(printf foo\<newline>#bar)), where removing the continuation keeps the hash inside the preceding word instead of starting a comment. Both are valid bash that parsed cleanly before. Skip word-start [[ ... ]] as a balanced region (a << inside a conditional is never a heredoc operator; a bracket in argument position keeps the character scan), and walk back over \+newline pairs before classifying a # as a comment. The extglob conditional case joins the differential fixtures; the continuation case is unit-only because the reference parser errors on it. * fix(tree-sitter-bash): scan substitutions as part of heredoc delimiters A heredoc delimiter containing a substitution (echo $(cat <<$(foo)\nbody\n$(foo)\n)) was truncated at the first paren, so the queued delimiter never matched the body closing line and the scan swallowed the rest of the range, regressing valid bash to hasError and an unanalyzable permission verdict. scanHeredocDelimiter now scans $( ), ${ }, $[ ], and backtick regions wholesale as part of the delimiter word (recursing with the heredoc-aware statement scanner for $( )), mirroring how bash treats the whole word as the delimiter. The case stays unit-only because unquoted delimiters hit the already-registered heredoc-content-chunks structural difference with the reference parser.	2026-09-03 15:28:11 +08:00
Haozhe	e556088458	feat(tree-sitter-bash): add a pure-TypeScript bash parser and an agent-core-v2 bashParser service (#2016) ... * feat(tree-sitter-bash): scaffold pure-TypeScript bash parser package Add packages/tree-sitter-bash with the SyntaxNode tree model (UTF-16 code-unit offsets, tree-sitter-bash named-node type names), a parse budget (deadline + node cap) with an aborted ParseResult variant, and a placeholder parse() to be replaced by the real lexer/parser. * feat(tree-sitter-bash): add lexer and core recursive-descent parser Cover the permission-analysis grammar subset: lists, pipelines, commands, words, quotes, expansions, command/process substitution, subshells, the full redirect operator set, heredocs and comments, with tree-sitter-bash named-node type names. Long scan loops check the parse deadline via budget.progress() so large literals cannot exhaust the node cap; parse depth is bounded on all recursion chains. * feat(tree-sitter-bash): support the full bash grammar Add compound commands (if/while/until/for/c-style-for/select/case/ function/compound/do_group), test commands with reference-exact extglob/regex right-hand-side rules, a Pratt expression engine for arithmetic expansion shared across arith/c-for/test modes, arrays and subscripts, declaration/unset commands, ansi-c/translated strings and brace expressions. Reserved words are recognized only in statement position. Case-aware balanced scanning keeps command substitutions intact around case items, expression leftovers are kept as ERROR nodes instead of dropped, and recursion depth is bounded per chain (substitution 150, parse 500, lexer scan 1024). * test(tree-sitter-bash): add differential fixtures, corpus, fuzz and perf suites Turn the ad-hoc wasm comparison work into permanent infrastructure: a differential helper pinning reference-equivalent and known-difference samples against the real tree-sitter-bash wasm (478 match / 79 known-diff fixtures plus the official v0.25.0 corpus), a three-way consistency check between fixtures, the known-difference registry and the README, deterministic seeded fuzz with tree-integrity assertions, and performance smoke tests. Converges 15 further divergence groups found by systematic probing and documents the rest; the full suite is 853 tests in ~4s. * feat(agent-core-v2): add App-scope bashParser service Wrap the pure @moonshot-ai/tree-sitter-bash package as IBashParserService (L1, no dependencies): parse(source, options) returns a wire-safe BashParseResult whose nodes drop the cyclic parent link so trees can cross the RPC boundary. Budget exhaustion surfaces as { ok: false, reason: 'aborted' }, malformed input as hasError — never a throw. * chore: add changeset for the bash parser service * chore: update pnpmDeps hash after adding tree-sitter-bash package * fix(agent-core-v2): register bashParser service with ScopeActivation The registration was written against the removed InstantiationType API (#/_base/di/extensions); switch to ScopeActivation.OnDemand from #/_base/di/scope so the package typechecks and the service registers. * feat(kimi-inspect): add Bash Parser view Add a fourth icon-rail tab that exercises the App-scope bash parser service over the debug RPC surface: a source textarea with a parse budget (timeoutMs / maxNodes), a dropdown of curated examples adapted from the tree-sitter-bash differential fixtures, and an expandable syntax tree with per-node type, UTF-16 range and leaf text, plus hasError / aborted / node-count badges. * fix(agent-core-v2): snapshot bash syntax trees iteratively A long left-associative chain (e.g. an arithmetic expression with a few thousand operands) parses into a tree thousands of levels deep while still within budget; the recursive DTO conversion then overflowed the JS call stack and made parse throw RangeError, breaking the never-throws contract. Convert the tree with an explicit stack, the same approach as the parser's own materialize. * feat(kimi-inspect): add a deep-arithmetic example to the Bash Parser view A thousand-operand left-associative chain fills the textarea with a thousand-level binary_expression tree — the shape that once overflowed the DTO conversion. Deeper chains still parse in-process but cannot cross the JSON RPC transport (V8 call-stack limit in serialization), so the example stays within the wire limit. * chore: update pnpmDeps hash for the rebased lockfile The rebase onto main merged pnpm-lock.yaml, invalidating the recorded fetchPnpmDeps hash; use the hash CI computed for the merged lockfile.	2026-07-28 14:29:08 +08:00