mirror of
https://github.com/razzant/ouroboros.git
synced 2026-10-19 02:02:00 +00:00
docs: explain Terminal-Bench community submission closure
This commit is contained in:
Ouroboros
parent
6f636d384e
commit
28941cc50f
6 changed files with 14 additions and 13 deletions
|
|
@ -138,10 +138,10 @@ Ouroboros has reproducible self-reported state-of-the-art results on Terminal-Be
|
|||
|
||||
| Benchmark | Model | Ouroboros | Comparison | Status | Evidence |
|
||||
|-----------|-------|----------:|------------|--------|----------|
|
||||
| Terminal-Bench 2.1 | Claude Opus-5 high | **86.74%** after zeroing one disclosed reward-hack trial (raw: 86.97%) | Claude Code + Fable 5: 83.8% | Self-reported, submission closed without merge (September 19, 2026) | [submission](https://github.com/harbor-framework/terminal-bench-2-1/pull/175) · [run](https://hub.harborframework.com/jobs/2b145543-edeb-4a3b-b46f-4800310f1182) |
|
||||
| Terminal-Bench 2.1 | Claude Opus-5 high | **86.74%** after zeroing one disclosed reward-hack trial (raw: 86.97%) | Claude Code + Fable 5: 83.8% | Self-reported; community submissions closed | [submission](https://github.com/harbor-framework/terminal-bench-2-1/pull/175) · [run](https://hub.harborframework.com/jobs/2b145543-edeb-4a3b-b46f-4800310f1182) |
|
||||
| Terminal-Bench 2.1 | Claude Opus-4.8 high | **80.22%** | Claude Code: 78.9% | Self-reported, public run | [run](https://hub.harborframework.com/jobs/4b8e244f-8ab0-4d28-8218-7cf346282faa) |
|
||||
| Terminal-Bench 2.1 | GPT-5.5 | **84.3%** | Codex CLI: 83.1% | Self-reported, public run | [run](https://hub.harborframework.com/jobs/f02fd019-23e1-495f-af0a-ebd9a65f3079) |
|
||||
| Terminal-Bench 2.1 | Grok-4.5 | **84.94%** after a reward-hack audit | Cursor CLI: 79.3% · Hermes: 77.53% | Self-reported, submission closed without merge (September 19, 2026) | [submission](https://github.com/harbor-framework/terminal-bench-2-1/pull/146) |
|
||||
| Terminal-Bench 2.1 | Grok-4.5 | **84.94%** after a reward-hack audit | Cursor CLI: 79.3% · Hermes: 77.53% | Self-reported; community submissions closed | [submission](https://github.com/harbor-framework/terminal-bench-2-1/pull/146) |
|
||||
| OSWorld-Verified | Claude Opus-5 | **90.69%** | previous best on the public board: 90.19% | Self-reported, full traces | [full traces](https://huggingface.co/datasets/razzant/ouroboros-osworld-verified-opus5) |
|
||||
| OSWorld-Verified | Claude Sonnet-4.6 | **83.27%** | Pointer: 81.45% | Self-reported, full traces | [full traces](https://huggingface.co/datasets/razzant/ouroboros-osworld-verified-sonnet46) |
|
||||
| CL-Bench | Claude Sonnet-4.6 | **0.2301, rank 1** | previous top: 0.1960 | Self-reported, submission open | [submission](https://github.com/pgasawa/continual-learning-bench/pull/10) · [full traces](https://huggingface.co/datasets/razzant/ouroboros-clbench-traces) |
|
||||
|
|
|
|||
|
|
@ -36,15 +36,15 @@
|
|||
<main class="text-document text-document-wide">
|
||||
<p class="chapter-index">BENCHMARKS / PUBLIC EVIDENCE</p>
|
||||
<h1>Scores, status, and evidence.</h1>
|
||||
<p class="text-lede">These results are self-reported, and each row states what evidence is public. The GAIA row is marked while its scrubbed trace capsule is pending. Upstream submission status stays visible; the Terminal-Bench 2.1 submissions were closed without merge on September 19, 2026. The public runs and supported results remain available.</p>
|
||||
<p class="text-lede">These results are self-reported, and each row states what evidence is public. The GAIA row is marked while its scrubbed trace capsule is pending. Publication does not wait for a leaderboard maintainer to merge submissions. Terminal-Bench 2.1 closed community submissions on September 19, 2026.</p>
|
||||
|
||||
<div class="table-scroll" tabindex="0" role="region" aria-label="Ouroboros benchmark results"><table>
|
||||
<thead><tr><th>Benchmark</th><th>Model</th><th>Ouroboros</th><th>Comparison</th><th>Status and evidence</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Terminal-Bench 2.1</td><td>Claude Opus-5 high</td><td><strong>86.74%</strong> after zeroing one disclosed reward-hack trial (raw: 86.97%)</td><td>Claude Code + Fable 5: 83.8%</td><td>Self-reported with a <a href="https://github.com/harbor-framework/terminal-bench-2-1/pull/175">closed submission (not merged)</a> and a <a href="https://hub.harborframework.com/jobs/2b145543-edeb-4a3b-b46f-4800310f1182">public run</a>.</td></tr>
|
||||
<tr><td>Terminal-Bench 2.1</td><td>Claude Opus-5 high</td><td><strong>86.74%</strong> after zeroing one disclosed reward-hack trial (raw: 86.97%)</td><td>Claude Code + Fable 5: 83.8%</td><td>Self-reported with a <a href="https://github.com/harbor-framework/terminal-bench-2-1/pull/175">submission</a> and a <a href="https://hub.harborframework.com/jobs/2b145543-edeb-4a3b-b46f-4800310f1182">public run</a>; community submissions closed.</td></tr>
|
||||
<tr><td>Terminal-Bench 2.1</td><td>Claude Opus-4.8 high</td><td><strong>80.22%</strong></td><td>Claude Code: 78.9%</td><td>Self-reported with a <a href="https://hub.harborframework.com/jobs/4b8e244f-8ab0-4d28-8218-7cf346282faa">public run</a>.</td></tr>
|
||||
<tr><td>Terminal-Bench 2.1</td><td>GPT-5.5</td><td><strong>84.3%</strong></td><td>Codex CLI: 83.1%</td><td>Self-reported with a <a href="https://hub.harborframework.com/jobs/f02fd019-23e1-495f-af0a-ebd9a65f3079">public run</a>.</td></tr>
|
||||
<tr><td>Terminal-Bench 2.1</td><td>Grok-4.5</td><td><strong>84.94%</strong></td><td>Cursor CLI: 79.3%. Hermes: 77.53%.</td><td>Self-reported after a reward-hack audit, with a <a href="https://github.com/harbor-framework/terminal-bench-2-1/pull/146">closed submission (not merged)</a>.</td></tr>
|
||||
<tr><td>Terminal-Bench 2.1</td><td>Grok-4.5</td><td><strong>84.94%</strong></td><td>Cursor CLI: 79.3%. Hermes: 77.53%.</td><td>Self-reported after a reward-hack audit, with a <a href="https://github.com/harbor-framework/terminal-bench-2-1/pull/146">submission</a>; community submissions closed.</td></tr>
|
||||
<tr><td>OSWorld-Verified</td><td>Claude Opus-5</td><td><strong>90.69%</strong></td><td>Previous best on the public board: 90.19%</td><td>Self-reported with <a href="https://huggingface.co/datasets/razzant/ouroboros-osworld-verified-opus5">public task evidence</a>.</td></tr>
|
||||
<tr><td>OSWorld-Verified</td><td>Claude Sonnet-4.6</td><td><strong>83.27%</strong></td><td>Pointer: 81.45%</td><td>Self-reported with <a href="https://huggingface.co/datasets/razzant/ouroboros-osworld-verified-sonnet46">public task evidence</a>.</td></tr>
|
||||
<tr><td>CL-Bench</td><td>Claude Sonnet-4.6</td><td><strong>0.2301, rank 1</strong></td><td>Previous top: 0.1960</td><td>Self-reported with an <a href="https://github.com/pgasawa/continual-learning-bench/pull/10">open submission</a> and <a href="https://huggingface.co/datasets/razzant/ouroboros-clbench-traces">full traces</a>.</td></tr>
|
||||
|
|
|
|||
|
|
@ -96,7 +96,7 @@
|
|||
<div class="chapter-heading reveal"><p class="chapter-index">02 / EVIDENCE</p><h2 id="bench-title">Measured against the strongest harnesses.</h2><p class="lede">Ouroboros has reproducible self-reported state-of-the-art results on Terminal-Bench 2.1, OSWorld-Verified, and CL-Bench. In those model-matched results, it leads Codex, Claude Code, Cursor, and Hermes. The public SWE-bench Pro matched pair is a statistical tie with Codex CLI. A separate GAIA result is marked as awaiting its scrubbed trace capsule.</p></div>
|
||||
<p class="evidence-status reveal"><strong>Publication status</strong><span>Self-reported results are published with evidence and each submission's status.</span><a href="/benchmarks/">Read the evidence table</a></p>
|
||||
<div class="bench-grid">
|
||||
<figure class="media-panel bench-wide reveal"><img src="/assets/bench-terminal-bench.svg?v=7a4871c0bb" alt="Self-reported Terminal-Bench 2.1 results: Ouroboros 86.74% with Claude Opus-5 after a disclosed reward-hack correction, ahead of Claude Code, Codex, Cursor, and Hermes on matched models" loading="lazy" width="1800" height="1478"><figcaption><span>Terminal-Bench 2.1 · 86.74% corrected · self-reported</span><span><a href="https://github.com/harbor-framework/terminal-bench-2-1/pull/175">Closed submission (not merged)</a> · <a href="https://hub.harborframework.com/jobs/2b145543-edeb-4a3b-b46f-4800310f1182">Full run</a></span></figcaption></figure>
|
||||
<figure class="media-panel bench-wide reveal"><img src="/assets/bench-terminal-bench.svg?v=7a4871c0bb" alt="Self-reported Terminal-Bench 2.1 results: Ouroboros 86.74% with Claude Opus-5 after a disclosed reward-hack correction, ahead of Claude Code, Codex, Cursor, and Hermes on matched models" loading="lazy" width="1800" height="1478"><figcaption><span>Terminal-Bench 2.1 · 86.74% corrected · self-reported</span><span><a href="https://github.com/harbor-framework/terminal-bench-2-1/pull/175">Submission</a> · <a href="https://hub.harborframework.com/jobs/2b145543-edeb-4a3b-b46f-4800310f1182">Full run</a></span></figcaption></figure>
|
||||
<figure class="media-panel reveal"><img src="/assets/bench-osworld.svg?v=26e6cdc7db" alt="Self-reported OSWorld-Verified results: Ouroboros 90.69% with Claude Opus-5 and 83.27% with Claude Sonnet-4.6" loading="lazy" width="1800" height="1000"><figcaption><span>OSWorld-Verified · 90.69% · self-reported</span><span><a href="https://huggingface.co/datasets/razzant/ouroboros-osworld-verified-opus5">Full traces</a></span></figcaption></figure>
|
||||
<figure class="media-panel reveal"><img src="/assets/bench-cl-bench.svg?v=633474d172" alt="Self-reported CL-Bench result: Ouroboros normalized reward 0.2301 with Claude Sonnet-4.6, rank 1" loading="lazy" width="1800" height="760"><figcaption><span>CL-Bench · 0.2301 · self-reported rank #1</span><span><a href="https://github.com/pgasawa/continual-learning-bench/pull/10">Open submission</a> · <a href="https://huggingface.co/datasets/razzant/ouroboros-clbench-traces">Full traces</a></span></figcaption></figure>
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -36,15 +36,15 @@
|
|||
<main class="text-document text-document-wide">
|
||||
<p class="chapter-index">BENCHMARKS / PUBLIC EVIDENCE</p>
|
||||
<h1>Scores, status, and evidence.</h1>
|
||||
<p class="text-lede">These results are self-reported, and each row states what evidence is public. The GAIA row is marked while its scrubbed trace capsule is pending. Upstream submission status stays visible; the Terminal-Bench 2.1 submissions were closed without merge on September 19, 2026. The public runs and supported results remain available.</p>
|
||||
<p class="text-lede">These results are self-reported, and each row states what evidence is public. The GAIA row is marked while its scrubbed trace capsule is pending. Publication does not wait for a leaderboard maintainer to merge submissions. Terminal-Bench 2.1 closed community submissions on September 19, 2026.</p>
|
||||
|
||||
<div class="table-scroll" tabindex="0" role="region" aria-label="Ouroboros benchmark results"><table>
|
||||
<thead><tr><th>Benchmark</th><th>Model</th><th>Ouroboros</th><th>Comparison</th><th>Status and evidence</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Terminal-Bench 2.1</td><td>Claude Opus-5 high</td><td><strong>86.74%</strong> after zeroing one disclosed reward-hack trial (raw: 86.97%)</td><td>Claude Code + Fable 5: 83.8%</td><td>Self-reported with a <a href="https://github.com/harbor-framework/terminal-bench-2-1/pull/175">closed submission (not merged)</a> and a <a href="https://hub.harborframework.com/jobs/2b145543-edeb-4a3b-b46f-4800310f1182">public run</a>.</td></tr>
|
||||
<tr><td>Terminal-Bench 2.1</td><td>Claude Opus-5 high</td><td><strong>86.74%</strong> after zeroing one disclosed reward-hack trial (raw: 86.97%)</td><td>Claude Code + Fable 5: 83.8%</td><td>Self-reported with a <a href="https://github.com/harbor-framework/terminal-bench-2-1/pull/175">submission</a> and a <a href="https://hub.harborframework.com/jobs/2b145543-edeb-4a3b-b46f-4800310f1182">public run</a>; community submissions closed.</td></tr>
|
||||
<tr><td>Terminal-Bench 2.1</td><td>Claude Opus-4.8 high</td><td><strong>80.22%</strong></td><td>Claude Code: 78.9%</td><td>Self-reported with a <a href="https://hub.harborframework.com/jobs/4b8e244f-8ab0-4d28-8218-7cf346282faa">public run</a>.</td></tr>
|
||||
<tr><td>Terminal-Bench 2.1</td><td>GPT-5.5</td><td><strong>84.3%</strong></td><td>Codex CLI: 83.1%</td><td>Self-reported with a <a href="https://hub.harborframework.com/jobs/f02fd019-23e1-495f-af0a-ebd9a65f3079">public run</a>.</td></tr>
|
||||
<tr><td>Terminal-Bench 2.1</td><td>Grok-4.5</td><td><strong>84.94%</strong></td><td>Cursor CLI: 79.3%. Hermes: 77.53%.</td><td>Self-reported after a reward-hack audit, with a <a href="https://github.com/harbor-framework/terminal-bench-2-1/pull/146">closed submission (not merged)</a>.</td></tr>
|
||||
<tr><td>Terminal-Bench 2.1</td><td>Grok-4.5</td><td><strong>84.94%</strong></td><td>Cursor CLI: 79.3%. Hermes: 77.53%.</td><td>Self-reported after a reward-hack audit, with a <a href="https://github.com/harbor-framework/terminal-bench-2-1/pull/146">submission</a>; community submissions closed.</td></tr>
|
||||
<tr><td>OSWorld-Verified</td><td>Claude Opus-5</td><td><strong>90.69%</strong></td><td>Previous best on the public board: 90.19%</td><td>Self-reported with <a href="https://huggingface.co/datasets/razzant/ouroboros-osworld-verified-opus5">public task evidence</a>.</td></tr>
|
||||
<tr><td>OSWorld-Verified</td><td>Claude Sonnet-4.6</td><td><strong>83.27%</strong></td><td>Pointer: 81.45%</td><td>Self-reported with <a href="https://huggingface.co/datasets/razzant/ouroboros-osworld-verified-sonnet46">public task evidence</a>.</td></tr>
|
||||
<tr><td>CL-Bench</td><td>Claude Sonnet-4.6</td><td><strong>0.2301, rank 1</strong></td><td>Previous top: 0.1960</td><td>Self-reported with an <a href="https://github.com/pgasawa/continual-learning-bench/pull/10">open submission</a> and <a href="https://huggingface.co/datasets/razzant/ouroboros-clbench-traces">full traces</a>.</td></tr>
|
||||
|
|
|
|||
|
|
@ -95,7 +95,7 @@
|
|||
<div class="chapter-heading reveal"><p class="chapter-index">02 / EVIDENCE</p><h2 id="bench-title">Measured against the strongest harnesses.</h2><p class="lede">Ouroboros has reproducible self-reported state-of-the-art results on Terminal-Bench 2.1, OSWorld-Verified, and CL-Bench. In those model-matched results, it leads Codex, Claude Code, Cursor, and Hermes. The public SWE-bench Pro matched pair is a statistical tie with Codex CLI. A separate GAIA result is marked as awaiting its scrubbed trace capsule.</p></div>
|
||||
<p class="evidence-status reveal"><strong>Publication status</strong><span>Self-reported results are published with evidence and each submission's status.</span><a href="/benchmarks/">Read the evidence table</a></p>
|
||||
<div class="bench-grid">
|
||||
<figure class="media-panel bench-wide reveal"><img src="/assets/bench-terminal-bench.svg?v=7a4871c0bb" alt="Self-reported Terminal-Bench 2.1 results: Ouroboros 86.74% with Claude Opus-5 after a disclosed reward-hack correction, ahead of Claude Code, Codex, Cursor, and Hermes on matched models" loading="lazy" width="1800" height="1478"><figcaption><span>Terminal-Bench 2.1 · 86.74% corrected · self-reported</span><span><a href="https://github.com/harbor-framework/terminal-bench-2-1/pull/175">Closed submission (not merged)</a> · <a href="https://hub.harborframework.com/jobs/2b145543-edeb-4a3b-b46f-4800310f1182">Full run</a></span></figcaption></figure>
|
||||
<figure class="media-panel bench-wide reveal"><img src="/assets/bench-terminal-bench.svg?v=7a4871c0bb" alt="Self-reported Terminal-Bench 2.1 results: Ouroboros 86.74% with Claude Opus-5 after a disclosed reward-hack correction, ahead of Claude Code, Codex, Cursor, and Hermes on matched models" loading="lazy" width="1800" height="1478"><figcaption><span>Terminal-Bench 2.1 · 86.74% corrected · self-reported</span><span><a href="https://github.com/harbor-framework/terminal-bench-2-1/pull/175">Submission</a> · <a href="https://hub.harborframework.com/jobs/2b145543-edeb-4a3b-b46f-4800310f1182">Full run</a></span></figcaption></figure>
|
||||
<figure class="media-panel reveal"><img src="/assets/bench-osworld.svg?v=26e6cdc7db" alt="Self-reported OSWorld-Verified results: Ouroboros 90.69% with Claude Opus-5 and 83.27% with Claude Sonnet-4.6" loading="lazy" width="1800" height="1000"><figcaption><span>OSWorld-Verified · 90.69% · self-reported</span><span><a href="https://huggingface.co/datasets/razzant/ouroboros-osworld-verified-opus5">Full traces</a></span></figcaption></figure>
|
||||
<figure class="media-panel reveal"><img src="/assets/bench-cl-bench.svg?v=633474d172" alt="Self-reported CL-Bench result: Ouroboros normalized reward 0.2301 with Claude Sonnet-4.6, rank 1" loading="lazy" width="1800" height="760"><figcaption><span>CL-Bench · 0.2301 · self-reported rank #1</span><span><a href="https://github.com/pgasawa/continual-learning-bench/pull/10">Open submission</a> · <a href="https://huggingface.co/datasets/razzant/ouroboros-clbench-traces">Full traces</a></span></figcaption></figure>
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -342,15 +342,16 @@ def test_readme_and_homepage_state_upstream_status_plainly():
|
|||
for text in (readme, homepage):
|
||||
for line in text.splitlines():
|
||||
if "terminal-bench-2-1/pull/" in line:
|
||||
assert "closed" in line.lower()
|
||||
if text == readme:
|
||||
assert "Self-reported; community submissions closed" in line
|
||||
assert "submission open" not in line.lower()
|
||||
assert "open submission" not in line.lower()
|
||||
for root in (SITE, DOCS):
|
||||
benchmarks = (root / "benchmarks/index.html").read_text(encoding="utf-8")
|
||||
assert "closed without merge on September 19, 2026" in benchmarks
|
||||
assert "closed community submissions on September 19, 2026" in benchmarks
|
||||
for line in benchmarks.splitlines():
|
||||
if "terminal-bench-2-1/pull/" in line:
|
||||
assert "closed submission (not merged)" in line
|
||||
assert "community submissions closed" in line
|
||||
|
||||
|
||||
def test_russian_talk_is_secondary_and_explicitly_labelled():
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue