用行动赢得尊重

Alan Hou

Notes on coding, AI, and technology.

记录编程、人工智能与技术探索。

Benchmarks as Software: Terminal-Bench 2.1 and the Reality of Benchmark Maintenance Debt

基准即软件：Terminal-Bench 2.1 与大模型智能体评测的“技术债”危机

Even carefully validated coding benchmarks decay rapidly as frontier models discover valid solutions that rigid verifiers reject. Stanford researchers Christopher Ré, Scott Linderman, and team audit Terminal-Bench 2.0, identifying defects in roughly 30% of tasks. Releasing Terminal-Bench 2.1, the authors reveal that repaired tasks shift agent accuracy by over 10 percentage points and leaderboard ranks by up to three positions, while demonstrating that wall-clock timeouts artificially create massive 20-30pp harness gaps.

Nov 7, 2026 Read article

Alan Hou

Benchmarks as Software: Terminal-Bench 2.1 and the Reality of Benchmark Maintenance Debt

Recent Posts / 近期文章

Spectral and Subspace Dynamics of LoRA: Why Reinforcement Learning Demands Different Optimization Than SFT

NEO-VLA: Towards Native, Encoder-Free Vision-Language-Action Policies

Visual Token Routing: Learning to Keep Tokens That Matter for Computer-Use Agents

DreamX: Repurposing Video Diffusion Models as Universal Latent Action Learners

You Don't Need Long Context: Semantic State Space Models and the 7x Inference Penalty

Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap

WebHarbor: Docking Real Websites for Evolving GUI Agent Environments

ACCT: Asymmetric Cross-LLM Context Transfer

Hierarchical World Models with Implicit Dynamics (ID-WM)

Series / 系列文章

Nobel Prize in Economics (1969–2025)

Odoo 12 Development Cookbook

Odoo 14 Development Cookbook

TouchDesigner Tutorials

Mastering Python Scripting for System Administrators

Odoo 12 Development Essentials

Hung-yi Lee ML 2025

Dwarkesh Podcast Notes

Andrej Karpathy LLM Notes

freeCodeCamp