Agents20d ago
CheckerBench: ECNU and Peking University Introduce the First Executable Benchmark for Coding Agents Synthesizing Static-Analysis Checkers
Prevailing coding agent benchmarks evaluate isolated patch synthesis (e.g. SWE-bench) or vulnerability detection, neglecting whether agents can construct reusable, production-grade static-analysis checkers from scratch. Developing static analyzers requires interpreting defect specifications, traversing multi-file AST semantics, writing engine-specific logic, and refining implementations through iterative compilation and diagnostic feedback. Researchers from East China Normal University, SJTU, and Peking University unveil CheckerBench (arXiv:2610.07557), an executable suite comprising 300 tasks derived from 297 CVEs across 167 repositories, 85 CWEs, and five programming languages. Powered by the CheckerLab harness, evaluation across 21 model-harness configurations reveals a mean Pass@1 rate of only 32.30% (peak 45.33%), demonstrating that end-to-end program analysis tool synthesis represents a profound new frontier for software engineering agents.
Agents21d ago
SafeActBench: NUS Investigates the Broken Evidence-to-Action Chain in Tool-Using Agents Across Six Operational Domains
Tool-using agents execute consequential modifications to external systems, yet nominal task success frequently conceals unverified actions executed without prior evidence. Researchers from the National University of Singapore (NUS) present a rigorous empirical study on the breakdowns along the evidence-to-action chain (arXiv:2610.07753). Evaluating ten model-harness configurations reveals a striking paradox: high static action assessment accuracy routinely masks brittle interactive execution. Failures overwhelmingly originate prior to execution: agents terminate prematurely before collecting requisite evidence, or initiate state-altering actions before foundational proofs are established. The authors formulate SafeActBench, spanning 656 rigorous cases across six operational domains and five protocols progressing from static judgment to complex multi-action workflows. Supported by a provenance-bound Evidence Ledger and deterministic trajectory evaluator, findings demonstrate that while single-step execution is stable once evidence is verified, multi-action workflows frequently collapse under unresolved prerequisites and truncated state transitions.
Tooling2d ago
X discussion: engineer Arpit Bhayani says "no human will ever write code better than LLMs", drawing 7K+ likes, 240+ replies and a debate over AI coding ability
At 13:37 UTC on 2026-10-25, engineer Arpit Bhayani (@arpit_bhayani) posted: "No human will ever write code better than LLMs. Accept this fact, stop pretending, and move on." (post). When we pulled it, the post had about 7,193 likes, 496 reposts, 104 quotes, 241 replies, 529 bookmarks and 174K views, and it reached X's trending stories. Replies are split: some agree LLMs already write code faster and better, while others push back that LLMs often produce more code than a task needs and that production edge cases, architecture and final verification still depend on people. This is an X discussion presented as opinions, not a proven conclusion.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.