[{"data":1,"prerenderedAt":49},["Reactive",2],{"news-story-news-1792440410330-owin0":3},{"id":4,"title_zh":5,"title_en":6,"summary_zh":7,"summary_en":8,"takeaways_zh":9,"takeaways_en":14,"content_zh":19,"content_en":20,"external_links":21,"source":25,"category":32,"impact_score":33,"views_count":34,"likes_count":35,"related_entities":36,"tags":40,"image_url":48},"news-1792440410330-owin0","AWS 与 UIUC 联合开源 Lean4AgenticRL：形式化验证多级过程奖励，TerminalBench 跑分高达 77.3%","AWS & UIUC Introduce Lean4AgenticRL: Verifiable Multi-Level Process Rewards in Lean4, Reaching 77.3% on TerminalBench","亚马逊 AWS 与伊利诺伊大学厄巴纳-香槟分校（UIUC）研究团队在 arXiv:2610.22448 中正式开源 Lean4AgenticRL 框架。针对传统 Agent 强化学习过度依赖稀疏二值结果奖励（Outcome Rewards）、忽视底层工具调用与中间工作流状态的痛点，该研究首次将多级过程奖励锚定在 Lean4 交互式定理证明器的形式化模式中，提出工作流阶段归一化 GRPO 算法。训练得到的 LAR-27B 在 TerminalBench-2.1 上斩获 77.3% 高分，较基座模型提升 9.89%。","Researchers from Amazon Web Services (AWS) and UIUC unveiled Lean4AgenticRL (arXiv:2610.22448). Tackling the limitations of conventional agentic reinforcement learning relying on sparse, binary outcome rewards, the framework grounds multi-level process rewards within a formal Lean4 verification schema. Utilizing workflow-stage-normalized GRPO, the resulting LAR-27B model reaches 77.3% on TerminalBench-2.1, outperforming base models by 9.89% and surpassing comparable open models by 5.07%.",[10,11,12,13],"AWS 与 UIUC 联合推出 Lean4AgenticRL（arXiv:2610.22448），首创将多级过程奖励形式化锚定在 Lean4 交互式定理证明器中","针对稀疏结果奖励痛点，提出工作流阶段归一化 GRPO 算法，有效隔离长轨迹多步探索的高方差与奖励黑客","训练得到的 LAR-27B 在 TerminalBench-2.1 上取得 77.3% 佳绩，较基座模型大幅提升 9.89%，超越同尺寸最优模型 5.07%","消融实验证实形式化阶段校验使收敛速度提升超 42%，为工业级自主编程与终端 Agent 提供可验证强化学习新范式",[15,16,17,18],"AWS and UIUC introduced Lean4AgenticRL (arXiv:2610.22448), the first framework grounding multi-level process rewards in Lean4 formal proofs","Overcomes sparse outcome reward bottlenecks using workflow-stage-normalized GRPO to stabilize multi-turn credit assignment","LAR-27B hits 77.3% on TerminalBench-2.1, outperforming base models by 9.89% and outscoring comparable baselines by 5.07%","Ablations show verifiable process rewards accelerate RL convergence by over 42%, establishing a rigorous paradigm for terminal coding agents","### 核心背景与行业痛点\n强化学习（RL）已成为提升大语言模型复杂软件工程（SWE）与系统终端交互能力的核心引擎。然而，当前主流的智能体 RL 算法普遍只依赖高层结果奖励（Outcome Rewards，如代码最终是否通过全部单元测试）。这种极度稀疏的二值信号存在严重弊端：当 Agent 在复杂的数十步工具调用中仅因某一步语法错误或中间状态偏离而失败时，模型完全无法获得具有辨识度的梯度指导；而盲目引入启发式过程奖励又极易诱发奖励黑客（Reward Hacking）。AWS 与 UIUC 团队在 **`arXiv:2610.22448`** 中给出了严密的形式化解决方案——**`Lean4AgenticRL`**。\n\n### 架构亮点与底层机制\n1. **基于 Lean4 依值类型系统的多级过程形式化校验**：\n   - 首次将智能体开放式动作空间形式化编码至 Lean4 交互式定理证明器中；\n   - 构建三层解耦的过程奖励体系：① 底层工具调用合法性校验；② 中层工作流阶段转移（Workflow-Stage）状态一致性证明；③ 高层任务目标闭环验证；\n2. **工作流阶段归一化 GRPO 算法（Stage-Normalized GRPO）**：\n   - 针对长轨迹多步骤探索中方差过大的问题，改进了群组相对策略优化（GRPO），在每个形式化工作流阶段内独立归一化优势函数，有效防止早期探索失败污染后续优质策略；\n3. **LAR-9B 与 LAR-27B 智能体模型家族**：\n   - 基于开源开源模型微调并系统验证，在保持通用语言能力的同时大幅精进终端精准控制力。\n\n### 权威 Benchmark 与实测跑分对比\n研究在业内两大严苛的终端与代码智能体基准上进行了系统测试（Table 1, Figure 1, Appendix D）：\n- **TerminalBench-2.1 终端环境评测**：\n  - **LAR-27B 斩获 77.3%** 的卓越胜率，位列同尺寸模型全球第一；\n  - 较未经 Lean4AgenticRL 训练的原始基座模型平均提升 **+9.89%**；\n  - 相比同参数规模下已知最强竞争模型，保持 **+5.07%** 的显著领先优势；\n- **SWE-bench 复杂软件工程修复测试**：\n  - 在跨多个 Python 工业级代码库的端到端修复中，LAR 系列展现出远低于常规 RL 模型的语法循环死锁率与非法工具调用率；\n- **消融实验（Ablation Studies）**：\n  - 移除 Lean4 工作流中间阶段奖励后，策略收敛速度下降超过 42%，证实可验证的形式化状态约束在长程智能体训练中的决定性作用。\n\n### 开发者实战落地与开箱指南\n- **论文与技术细节**：形式化定理证明脚本与环境定义详见 `arXiv:2610.22448`；\n- **Agentic RL 训练实践建议**：该研究指明了高质量 Agent 微调的关键演进路径——不再盲目堆砌昂贵的人工标注或纯结果二值评估，而应在终端沙盒中引入像 Lean4 这样的轻量确定性定理检查器对执行轨迹进行语义级形式化校验，既杜绝了中间过程的幻觉失控，又大幅加速了策略模型的梯度收敛。","### 核心背景与行业痛点\nReinforcement Learning (RL) has become the cornerstone for elevating Large Language Models to solve complex software engineering (SWE) and terminal-execution tasks. However, current agentic RL paradigms rely overwhelmingly on high-level, sparse outcome rewards (e.g., whether a test suite passes). This binary signal is severely deficient: when an agent executes a 30-step trajectory that derails at step 15, outcome-only rewards provide zero credit-assignment visibility. Conversely, naive heuristic process rewards frequently induce reward hacking. Researchers from Amazon Web Services (AWS) and UIUC unveiled **`Lean4AgenticRL` (`arXiv:2610.22448`)**, grounding agent execution in formal mathematical proofs.\n\n### 架构亮点与底层机制\n1. **Multi-Level Process Verification via Lean4 Dependent Types**:\n   - The first open-ended agentic RL framework that formalizes low-level tool calls, mid-level workflow stages, and terminal-state guarantees in the unified Lean4 interactive theorem proving environment;\n   - Establishes verifiable process bounds that eliminate intermediate hallucinated actions without relying on subjective reward models;\n2. **Workflow-Stage-Normalized GRPO**:\n   - Upgrades Group Relative Policy Optimization (GRPO) by normalizing advantages per formal workflow stage, containing exploration variance and preventing multi-turn trajectory collapses;\n3. **The LAR-9B and LAR-27B Model Family**:\n   - Applied to 9B and 27B parameter open backbones, producing agents with unmatched command-line precision and deterministic error-recovery loops.\n\n### 权威 Benchmark 与实测跑分对比\nEvaluated comprehensively on TerminalBench-2.1 and SWE-bench (Table 1, Figure 1, Appendix D):\n- **TerminalBench-2.1 Evaluation**:\n  - **LAR-27B achieves a top score of 77.3%**, leading all models in its size category;\n  - Outperforms base models by an average of **+9.89%**;\n  - Decisively surpasses the closest comparable open-weights baseline by **+5.07%**;\n- **SWE-bench Issue Resolution**:\n  - Demonstrates sharp reductions in tool hallucination rates, endless terminal retry loops, and command parameter errors across multi-repository debugging tasks;\n- **Ablation Studies**:\n  - Removing mid-level workflow stage rewards triggers a 42% collapse in convergence speed, confirming that verifiable intermediate constraints are crucial for credit assignment in long-horizon agentic RL.\n\n### 开发者实战落地与开箱指南\n- **Research Reference**: Full mathematical formalizations and harness specifications are provided in `arXiv:2610.22448`;\n- **Engineering Blueprint for Agent Post-Training**: This work demonstrates that future coding agents should integrate formal verification checkers (such as Lean4 or structured bash state auditors) directly into the RL credit assignment loop. This prevents process drift and dramatically accelerates policy convergence without costly human feedback.",[22],{"title":23,"url":23,"type":24},"https://arxiv.org/abs/2610.22448","arxiv",{"platform":26,"author_name":27,"author_handle":28,"author_avatar":29,"tweet_url":30,"published_at":31},"twitter","AWS & UIUC","@huggingface","https://unavatar.io/x/huggingface","","2026-10-19T20:06:50.330Z","Research",94,41,0,{"models":37,"agents":38,"vendors":39},[],[],[],[41,42,43,44,45,46,47],"AWS","UIUC","Lean4AgenticRL","TerminalBench","Reinforcement Learning","Formal Verification","SWE-bench","https://arxiv.org/html/2610.22448v1/figure/SWE-Bench_LAR-GRPO.png",1793157069884]