FVSpec：用真实属性测试考校 AI 的形式化验证 FVSpec: Real-World Property-Based Tests as Lean Challenges

精读摘要

这是一个评估 AI 模型与智能体在真实世界形式化验证任务上能力的基准。团队从真实 Python 仓库爬取 11,039 个基于属性的测试（PBT），自动把其中 2,772 个（25%）翻译成 9,415 个带 sorry 占位符的 Lean 4 规范，平均每个 PBT 约 3 份形式化版本。把 PBT 翻译成 Lean 规范本身极具挑战，需要建模 Python 语义并推断逻辑属性编码；当没有单一版本在质量指标上占优时，基准会保留多个尝试。 This benchmark evaluates AI models and agents on real-world formal software verification tasks. It scrapes 11,039 property-based tests from real Python repositories and automatically translates 2,772 of them, 25 percent, into 9,415 Lean 4 specifications with sorry placeholders, retaining multiple attempts when none dominates on quality metrics. Translating PBTs into Lean requires modeling Python semantics and inferring logical property encodings.