Mass-producing hard formal proof tasks to train LLMs
TheoremSmith is an open-source pipeline for automatically generating synthetic RL environments for formal verification. It takes real open-source Lean projects, strips out dependency proofs to create seed tasks, synthesizes new longer-horizon theorems on top of those roots, and uses an adversarial QA pipeline (including a sandboxed proof-checking tool called Lean Comparator) to ensure tasks are difficult and resistant to reward hacking. It has generated hundreds of tasks with 30+ minute horizons for frontier models.
Represents a Lean project as a dependency DAG, identifies top-level theorems, and recursively blanks their dependencies so an agent must re-derive proofs from scratch. If a dependency appears in a theorem's statement, its statement is kept and only its proof is blanked, preventing agents from rewrit
Synthesizes new theorems that depend on existing root theorems, then compiles and verifies them, repeating iteratively to build multi-step tasks designed to train models on long, complex problems (inspired by Recursive Synthesis for Long-Horizon Terminal Tasks).
Checks an untrusted AI-generated proof against the original statement by compiling it in a sandbox, comparing statements, using the Lean kernel to verify correctness, and checking whether the solution uses any extra axioms — designed to prevent reward hacking.
Task creation runs through sourcing (license check, compilation, complexity), task creation (oracle run must pass, blank solution must fail), solve rollouts (failures must come from difficulty, not task bugs), and adversarial reward-hacking rollouts (every hack attempt must fail Comparator; successf
The full task-generation pipeline is open-sourced, with a detailed walkthrough referenced in an accompanying post on automating RL environment creation.
Generated hundreds of tasks with 30+ minute horizons for frontier models that consistently passed internal QA, and the authors state throughput can be scaled effectively.
An agent searches for open-source Lean projects as seeds, filtered by permissive license, compilability, and sufficient size/complexity.
Seed tasks are created by blanking dependency proofs of top-level theorems, and longer tasks are synthesized by generating new theorems that depend on existing roots.
A model attempts each task while an agent analyzes trajectories to confirm the task is difficult and that failures stem from difficulty rather than task defects.
A model is given knowledge of the environment and verifier and attempts to reward-hack; every attempt must fail Lean Comparator, and successful hacks trigger task fixes and re-runs.
It automatically generates synthetic formal verification RL environments — tasks paired with verifiers — by taking open-source Lean projects, blanking dependency proofs of top-level theorems to create seed tasks, and synthesizing new longer-horizon theorems on top of those roots.
Researchers and teams post-training LLMs with reinforcement learning on verifiable domains, particularly those wanting hard, realistic proof tasks in Lean.
Agents must fill answer files for each proof; Lean Comparator then compiles the untrusted solution in a sandbox, compares the statement against the original, verifies it with the Lean kernel, and checks for extra axioms. A dedicated adversarial phase has models attempt hacks, and any successful hack triggers a task fix and re-run.
No pricing is mentioned — the pipeline is fully open-sourced.
Yes. The task-creation pipeline has been fully open-sourced by the author.
The author reports that TheoremSmith generated hundreds of tasks with 30+ minute horizons for frontier models, which consistently passed internal QA. However, there is no direct evidence yet that training on these tasks improves real-world proof ability.
Unproven transfer to real-world proof tasks, seeds skewed toward math rather than code verification, possible pretraining contamination from public seed projects, and imperfect LLM-based quality checks that may let through broken or hackable tasks.
It currently works with Lean. Adapting it to other tools like Isabelle, Dafny, and Verus is listed as future work.