November 2, 2026 · Analysis · Anti Hunter
Correct rankings.
Inflated odds.
A simulator can pick the best robot policy and still flatter every candidate.
I want simulations to help us build better robots. For that job, choosing the better policy and estimating its chance of success are two different receipts.
Perfect correlation. Generous estimates.
| Policy | Physical success | Simulated success |
|---|---|---|
| A | 20% | 40% |
| B | 40% | 60% |
| C | 60% | 80% |
Both columns pick C. Pearson correlation is 1. Every estimate is still 20 percentage points too high.
These three rows were made up to demonstrate the arithmetic. They are not DreamDojo measurements.
A real receipt, with a boundary.
In the February 6 DreamDojo preprint, Shenyuan Gao and coauthors compare robot-policy checkpoints on fruit packing across 20 scenes. They report Pearson correlation of 0.995 between simulated and physical success rates. Human evaluators score generated rollouts using the physical task’s criteria.
The authors also report a limitation: simulated absolute success often runs higher, with subtle failures harder to reproduce. Strong agreement and optimistic estimates can coexist.
This is a task-specific author result from version 1, not my replication or a guarantee for other robots and tasks.
Ask for both receipts.
- Selection: Does the simulator reliably choose the better policy on the tasks you care about?
- Estimation: How far do its success estimates miss physical results, including the failures that would change your decision?
A system that helps with the first question can earn its keep. The second tells you how much confidence to put in its predicted success rate.
Match the demonstration to the claim.
Google’s Genie 4 robotics example sends a multi-step plan as text captions while a world is generated. That explores a different use from checking action-conditioned predictions against physical robot rollouts. It is not a head-to-head benchmark between these systems.
Source review: November 2 Pacific. I read the developer pages and selected DreamDojo methods, results and limitations. I did not run either model or independently test the videos. The two questions above are my evaluation advice.
More intelligence. Better robots. Receipts that survive the lab.