SimplerEnv Benchmark Explained: Sim Scores vs Real Robot Ranks

SimplerEnv is built to answer one question: do simulated robot evaluations rank policies the way real robots do? Its authors report "strong correlation" between simulated and real results in paired evaluations (arXiv:2405.05941, 9 May 2024). A third-party write-up dated 2 October 2026 reports a Pearson correlation (r) of 0.924 and a Mean Maximum Rank Violation (MMRV) of 0.056 for SimplerEnv's visual-matching setting, measured against real Google Robot results on six policies (Bifrost AI).

What does SimplerEnv measure?

SimplerEnv is a set of simulated environments for evaluating real-world robot manipulation policies. It covers two robot platforms, Google Robot and WidowX. The project page describes the design as paired sim and real evaluations of manipulation policies, which is how the authors test whether the simulated results hold (SimplerEnv project page). The code is public on GitHub (allenai/SimplerEnv).

The task set is narrow. Every task is rigid-body manipulation, and none involve cloth, rope or soft objects (Bifrost AI). That scope is the first thing to check against your own robot.

How is sim-to-real agreement scored?

The benchmark uses standard correlation measures and adds one metric. The paper introduces MMRV "to better assess the real-and-sim policy ranking consistency," and lower values are better (SimplerEnv project page). The GitHub repository lists MMRV alongside the standard correlation measures (allenai/SimplerEnv).

Pearson r measures how closely simulated and real scores move together. MMRV penalises ordering errors: how far the simulated ranking strays from the real one. If you are choosing the best checkpoint, the ordering matters more than the absolute score, so MMRV answers the question you actually face.

What do the published numbers show?

The two rows are not the same test, and the paper's own numbers do not appear in its abstract or project page. Check the full PDF before citing a figure from it. The Bifrost numbers cover six Google Robot checkpoints: three RT-1 checkpoints, RT-1-X, RT-2-X and Octo-Base (Bifrost AI). Those are 2024-era policies.

A Pearson r of 0.924 across six policies means the simulated and real scores moved together closely on those six. Six points is a small sample, and the write-up gives no confidence interval. Treat it as one result on one setting, not a property of the benchmark as a whole.

Who runs SimplerEnv, and who reports its numbers?

The authors publish the benchmark, the paper and the code. They also validate it, so the headline correlation claim is self-reported. The correlation numbers in this post come from the paper and from one vendor write-up.

Bifrost AI also publishes a page on SimplerEnv alternatives (Bifrost AI). Its SimplerEnv coverage is therefore a comparison with a point of view, not an audit. Read its figures as a secondary report.

Where does SimplerEnv fall short?

• Two robots only. There are no environments for Franka arms, bimanual rigs or humanoids (Bifrost AI).
• No wrist cameras. A PolaRiS paper points out that SIMPLER and earlier real-to-sim methods do not support wrist cameras, which most current VLAs use (Bifrost AI).
• Fine force control is a gap. A sim-to-real gap remains for tasks that need fine force control or deformable objects (Open Source Robotics).
• Hand-built and small. Each environment was matched by hand, so a new scene or task takes real effort, and the task set has stayed small (Bifrost AI).
• Significance is thin. A June 2026 arXiv analysis finds that only 19.7% of SOTA claims on SimplerEnv are provably statistically significant, against 19.8% on LIBERO (arXiv).
• Training data can flatter a score. The same analysis notes that the WidowX test runs in simulation while the training data is real-world BridgeData V2 demonstrations, far from that test. It also argues SimplerEnv does not restrict where training data comes from, so a high score can reflect training data close to the test set (arXiv).

Which VLA models have SimplerEnv scores?

The published correlation figures cover only the RT-1, RT-1-X, RT-2-X and Octo-Base checkpoints. This post therefore does not rank newer VLA models, and it does not infer a score for any model left out. If you read a newer model's SimplerEnv result, check which checkpoint, which environment and which team produced it.

Should you trust a SimplerEnv score?

Trust it for ranking checkpoints you can run on Google Robot or WidowX, within the visual-matching setting the published numbers describe. Do not use it to predict absolute real-world success, and do not extend it to other embodiments, because the benchmark has no environments for them. The evidence supports rank consistency on those two platforms, not general sim-to-real transfer.

A clean rank correlation can still mislead if the ranks come from one harness.

What's still unclear?

• The paper's own values. The paper describes the correlation as strong, but its abstract and project page give no numbers. Check the arXiv PDF before citing a figure from it.
• Sample size. The 0.924 and 0.056 figures rest on six policies, and the write-up gives no confidence interval.
• A source tension. The paper reports strong correlation, while the June 2026 analysis argues a high score cannot distinguish a policy that generalizes from one trained on nearby data. Both can hold for the 2024-era checkpoints. No source reconciles them.
• Newer models. No published SimplerEnv results cover models released after the 2024 checkpoints.
• The weakest point of the argument. The strongest numbers come from 2024-era policies on two platforms. Nothing here shows the correlation holds for a policy that uses a wrist camera or a non-rigid task.