What open robot datasets actually contain — and what nobody checks

Every entry in this ledger is a dataset we downloaded and ran tooling against, at full resolution, on real files. Numbers are measured, not quoted from dataset cards. Where the tooling is wrong, the entry says so.

Disclosure. The audit tool used here, robot-data-audit (RDA), is maintained by the author of this ledger. That is a conflict of interest, and it cuts both ways: the ledger is not an independent certification, and several of the findings below are cases where RDA produced incorrect verdicts. The one such defect published so far was fixed upstream and released as 0.12.2 on 2026-11-03 — the entry keeps both the wrong result and its correction. Every claim links to a command you can re-run.

01The ledger

One row per dataset. Structural covers the parquet and video layers — row counts, dtypes, timestamps, frame integrity. Annotation covers the language layer — subtask segmentation, timestamp monotonicity, temporal coverage. Verdict is our judgement after reconciling tool output against independent re-measurement, not the raw exit code.

Dataset	Scale	Format	Structural	Annotation	Verdict	Audited
FineART Scale AI × Hugging Face LeRobot	40,543 ep 1,718 h	v3.0	6/6 clean	uncovered	pass with caveats	2026-11-02
droid_1.0.1	—	—	queued	queued	queued	—
libero	—	—	queued	queued	queued	—
pusht	—	—	queued	queued	queued	—
aloha_sim_transfer_cube_human	—	—	queued	queued	queued	—

Queued entries are listed because they share the same v3.0 layout and the same open tooling path. They have not been audited; they carry no verdict. Rows are added only after a full run on downloaded data.

02Why a ledger, and not a leaderboard

A dataset that is structurally valid can still be semantically broken. The failure modes that actually cost training runs — a camera that stopped recording while the state stream kept going, a subtask timeline that drifts by seconds, a frame count that disagrees with the video container — are invisible to a load that succeeds. They surface as a policy that trains fine and fails in evaluation, or as a two-day debugging session in the middle of a run.

The problem is not that these checks are hard. Most of them are trivial to compute once the data is on disk. The problem is that nobody runs them: the dataset is 1.86 TB, the download takes hours, and by the time it lands the team wants to train, not to audit.

So this ledger has one narrow purpose: run the checks people skip, publish the numbers, and be explicit about what the checks miss. It is a public record, not a score. A dataset scoring badly here is not a bad dataset — it means a specific property was measured and found wanting, with the exact measurement attached.

Covered

Structural integrity

Schema conformance, NaN/Inf, timestamp validity, video frame counts against container metadata, stream presence, A/V offset and drift.

Covered, with effort

Annotation consistency

Subtask counts against episode metadata, timestamp monotonicity, empty text, temporal coverage at episode boundaries. Requires custom code — no mainstream tool ships this.

Not covered

Semantic correctness

Whether the label describes what the robot is doing. This is not an automated check and no entry in this ledger should be read as claiming it.

03Method

Four rules govern every entry. They exist because dataset audits fail in a predictable way: the tool reports something, nobody re-measures it, and a false positive becomes a published conclusion.

Download, do not infer. Every entry is produced by running on real files. Sample size and selection are stated explicitly; conclusions about format and mechanism are separated from conclusions about the full distribution.
Re-measure anything the tool flags. A verdict is never a raw exit code. When RDA excludes an episode, the underlying signal is decoded again from scratch — at full resolution, independent of the tool — before it is reported.
Separate fact from inference. Measured values are stated as measurements. Generalisations (“this will recur on other datasets of the same family”) are labelled as inference and left unverified until a second dataset is audited.
Publish the commands. If a result cannot be reproduced from the commands in the entry, it does not belong in the ledger.

Verdict vocabulary

Verdict	Meaning
pass	All covered checks ran and returned clean, on a sample large enough to state it.
pass with caveats	Covered checks are clean, but either the sample is small, a check returned no verdict, or a layer of the dataset is not reachable by current tooling.
fail	A defect was confirmed by independent re-measurement, not merely flagged.
uncovered	The dataset has this layer; the tooling has no metric for it. Recorded as a tooling gap, not as a dataset defect.

What this ledger does not do

It does not certify a dataset. Absence of findings is bounded by the sample and the metric set.
It does not score datasets against each other. Coverage differs; a single number would hide that.
It does not audit semantics. A clean run says nothing about whether the labels are true.

04Corrections and requests

If an entry misrepresents your dataset, say so and it gets corrected in place, with the correction visible — the FineART entry already contains one such correction, kept rather than deleted.

If you maintain a dataset and want it audited, or you have in-house robot data that needs the same treatment with a private report, get in touch at luoxi@touchlabelai.cn or open an issue on GitHub. The audit path is the same; only the distribution differs.

Tool: robot-data-audit · entries are generated from JSON audit output plus independent verification scripts; both are kept alongside each entry.