Robots Don’t Need More Video. They Need the Right Failures.

1. Everyone is selling data. Who actually gets paid?

Everyone and their mother is getting into data right now, from buying company internal docs to paying consumers to video everyday tasks.

What’s very clear is that data is what’s valuable. What’s less clear is who benefits and where the money actually goes.

2. Why the LLM data playbook worked, and where it’s heading

For regular text data, AI has now seen most of the useful internet.

In the words of Mercor CEO Brendan Foody, “the internet was a one-time sponsorship.” The rest of knowledge is tied up in businesses and workflows.

While Mercor and others have built giants buying and labeling data and setting up environments for domain experts to come in, the rush of everyone selling commodity data will slow down too.

The low-value work will be automated and consolidate into a handful of power players. Just flipping data without advanced infrastructure isn’t a long-term way to stay around.

Building environments for reinforcement learning is hot right now too. But for them to be sustainable in the long term, you have to be able to make them from either (1) domain expertise or (2) specialized data.

Otherwise it comes down to a compute game, and the labs have everyone beat.

More compute lets us generate more attempts and run more checks. The question is whether those checks measure what we actually care about. If our scoring system rewards the wrong thing, scaling the training process can reinforce the mistake..

Reinforcement learning generally needs three pieces—a task, attempts, and a score.

Easy to score: A math problem with one right answer, or code that either passes its unit tests or doesn’t. Training on checks like these is called RLVR, reinforcement learning with verifiable rewards.

Hard to score: A legal redline, medical advice, a poem. “Good” is a judgment call.

Just having compute generate these usually results in either wrong or reward-hacking behavior. It takes expertise to craft the environments and make them actually useful for improving the models.

So compute can generate a thousand redlines of a contract. It can’t tell you which ones a real lawyer would sign off on.

That’s the judgment that lives in a rubric written by an expert, and the lab can’t make one with just more GPUs.

3.why robotics data is different

The internet, by its nature, can be trained on mostly natively right now for LLMs, and those models are increasingly used in data generation and annotation.

That’s not nearly as true for robotics data, even though the amount of video is staggering.

Why?

With language, there’s no need to describe language with language. Even a task that seems simple to us is a whole different ball game when it comes to understanding video and actions in the real world.

Video doesn’t physical action doesn’t encode all the obvious effects of what’s happening or what would have happened, the force, grip contact and recovery data—this requires complex mapping and tracking and/or telemetry.

A video of a person moving a cup gives us some of this information. We can see the cup, the person reaching for it, and where it ends up.

But the video might tell us very little about how tightly they held it or how they adjusted when it started slipping. A robot with a different hand still has to learn how to carry out the movement.

Another big challenge is the literal physical action. Besides the spatial and logical or 3D reasoning, even assuming all of that is correct, the model now has to translate it to a robot interface, whether hands or grippers.

That involves speed, force, grip strength, and what it takes to navigate with it.

The models in play

Say we want a robot to pick up a cup and put it on a shelf. One approach might use a vision-language-action model, or VLA.

A VLM, or vision-language model, connects images or video with language. It could identify the cup in front of the robot and interpret what we mean by “put it on the upper shelf.”

A VLA connects those observations and instructions to physical actions. It’s similar to a multimodal LLM, except that it outputs action trajectories and coordinates for the physical robot, not just text instructions.

It often also receives measurements of the robot’s current state, such as joint positions and gripper opening. It can serve as the robot’s policy, meaning the system that chooses what action to take next.

In systems such as NVIDIA’s GR00T, an action head generates a short sequence of movement targets from the model’s internal numerical features and robot measurements.

Some action heads use diffusion or flow matching: they start with random noise and progressively turn it into an action sequence suited to the situation. A controller translates the targets into motor commands.

A world model predicts how a situation may change. When trained to account for proposed actions, it can help estimate what would happen if the robot moved the cup a certain way.

That can support planning: compare possible movements by their predicted consequences before choosing one.

People are very excited about world models as a different paradigm from LLMs, which fundamentally predict text.

(Multimodal LLMs translate video and images into the same autoregressive process and embeddings, which is essentially what makes them VLMs.)

4. How the data actually gets made

While there’s tons of content on the internet, from YouTube to movies to video games, the proper annotation pipeline is a lot harder with ready-made videos.

Companies like Surge, Scale, DoorDash and Kled let regular people record and upload specific tasks, or use the footage and then split and annotate it.

But inevitably the low-hanging fruit, simple clips of moving this or that, will end up with world and vision-language models in a pipeline doing the annotation.

Here’s a small breakdown →

Now, as models improve, we should be able to get more out of footage that already exists.

Models can help identify activities, find relevant clips, and estimate movements. Some predictive models learn from video without written labels.

When additionally trained to account for actions, a world model can help predict what will happen if the robot moves a certain way.

There is evidence that this transfer can improve.

Physical Intelligence’s human-to-robot study, broader training across robot tasks, environments, and bodies helped models learn more from human demonstrations.

The human recordings had hand-motion estimates and language annotations, and were mixed with robot data.

So to be clear, there was still considerable work involved in making that experience usable but the trajectory is there.

Research: Human-to-robot transfer

The paper is Emergence of Human to Robot Transfer in Vision-Language-Action Models, published Dec 16, 2025, with a plain-language post.

What it found:

Broader robot pretraining helped the model benefit more from human demonstrations. They combined prepared human recordings, including estimated hand movements and language annotations, with relevant robot data during fine-tuning.

The resulting policies achieved roughly twice the performance across four tested generalization scenarios. Their analysis also suggested that broader pretraining helped the model develop more similar representations of human and robot examples.

5. Data is an answer to a specific failure

The value will lie, similar to Mercor’s and the best of the RL labs, in domain expertise.

That doesn’t necessarily mean literally being a welding expert or a cup-moving expert.

It means knowing the general task and the job at hand, knowing which failure modes and edge conditions matter for the task. Which will still need a human in the loop for a while.

The human alpha is in deciding how to use a piece of footage, separate sections, and label them in a specific way to teach models things they currently aren’t performant at.

So it’s not just generic cup placement but data of cups being placed at a specific angle, across different ranges of cause and effect and conditions where models will struggle to annotate out of the box.

Suppose our robot can already pick up most cups, but keeps dropping wet ones.

Another thousand ordinary cup videos may add very little.

We might need demonstrations with different materials, moisture levels, grips, and lifting speeds. We might also need examples showing how to recover after the cup starts slipping.

The point of that dataset is to answer a specific question about the failure.

Of course, someone first has to check that more training is the right fix. If the gripper is physically unsuitable, recording more attempts won’t change its hardware.

Even a success label takes some thought.

Did the cup reach the shelf?

Did it stay there after release?

Did the robot spill what was inside?

The training feedback needs to reflect the job we actually want done.

6. The RL layer

That brings us to reinforcement learning, or RL.

A robot attempts a task and receives a numerical reward based on performance. Training uses that feedback to update its policy, meaning the model that selects its actions, toward behavior that earns more reward.

One promising direction is reinforcement learning with verifiable rewards, or RLVR. The appeal is that the system can check whether an attempt met specific requirements and use that result as training feedback.

Take for example a robot inserting an electrical connector. A physical test rig could measure insertion depth and check whether the connection conducts electricity. For this task, the rig plays a role similar to like a unit test in software.

But passing those checks only establishes what we actually tested. The connector could conduct electricity while still being loose. We have to make sure the checks reflect the full job we want done.

Other training systems use feedback that involves more judgment. HIL-SERL uses a model trained to recognize successful attempts, alongside human corrections. Physical Intelligence’s RECAP combines demonstrations, robot attempts, outcome labels, and corrections.

These are different ways of supplying feedback for reinforcement learning. A learned model can make mistakes, and human labels require time and judgment. The business opportunity includes figuring out which feedback is reliable enough for the task and making the process affordable to repeat.

Someone has to reset the scene for another attempt, handle failures, and test whether the updated model works on examples it wasn’t trained on.

Making that process dependable is a substantial opportunity in its own right.

On paper, an RL environment is pretty straightforward.

A robot attempts a task, receives a reward, and tries again.

In the real world though, someone has to build and maintain the entire process that makes those attempts possible.

Take our electrical connector example →

We need a robot, cameras and sensors, different connectors and insertion conditions, and a reliable way to verify whether each attempt succeeded.

After every attempt, someone has to reset the scene, remove the connector, reposition the objects, and prepare the next trial.

If something breaks or gets stuck, someone has to intervene.

Today, much of this work is still handled by robotics researchers, engineers, technicians, and trained operators.

Systems like HIL-SERL use human interventions during training, while newer research is working toward automating the resets and evaluations.

Then there’s the evaluation itself.

We have to test whether the robot actually improved on conditions it hasn’t trained on, such as different connector angles, tolerances, and positions, because otherwise, we might just be teaching it to succeed in one particular setup as opposed to learning the actual task.

The cost of real-world learning

And this does not come cheap.

A research-grade robotic data collection setup can cost tens of thousands of dollars in hardware alone.

For reference, the DROID project’s published equipment list estimates around $46,000 for one robot collection setup.

Operating costs then include personnel, maintenance, engineering, and repeated physical trials.

Building an environment where robots can repeatedly attempt tasks, encounter meaningful failures, receive trustworthy feedback, and be evaluated against new conditions is a business in itself.

And unlike a dataset that’s sold once, an RL environment could potentially keep generating value as new models come along and need to be trained and evaluated.

7. Who actually gets paid

Still, difficult work does not automatically make a great business.

If competitors can produce the same results at similar cost, specialized collection can face price pressure too.

Access to a particular factory, useful measurements others can’t get, or the ability to solve customer problems consistently could matter much more than how many hours a supplier records.

Robotics operators themselves have a big advantage, having their machines generate experience while working.

An outside supplier has to offer something worth buying beyond what the customer can collect internally.

A robot can transfer skills between settings, but the customer still needs evidence that it works under their conditions. A successful demonstration elsewhere doesn’t establish that it can handle their objects, tolerances, and failure cases reliably..

8. Four questions to ask any robotics data company

So when we look at a robotics data company:

1. What does this data help the robot learn?

1. Can the supplier demonstrate the improvement?

1. What happens to that advantage when the next model needs fewer examples and inevitably improves?

1. Does it teach anything beyond the actual footage (the pure task), such as real spatial understanding and predictive value, as opposed to just a simple task?

9. What’s next

Data efficiency

NVIDIA says synthetic data for GR00T N1.5 took 36 hours versus nearly three months by hand.

If that gap widens, hours of human footage lose value.

YouTube to first-person

A Labellerr co-founder told CNN that if AI can turn online video into first-person view, it could substitute for collected footage.

Wages

Human-data work , so price pressure shows up first there.

Where the value goes

World models and synthetic data keep improving, demand for human-collected hours can fall faster than expected.

Physical Intelligence found human video helps most when robot pretraining is already diverse, the results suggest that diverse robot experience and human demonstrations can reinforce each other. As models improve, they may get more value from recordings that were previously difficult to use. Whether that raises demand for footage or reduces the amount customers need to buy is still an open question..

So whether it’s world models or labeling and annotating pipelines getting better and better, while data will be worth something, where the value goes is a different story.

To be clear —>Not an expert, I am constantly learning and would welcome any push-back/criticism and challenge’s i would love to learn more .