Are brain waves the next unlock in physical AI?

The forefront of physical AI is a game of Jenga in a warehouse in San Leandro, California.

The warehouse is occupied by Encord, a company that builds data tools used to train AI models. Andrew Ceja is a pilot (the company’s term for robot trainer). He carefully pulls wooden blocks from a swinging tower while wearing a headset equipped with cameras that track what he sees. That alone would be pretty common for collecting robot training data, but this headset also includes sensors that measure brain waves as you carefully disassemble a block tower.

Encord is one of a small but growing number of startups that are convinced that the next real constraint on humanoid and warehouse robotics will not be model architecture, but the scarcity of real physical training data. Rather than helping robotics companies manage the data they have, Encord is building its business around manufacturing the data they don’t have.

The brainwave headset Ceja is wearing is made by Zander Labs, a German neuroscience startup that bets on measuring brain activity to infer mental states such as error, intent, and surprise, which can generate more useful datasets for training models. Encord’s Zander operation is currently in pilot operation. Encord says the goal is to build an initial brainwave-tagged data set, run it through customer robot models, and evaluate whether it actually improves performance before deciding whether to scale up.

Lucas Gehrke, a Zander neuroscientist overseeing the work, says the amount of brain activity used at any point during a given task provides clues to modelers trying to figure out when to deploy the most effortless models.

According to Vineeth Velmurugan, director of robotics learning at Encord, this is the “cutting edge” of efforts to solve robotics data bottlenecks. Velmurugan, a veteran of OpenAI’s robotics lab and warehouse automation company Berkshire Grey, joined Encord to build the company’s internal data generation team.

Encord was founded to help companies building machine vision applications annotate data and evaluate models. As customers (Velmurugan says he works with many major robotics companies but is not authorized to name them) began applying end-to-end learning to robot manipulation tasks, executives realized they needed to generate training data themselves, not just manage it. “There is no data at all,” Velmurugan said.

Confidence that generative AI can do for robots what it did for chatbots continues to hit the same wall. Self-driving car companies collect real-world data directly, but it is difficult to scale. Training through video is effective, but it lacks the fidelity of real data. Velmurugan says breaking through this would require a data set five times the size of the YouTube video corpus. This is a scale that helps explain why data generation itself has become a business rather than just a research problem.

Meet your egocentric data needs.

Companies building robot brains are now turning to two main sources. One is “egocentric” video collected by camera-wearing workers, often augmented with additional camera angles and other metrics, and the other is data from remotely operated robots. Encord does both, collecting egocentric data from multiple plants around the world and using its San Leandro facility to experiment with new modalities, such as brain waves, or to collect data sets for specific technologies for fine-tuning.

When TechCrunch visited, the pilots were generating data on tasks such as pouring coffee from a pot into a mug (very slowly) and stacking poker chips, using a pair of robotic arms, one directly controlled by a human operator and a leader-follower device that mimics its movements. “Every humanoid company has been asking us for these parts,” Velmurugan said.

Storage shelves contained vases, books, plastic vegetables, cat litter trays and scoops, boxes of fake flowers in bags and bundles of wire – operator training stock for household tasks.

At one of these stations, another pilot, Sofia Infante, steers a robotic arm to plug in and unplug Ethernet cables from the back of a server. These are the types of tasks data center operators would like to automate if only robots could operate them with the necessary precision. I went back behind the controls and figured out why I still couldn’t reach them. Pincers are much less dexterous than human fingers and lack the degree of freedom we take for granted in our arms.

Another new data method being developed by Encord uses a set of sensors strapped to the forearm to detect electrical signals from muscles. Videos of a human hand manipulating an object typically don’t capture the entire hand, but Velmurugan hopes to be able to build a more robust understanding of the model by creating a 3D depiction of the hand’s position at any time based on arm sensors.

Encord’s dataset is annotated with a physical description of what each video contains (“right hand tightening bolt”), helping the LLM-based model understand what’s happening. Velmurugan estimates that this kind of dense annotation is 100 times more valuable than “junky ego data” for training on a specific task, and costs only 20 times more to write on paper.

But “20x more” is still real money, and that’s the problem. The way LLM makers build their models by pulling from Stack Overflow and the rest of the web, by scraping text from the Internet, costs Frontier Labs absolutely nothing. Generating physical training data does not, and this is a limitation of the physical AI-LLM comparison. This kind of data must be manufactured rather than simply collected, which changes the economics of building these models.

Velmurugan says progress is being made through Encord’s visibility into programs across the industry. He sees both startups and pioneering labs figuring out what works and what doesn’t to improve physical AI models. At the same time, a vantage point among many robotics companies is also part of Encord’s strategy. Find out which data technologies are gaining traction across your industry before a single customer knows about them.

That will keep the dozen or so pilots at the Encord facility busy. Both Infante and Ceja are part of a burgeoning workforce developing the building blocks of neural networks. Before joining Encord, they worked at Scale, another AI data annotation company.

Ceja developed an interest in technology while working for a waste management company and was tasked with keeping robotic waste sorters in good condition. Now that the Jenga tower has collapsed, he says he enjoys the challenge of solving robot training challenges. “It’s something new every day!”