Google’s Genie world model can now simulate real-world streets with Street View.

We’ve all opened up Street View in Google Maps to show a friend what our childhood home looked like, or put a little human icon on a Paris street to make sure we’ve booked a hotel in a cool neighborhood. Imagine being able to do so in a more immersive and interactive way, where you could actually simulate the streets and their surroundings, and even do things like adjust the weather or see what it would look like in a “after tomorrow” scenario.

This is one of Google’s latest integration goals. Starting today, Google DeepMind connects Street View to Project Genie, the company’s universal world model that can create a variety of interactive experiences. This is a new feature released at the Google I/O developer conference.

“It’s a very powerful agent (and robotics) use case for humans to play with, and that’s always been the theme of Genie,” Jack Parker-Holder, a research scientist on DeepMind’s open team, told TechCrunch.

He gave the example of a new robot deployed in London, which barely sees the sun. Parker-Holder says Genie can simulate the rare instances of sunlight reflecting off a Victorian house, ensuring the rays don’t hit the robot.

“At the same time, you might say, ‘I’m going to go to New York, but I can’t go this time of year,’” he continued. “‘It’s going to snow. I want to see what those blocks look like in the snow.’”

Google has been collecting street view data for 20 years through camera-equipped cars and individuals wearing “tracking backpacks.” The tech giant has collected 280 billion images across 110 countries and seven continents.

“Street View gives you tons of images from all over the world,” said Jack. “You can imagine how potentially powerful it is to combine a rich source of real-world information and data with the ability to simulate the world.”

Google launched its latest world model, Genie 3, for a research preview last August, and in January opened access to the tool to Google AI Ultra subscribers in the U.S., allowing customers to create interactive game worlds from text prompts or images. The goal is to use Genie for educational experiences, games, and robotic training.

Genie 3 is already helping power one of Waymo’s simulators to train self-driving cars for “very rare events” like tornadoes or accidental elephant encounters. Adding Street View data here could help Waymo prepare for launch in more cities around the world.

Waymo has expanded to 11 U.S. cities and has its own simulators it relies on to test its AI drivers in several cities. Parker-Holder says the difference with the Genie is that it’s all done from an automotive perspective. Street View not only allows you to simulate a world anchored in real places, but also allows you to shift your perspective to other types of agents, such as humans or robots.

Google is rolling out Genie’s Street View to select Ultra users in the U.S. starting today, and access will become available at scale over time. Global Ultra users will gain access over the next few weeks on a company-by-company basis.

The researchers’ goal is to get this new feature into the hands of as many people as possible, according to Diego Rivas, product manager at DeepMind. He cautioned that Street View in particular and Genie in general are still experimental, so there’s a lot of room for improvement in terms of accuracy.

From the samples the Google team showed me, including an underwater simulation of my neighborhood, the results were impressive and striking. But it’s still not realistic video game quality. The model does not yet recognize physics and therefore does not yet understand cause and effect. For example, in a simulation of a woman running through snow-covered Joshua trees, she ran right through the cacti and bushes.

Compare, for example, Google’s image generator Nano Banana, which can now generate perfect text in infographics, or video generator Veo, which understands paper boats drifting in water currents, smoke billowing into the air, and fabric draping over shapes.

The physics is not hard coded into these models. Like living beings, they learn intuitively over time through passive observation.

“For these kinds of models, they can be six to 12 months behind video in terms of accuracy and quality, so I think that’s an issue we’re going to solve,” Parker-Holder said.

Google Maps director Jonathan Herbert, who joined the Street View team as an intern 12 years ago, said Genie is not yet able to faithfully reconstruct streets. He believes the real innovation is the spatial continuity of AI. When you rotate 360 ​​degrees, the AI ​​accurately remembers and simulates the environment behind you. From that point on, the model can build new environments on top of it.

“We’ve been thinking for a long time about how we can build the world’s best and richest models based on street view data,” Herbert said. “It’s been our idea for quite some time to use map data in new ways and for new kinds of AI research.”