Connect with us

Technology & Innovation

Google DeepMind’s new generative model makes Super Mario-like games from scratch

Diane Davis

Published

on

example of game generated from a crayon sketch

“It’s cool work,” says Matthew Gudzial, an AI researcher at the University of Alberta, who developed a similar game generator a few years ago. 

Genie was trained on 30,000 hours of video of hundreds of 2D platform games taken from the internet. Others have taken that approach before, says Gudzial. His own game generator learned from videos to create abstract platformers. Nivida used video data to train a model called GameGAN, which could produce clones of games like Pac-Man.

But all of these examples trained the model with input actions, button presses on a games controller, as well as video footage: a video frame showing Mario jumping was paired with the “jump” action, and so on. Tagging video footage with input actions takes a lot of work, however. This has limited the amount of training data available. 

In contrast, Genie was trained on video footage alone. It then learned which of eight possible actions would cause the game character in a video to change its position. This turned countless hours of existing online video into potential training data. 

Genie can generate simple games from hand-drawn sketches

GOOGLE DEEPMIND

Genie generates each new frame of the game on the fly depending on the action the player takes. Press jump and Genie updates the current image to show the game character jumping; press left and the image changes to show the character moved to the left. The game ticks along action by action, each new frame generated from scratch as the player plays. 

Future versions of Genie could run faster. “There is no fundamental limitation that prevents us from reaching 30 frames per second,” says Tim Rocktäschel, a research scientist at Google DeepMind who leads the team behind the work. “Genie uses many of the same technologies as contemporary large language models, where there has been significant progress in improving inference speed.” 

Genie learned some common visual quirks found in platformers. Many games of this type use parallax, where the foreground moves sideways faster than the background. Genie often adds this effect to the games it generates.  

While Genie is an in-house research project and won’t be released, Gudzial notes that the Google DeepMind team says it could one day be turned into a game-making tool—something he’s working on too. “I’m definitely interested to see what they build,” he says.

Technology & Innovation

This self-driving startup is using generative AI to predict traffic

Diane Davis

Published

on

A diptych view of the same image via camera and LiDAR.

While autonomous driving has long relied on machine learning to plan routes and detect objects, some companies and researchers are now betting that generative AI — models that take in data of their surroundings and generate predictions — will help bring autonomy to the next stage. Wayve, a Waabi competitor, released a comparable model last year that is trained on the video that its vehicles collect. 

Waabi’s model works in a similar way to image or video generators like OpenAI’s DALL-E and Sora. It takes point clouds of lidar data, which visualize a 3D map of the car’s surroundings, and breaks them into chunks, similar to how image generators break photos into pixels. Based on its training data, Copilot4D then predicts how all points of lidar data will move. Doing this continuously allows it to generate predictions 5-10 seconds into the future.

Waabi is one of a handful of autonomous driving companies, including competitors Wayve and Ghost, that describe their approach as “AI-first.” To Urtasun, that means designing a system that learns from data, rather than one that must be taught reactions to specific situations. The cohort is betting their methods might require fewer hours of road-testing self-driving cars, a charged topic following an October 2023 accident where a Cruise robotaxi dragged a pedestrian in San Francisco. 

Waabi is different from its competitors in building a generative model for lidar, rather than cameras. 

“If you want to be a Level 4 player, lidar is a must,” says Urtasun, referring to the automation level where the car does not require the attention of a human to drive safely. Cameras do a good job of showing what the car is seeing, but they’re not as adept at measuring distances or understanding the geometry of the car’s surroundings, she says.

Though Waabi’s model can generate videos showing what a car will see through its lidar sensors, those videos will not be used as training in the company’s driving simulator that it uses to build and test its driving model. That’s to ensure any hallucinations arising from Copilot4D do not get taught in the simulator.

The underlying technology is not new, says Bernard Adam Lange, a PhD student at Stanford who has built and researched similar models, but it’s the first time he’s seen a generative lidar model leave the confines of a research lab and be scaled up for commercial use. A model like this would generally help make the “brain” of any autonomous vehicle able to reason more quickly and accurately, he says.

“It is the scale that is transformative,” he says. “The hope is that these models can be utilized in downstream tasks” like detecting objects and predicting where people or things might move next.

Continue Reading

Technology & Innovation

Methane leaks in the US are worse than we thought

Diane Davis

Published

on

Methane leaks in the US are worse than we thought

Methane emissions are responsible for nearly a third of the total warming the planet has experienced so far. While there are natural sources of the greenhouse gas, including wetlands, human activities like agriculture and fossil-fuel production have dumped millions of metric tons of additional methane into the atmosphere. The concentration of methane has more than doubled over the past 200 years. But there are still large uncertainties about where, exactly, emissions are coming from.

Answering these questions is a challenging but crucial first step to cutting emissions and addressing climate change. To do so, researchers are using tools ranging from satellites like the recently launched MethaneSAT to ground and aerial surveys. 

The US Environmental Protection Agency estimates that roughly 1% of oil and gas produced winds up leaking into the atmosphere as methane pollution. But survey after survey has suggested that the official numbers underestimate the true extent of the methane problem.  

For the sites examined in the new study, “methane emissions appear to be higher than government estimates, on average,” says Evan Sherwin, a research scientist at Lawrence Berkeley National Laboratory, who conducted the analysis as a postdoctoral fellow at Stanford University.  

The data Sherwin used comes from one of the largest surveys of US fossil-fuel production sites to date. Starting in 2018, Kairos Aerospace and the Carbon Mapper Project mapped six major oil- and gas-producing regions, which together account for about 50% of onshore oil production and about 30% of gas production. Planes flying overhead gathered nearly 1 million measurements of well sites using spectrometers, which can detect methane using specific wavelengths of light. 

Sherwin et al., Nature

Here’s where things get complicated. Methane sources in oil and gas production come in all shapes and sizes. Some small wells slowly leak the gas at a rate of roughly one kilogram of methane an hour. Other sources are significantly bigger, emitting hundreds or even thousands of kilograms per hour, but these leaks may last for only a short period.

The planes used in these surveys detect mostly the largest leaks, above roughly 100 kilograms per hour (though they catch smaller ones sometimes, down to around one-tenth that size, Sherwin says). Combining measurements of these large leak sites with modeling to estimate smaller sources, researchers estimated that the larger leaks account for an outsize proportion of emissions. In many cases, around 1% of well sites can make up over half the total methane emissions, Sherwin says.

But some scientists say that this and other studies are still limited by the measurement tools available. “This is an indication of the current technology limits,” says Ritesh Gautam, a lead senior scientist at the Environmental Defense Fund.

Continue Reading

Technology & Innovation

The Download: What social media can teach us about AI

Diane Davis

Published

on

The Download: What social media can teach us about AI

June 2023

Astronomy should, in principle, be a welcoming field for blind researchers. But across the board, science is full of charts, graphs, databases, and images that are designed to be seen.

So researcher Sarah Kane, who is legally blind, was thrilled three years ago when she encountered a technology known as sonification, designed to transform information into sound. Since then she’s been working with a project called Astronify, which presents astronomical information in audio form. 

For millions of blind and visually impaired people, sonification could be transformative—opening access to education, to once unimaginable careers, and even to the secrets of the universe. Read the full story.

—Corey S. Powell

We can still have nice things

A place for comfort, fun and distraction to brighten up your day. (Got any ideas? Drop me a line or tweet ’em at me.)

+ It’s time to get into metal detecting (no really, it is!)
+ Meanwhile, over on Mars
+ A couple in the UK decided to get married on a moving train, because why not?
+ Even giant manta rays need a little TLC every now and again.


Continue Reading

Trending