“Dreamer marks a significant step towards general AI systems,” Google DeepMind computer scientist Danijar Hafner explained, in a Nature study. The launch features a record-breaking achievement in artificial intelligence: an automatic machine learning system that is able to learn the nuances of Minecraft, one of the world’s most popular games with 100 million active monthly players. It’s not a game, though; it’s AI itself in the future.
Minecraft is a procedurally generated 3D world with an extremely wide range of landscapes, from deserts to forests, and challenges the players’ skills by crafting objects, constructing houses, and survival in attempting to survive. Its biggest held accomplishment is diamond gathering, where the players are challenged by attempting to hit a line of challenging goals. These are things like felling trees to get wood, crafting tables, hammering out pickaxes, and digging way out of the ground in without a predefined path. “Every time you play Minecraft, it’s a new, randomly generated world,” Hafner explained. That unpredictability is what makes it such an excellent test case for AI systems that need to generalize learning from surreal worlds.
Step forward, Dreamer, a learning computer that learns its world through trial and error through reinforcement learning. Contrary to past systems, where past systems learned from human input or video feed, Dreamer is able to do all this independently, learning what actions give rewards and how to optimize its processes over time. Reinforcement learning has been the big entrance to AI victories for decades, helping frameworks achieve breakthroughs in domains such as beating human masters at Go and Starcraft. Those frameworks weren’t very good at applying what they had learned in other domains. Dreamer shatters that trend.
Central to Dreamer’s innovation is its world model a neural network forecast of an action’s effect and picture of the future state. Hafner described this ability to abstract human thought by saying, “The world model really equips the AI system with the ability to imagine the future.” This forecasting ability enables Dreamer to make optimal choices, with fewer computing resources than in real-time action taking.
Dreamer’s performance is not Minecraft-specific. Researchers evaluated its performance on seven tasks, ranging from virtual robot control to playing video games, with more than 150 tasks. Dreamer achieved new state-of-the-art performance on four tasks, and we can see how versatile it is. DeepMind says that “World models carry the potential for substantial transfer between tasks. Therefore, we see training larger models to solve multiple tasks across overlapping domains as a promising direction for future investigations.” This suggests the wider implications of Dreamer’s ability, specifically robotics, trial-and-error learning that is expensive and usually not practical.
Hafner’s placement of Dreamer on Minecraft’s diamond challenge wasn’t on the agenda to begin with. “We built this whole algorithm without that in mind,” Hafner conceded. But what was supposed to occur became the ultimate test of Dreamer’s capability of out-of-box thinking to fix new problems. Earlier AI solutions tried learning to tackle this issue from human examples or step-by-step explicit instructions and thus the quality of Dreamer’s self-learning mechanism stands enhanced.
The algorithm is also impressive. Dreamer learns three networks: a world model to predict the consequence, a critic to predict the state value, and an actor to choose action. The networks train from experience replay on a single Nvidia V100 GPU, demonstrating the scalability of the system. DeepMind added a symlog transform, a symmetric logarithm employed to normalize to optimize, to reduce the heterogeneity of inputs and rewards across the environments. It allows Dreamer to generalize very well in most environments with no domain-specific tuning. The performance of Dreamer on Minecraft is taken as a metric of its effectiveness.
The algorithm learned from the game’s limited world reward providing only rewards back milestones up until diamond collection. It learned to use tools and harvest resources, overcoming the block-breaking wall a feat initially unfathomable because of game physics. DeepMind distorted reality in a bid to speed up breaking blocks such that the AI would not have to bother about button-mashing frivolity. Dreamer’s success has far-reaching implications. By showing that knowledge can be transferred and applied to solve new issues, Dreamer opens the door through which AI systems capable of optimizing real-world processes can continue. Hafner underscored this point on Twitter when he wrote, “[T]he main point of the algorithm is that it works out of the box on new problems, without needing experts to fiddle with it. So it’s a big step towards optimizing real-world processes.”
While Dreamer’s codebase is not yet public, Hafner has indicated it will be available in the near future. DreamerV2 is already on GitHub, giving a sense of just how cutting-edge this revolutionary technology is. DreamerV3 adds in features like improved replay buffers and a transition from TensorFlow to JAX, both of which make it even more scalable and awesome.
As machine learning continues to grow from strength to strength, software such as Dreamer is a harbinger of the future when computers will learn, develop, and master anything independently without any intervention from humans. In robotics, in video games, or in any other area, generalizing knowledge is a milestone towards building truly intelligent systems. Until that time, Dreamer’s diamond-gathering victory in Minecraft is both a milestone and a harbinger of what is to come.
For further details regarding Dreamer’s evolution and usage, see the Nature study, the InfoQ coverage, and the technical benchmarks that show it can do the work.
