OpenAI Five
OpenAI Five is a
By choosing a game as complex as
History
Development on the algorithms used for the bots began in November 2016. OpenAI decided to use
By June 2018, the ability of the bots expanded to play together as a full team of five and were able to defeat teams of amateur and semi-professional players.
Architecture
Each OpenAI Five bot is a neural network containing a single layer with a 4096-unit LSTM[18] that observes the current game state extracted from the Dota developer's API. The neural network conducts actions via numerous possible action heads (no human data involved), and every head has meaning. For instance, the number of ticks to delay an action, what action to select – the X or Y coordinate of this action in a grid around the unit. In addition, action heads are computed independently. The AI system observes the world as a list of 20,000 numbers and takes an action by conducting a list of eight enumeration values. Also, it selects different actions and targets to understand how to encode every action and observe the world.[19]
OpenAI Five has been developed as a general-purpose reinforcement learning training system on the "Rapid" infrastructure. Rapid consists of two layers: it spins up thousands of machines and helps them ‘talk’ to each other and a second layer runs software. By 2018, OpenAI Five had played around 180 years worth of games in reinforcement learning running on 256
OpenAI 1v1 bot (2017) | OpenAI Five (2018) | |
---|---|---|
CPUs | 60,000 CPU cores on Microsoft Azure | 128,000 pre-emptible CPU cores on the Google Cloud Platform (GCP) |
GPUs | 256 K80 GPUs on Azure | 256 P100 GPUs on the GCP |
Experience collected | ~300 years per day | ~180 years per day |
Size of observation | ~3.3kB | ~36.8kB |
Observations per second of gameplay | 10 | 7.5 |
Batch size | 8,388,608 observations | 1,048,576 observations |
Batches per minute | ~20 | ~60 |
Comparisons with other game AI systems
Prior to OpenAI Five, other AI versus human experiments and systems have been successfully used before, such as
Long run view: The bots run at 30
Partially observed state of the game: Players and their allies can only see the map directly around them. The rest of it is covered in a fog of war which hides enemies units and their movements. Thus, playing Dota 2 requires making inferences based on this incomplete data, as well as predicting what their opponent could be doing at the same time. By comparison, Chess and Go are "full-information games", as they do not hide elements from the opposing player.[25]
Continuous action space: Each playable character in a Dota 2 game, known as a hero, can take dozens of actions that target either another unit or a position. The OpenAI Five developers allow the space into 170,000 possible actions per hero. Without counting the perpetual aspects of the game, there are an average of ~1,000 valid actions each tick. By comparison, the average number of actions in chess is 35 and 250 in Go.
Continuous observation space: Dota 2 is played on a large map with ten heroes, five on each team, along with dozens of buildings and non-player character (NPC) units. The OpenAI system observes the state of a game through developers’ bot API, as 20,000 numbers that constitute all information a human is allowed to get access to. A chess board is represented as about 70 lists, whereas a Go board has about 400 enumerations.
Reception
OpenAI Five have received acknowledgement from the AI, tech, and video game community at large. Microsoft founder Bill Gates called it a "big deal", as their victories "required teamwork and collaboration".[8][26] Chess player Garry Kasparov, who lost against the Deep Blue AI in 1997, stated that despite their losing performance at The International 2018, the bots would eventually "get there, and sooner than expected".[27]
In a conversation with
In 2019,
It was OpenAI's hope that the technology could have applications outside of the digital realm. In 2018, they were able to reuse the same reinforcement learning algorithms and training code from OpenAI Five for Dactyl, a human-like robot hand with a neural network built to manipulate physical objects.[31] In 2019, Dactyl solved the Rubik's Cube.[32]
References
- ^ OpenAI. "OpenAI Five". openai.com/five. Archived from the original on 1 September 2018. Retrieved 10 October 2018.
- ^ Savov, Vlad (14 August 2017). "My favorite game has been invaded by killer AI bots and Elon Musk hype". The Verge. Archived from the original on 26 June 2018. Retrieved 25 June 2018.
- ^ Frank, Blair Hanley. "OpenAI's bot beats top Dota 2 player so badly that he quits". Venture Beat. Archived from the original on 12 August 2017. Retrieved 12 August 2017.
- ^ OpenAI (11 August 2017). "Dota 2". blog.openai.com. Archived from the original on 11 August 2017. Retrieved 12 August 2017.
- ^ OpenAI (16 August 2017). "More on Dota 2". blog.openai.com. Archived from the original on 16 August 2017. Retrieved 16 August 2017.
- ^ a b Simonite, Tom (25 June 2018). "Can Bots Outwit Humans in One of the Biggest Esports Games?". Wired. Archived from the original on 25 June 2018. Retrieved 25 June 2018.
- ^ Kahn, Jeremy (25 June 2018). "A Bot Backed by Elon Musk Has Made an AI Breakthrough in Video Game World". Bloomberg.com. Archived from the original on 27 June 2018. Retrieved 27 June 2018.
- ^ a b "Bill Gates says gamer bots from Elon Musk-backed nonprofit are 'huge milestone' in A.I." CNBC. 28 June 2018. Archived from the original on 28 June 2018. Retrieved 28 June 2018.
- ^ OpenAI (18 July 2018). "OpenAI Five Benchmark". blog.openai.com. Archived from the original on 26 August 2018. Retrieved 25 August 2018.
- ^ Vincent, James (25 June 2018). "AI bots trained for 180 years a day to beat humans at Dota 2". The Verge. Archived from the original on 25 June 2018. Retrieved 25 June 2018.
- ^ Savov, Vlad (6 August 2018). "The OpenAI Dota 2 bots just defeated a team of former pros". The Verge. Archived from the original on 7 August 2018. Retrieved 7 August 2018.
- ^ Simonite, Tom. "Pro Gamers Fend off Elon Musk-Backed AI Bots—for Now". Wired. Archived from the original on 24 August 2018. Retrieved 25 August 2018.
- ^ Quach, Katyanna. "Game over, machines: Humans defeat OpenAI bots once again at video games Olympics". The Register. Archived from the original on 25 August 2018. Retrieved 25 August 2018.
- ^ OpenAI (24 August 2018). "The International 2018: Results". blog.openai.com. Archived from the original on 24 August 2018. Retrieved 25 August 2018.
- ^ Wiggers, Kyle (13 April 2019). "OpenAI Five defeats professional Dota 2 team, twice". Venture Beat. Archived from the original on 13 April 2019. Retrieved 13 April 2019.
- ^ a b Statt, Nick (13 April 2019). "OpenAI's Dota 2 AI steamrolls world champion e-sports team with back-to-back victories". The Verge. Vox Media. Archived from the original on 15 April 2019. Retrieved 15 April 2019.
- ^ Wiggers, Kyle (22 April 2019). "OpenAI's Dota 2 bot defeated 99.4% of players in public matches". Venture Beat. Retrieved 22 April 2019.
- ^ "Understanding LSTM Networks". colah's blog. Archived from the original on 1 August 2017. Retrieved 27 August 2015.
- ^ a b c OpenAI (25 June 2018). "OpenAI Five". blog.openai.com. Archived from the original on 25 June 2018. Retrieved 25 June 2018.
- ^ "Why are AI researchers so obsessed with games?". QUARTZ. 4 August 2018. Archived from the original on 4 August 2018. Retrieved 4 August 2018.
- arXiv:1707.06347 [cs.LG].
- ^ Gabbatt, Adam (17 February 2011). "IBM computer Watson wins Jeopardy clash". The Guardian. Archived from the original on 21 September 2013. Retrieved 17 February 2011.
- ^ "Chess grandmaster Garry Kasparov on what happens when machines 'reach the level that is impossible for humans to compete'". Business Insider. Archived from the original on 29 December 2017. Retrieved 29 December 2017.
- ^ "DeepMind's Go-playing AI doesn't need human help to beat us anymore". Verge. 18 October 2017. Archived from the original on 18 October 2017. Retrieved 18 October 2017.
- ^ a b Knight, Will (25 June 2018). "A team of AI algorithms just crushed humans in a complex computer game". MIT Tech Review. Retrieved 25 June 2018.
- ^ "Bill Gates hails 'huge milestone' for AI as bots work in a team to destroy humans at video game 'Dota 2'". Business Insider. Archived from the original on 27 June 2018. Retrieved 27 June 2018.
- ^ "Garry Kasparov's Twitter". 24 August 2018. Retrieved 24 August 2018.
- ^ Park, Morgan (11 August 2018). "How the OpenAI Five tore apart a team of Dota 2 pros". PC Gamer. Retrieved 25 May 2020.
- ^ Gault, Matthew (17 August 2018). "OpenAI Is Beating Humans at 'Dota 2' Because It's Basically Cheating". Vice. Retrieved 25 May 2020.
- ^ Statt, Nick (30 October 2019). "DeepMind's StarCraft 2 AI is now better than 99.8 percent of all human players". The Verge. Retrieved 25 May 2020.
- arXiv:1808.00177v5 [cs.LG].
- arXiv:1910.07113v1 [cs.LG].