About

Getting models to play games is fun. This implementation trains Deep Q Networks of various dimensions & architectures on a couple different OpenAI Gyms. I did this because I was curious about the relationship between the complexity of the games and the size of the network architecture that would achieve the best result within those games in a low sample environment.

What is a Deep Q Network (DQN)?

The primary element in this project is a simple Deep Q Network (DQN) that is capable of taking and producing variable dimension inputs and outputs, and whose internal hidden layers can be increased in size and number. This DQN also has access to replay memories of previous state-reward-next state combinations from previous episodes within the same game. Finally, this variable DQN then can be applied to different OpenAI Gym reinforcement learning games, and be tested with varying network sizes to determine the relative strength of network sizes with respect to each game.

How did I go about testing this?

First, I implemented the simple DQN agent with variable hidden layer width and depth, and fixed all other hyperparameters. The dimensionality of the agent’s network is also dependent on the game’s given observation and action spaces, thus resulting in final network architectures like that pictured in fig. 1 (DQN for Acrobot-v1 with 4 hidden layers of width 256).

The DQN has an exponentially decaying epsilon greedy regime (with an episode-wise decay factor of 0.99), and trains on a batch size of 32 after each action step within an episode. Learning rate and gamma are kept at 0.01 and 0.99 respectively. State Reward Replay memory capacity was capped at 10,000 of the most recent memories.

This DQN is then used iteratively over each game and network architecture, recording reward and loss at the end of each episode. In keeping with the small data requirement of this assignment, the network trains for only 500 episodes on each game and network size pair. The three games tested were CartPole-v1, LunarLander-v2, and Acrobot-v1. Networks combinations of hidden layer width 256, 128, 64, 16, and depth of 8, 4, 2, 1 were all tested.

What were the results?

As expected there was a clear relationship between game complexity and best performing network size, with the simplest game (by input and output dimensions) being Cartpole-v1, and the most complex, Acrobot-v1. There is an obvious tradeoff for larger DQNs on simple tasks, as they produce much more variability, if they even train at all, in this low sample environment:

Here, the 256x8 DQN fails to train at all in the 500 episode window, whereas both the 256x2 and 16x1 DQNs do train on Cartpole, however with substantially differing behaviors. The larger network produces a much greater variability in outcomes, likely a product of weak gradient flow over the small dataset. This is in stark contrast to the extremely simple single hidden layer 16x1 network which produces far less variable outcome. Given a much longer training window, it seems probable that the larger networks would surpass the performance of the smaller networks (even on Cartpole, the simplest of the three games).

In LunarLander, some sufficiently large networks were able to train to varying degrees of success, however the smallest and largest networks could not effectively learn. This difference, when compared to the outcomes of Cartpole, point to a minimum requirement in network complexity for LunarLander above that of the smallest network sizes (16x1, 64x1). Another interesting observation from LunarLander is the bifurcation of reward outcomes. This might be explained by the way LunarLander is scored, as certain criteria (like lander-leg preservation) have binary, heavily weighted score outcomes that can lead to high reward variability.

Finally, Acrobot-v1, the most complex game of the three, proved quite challenging over these small training sizes, but produced some interesting results:

Once again, none of the smaller networks succeeded in training on this game. Beyond that, there was some strange behavior. Take the 128x1 example: for almost 250 episodes, the model is incapable of scoring points, yet suddenly at episode 250 the model suddenly begins to perform. The inverse is true for 256x2 - the model starts strong, but collapses at some point around episode 350. One potential explanation for this behavior is an underlying failure of the network architecture: a simple DQN, no matter the network size, is probably not well suited for more complex challenges like this one.