Machine learning

Reinforcement learning experiments

Q-learning and deep reinforcement learning in FrozenLake, CartPole, and a custom Tank Dodge environment.

PyTorch
Pygame
DQN
PPO
MLflow

Overview

Teaching an agent through consequences

Reinforcement learning is about learning what to do by observing the consequences of an action. This project explores that idea through simple control tasks and Tank Dodge, a custom game in which an agent must keep moving to avoid approaching enemies.

The agent receives a reward for surviving and a penalty for collision. Over training, it learns a movement policy from those signals rather than from a list of correct moves.

The collection progresses from tabular Q-learning to deep-learning approaches, connecting algorithm design with an environment that makes the agent’s behaviour visible.

Watch the agent

Saved PPO policy, seed 0, greedy actions on CPU. Blue is the agent; red squares are enemies. This is recorded playback of the actual environment, not a browser simulation.

Saved PPO policy versus random actions: 10 starting seeds, up to 1,000 steps per episode
Policy	Mean episode return	Mean episode length
Saved PPO checkpoint (greedy)	377.5	388.5 steps
Uniform random actions	224.5	235.5 steps

Both policies use seeds 0–9 with matched initial environment states. The random policy uses an independent action RNG. Return includes the −10 collision penalty. Explore the per-episode results.

Technical detail

Designing the learning problem

Tank Dodge exposes the player’s position, enemy positions, and relative positions as normalised coordinates. With three enemies, the observation contains 14 values. The policy chooses one of four movement directions.

Observe positions
Choose a movement
Advance the environment
Receive survival or collision reward
Update the policy

Using coordinates instead of pixels keeps the experiment focused on control and policy learning. The reward is +1 for a surviving step and −10 for a collision.

Comparing learning approaches

DQN learns action values using a replay buffer and an exploration policy. Double DQN separates action selection from value estimation. PPO uses an actor–critic model, generalised advantage estimation, and clipped policy updates.

Training and playback are separate workflows, allowing a saved policy to act without continuing to update its weights. The environment also supports headless execution for rollouts without rendering.

Learning over time

PPO episode rewards with moving and cumulative averages — PPO training rewards vary between episodes while the cumulative average improves. Training curves describe learning behaviour, rather than guaranteeing performance in every episode. Original artifact.

DQN episode rewards and rolling average during training — DQN training history. This run and the PPO training history use separate experiments and should not be read as a controlled head-to-head comparison. Original artifact.

The playback comparison above covers one saved PPO policy on ten starting seeds, with random actions as a baseline. Broader comparisons would need multiple independently trained policies and the same environment and training budget for each algorithm.