Mario Learns to Run

A program plays Super Mario Bros. on an NES emulator, thousands of times an hour, and gets a little better each time. This guide explains how, from the whole machine down to the bytes, with pieces you can operate as you read.

One sentence

An agent looks at the game, chooses a button press, gets a small number back that says whether that was a good idea, and adjusts its habits so that good ideas become more likely. Repeat ten million times.

The loop

Everything in this project is one loop, drawn below. Press Step to walk one agent step through it with the real values from the first frame of World 1-1.

Two phases, one plan

Phase 1 (this guide, running now) lets the agent read the game's memory directly: exact positions and speeds, no pixels. That makes learning fast, and it produces a teacher. Phase 2 will train a second agent on screen pixels only, the way a person sees the game, and use the Phase 1 teacher to get it started. Both phases play the same stages in the same order: 1-1, 1-2, 1-3, 1-4, 2-1.

Where things stand

Terminology

The words used everywhere below. Each is defined the way this project uses it.

Agent

The thing that chooses actions. Concretely: a neural network plus the code that samples from it.

Environment

The game wrapped in a fixed interface: reset, step, reward, done. Here, one emulator per environment.

Observation

What the agent sees at one step. In Phase 1, 30 numbers decoded from RAM (65 with the tile window on).

Action

One of 10 button combinations, held for 4 NES frames.

Agent step

4 NES frames, about 1/15 of a second. Every count in this project is in agent steps.

Reward

The number the environment hands back after a step. Mostly pixels moved right, minus a little for time, plus event bonuses and penalties.

Return

The sum of rewards over an episode, or discounted into the future from a given step.

Episode

One life: from a stage's first controllable frame until death, clear, stall, or the step cap. No extra lives.

Policy

The mapping from observation to action probabilities. The actor network.

Value

A second network's estimate of the return expected from a state. The critic. Used to judge whether an action turned out better than expected.

Advantage

How much better an action's outcome was than the value predicted. Positive: do it more. Computed with GAE.

PPO

Proximal Policy Optimization. The learning rule: move the policy toward higher-advantage actions, but only a little per update.

Rollout

128 steps from each of 16 environments, collected before one update. 2,048 transitions.

Entropy

How spread out the action probabilities are. High: exploring. Near zero: always the same button.

Curriculum

The stage ladder and the rules for moving up (promotion) or back (demotion).

Greedy eval

Playing with the most likely action every time, no sampling. The exam for promotion.

Teacher

The Phase 1 policy saved at the moment a stage was passed. Phase 2 learns from it.

Latch

A one-frame detector for an event (death, clear) derived from RAM bytes.

Snapshot

The emulator's full state, saved in memory. Every episode starts by restoring the stage's snapshot.

SPS

Agent steps per second across all environments. About 1,300 on the 5090, 2,200 on the Mini.

Checkpoint

Everything needed to resume a run exactly: both networks, optimizer, normalizer, curriculum pointer, random state.

How a neural network learns

Before the map of the machine, the one idea everything rests on. A network is a big pile of dials, and learning is turning them, a little at a time, in the direction that would have made the last decision better.

A neuron is a weighted vote

Start with one neuron. It has a few inputs, each a number: say, how close the next pit is, and how fast Mario is moving. Each input has a weight, a number that says how much that input matters and in which direction. The neuron multiplies each input by its weight, adds the results up, and pushes the sum through a squashing function so the answer lands between 0 and 1. Read the answer as "how much do I want to jump right now?". A big positive weight on "pit is close" means a nearby pit pushes the answer up. A negative weight would push it down. That is the entire mechanism. There is nothing else inside a neuron.

A network is neurons stacked

The Phase 1 policy is 30 inputs feeding 256 neurons, feeding another 256, feeding 10 outputs. Every arrow between two layers is one weight, so this small network has about 75,000 dials (30 × 256, plus 256 × 256, plus 256 × 10, plus one bias per neuron). The first layer's neurons learn to notice simple things, such as "an enemy is close and to the right". The second layer combines those notices into situations. The last layer turns situations into a score for each of the ten buttons, and the scores become probabilities. Nobody designs what each neuron notices. That is what the learning does.

Learning is turning the dials, a little, in a known direction

Suppose the network chose "run right" and the outcome was better than the critic expected (a positive advantage). We want "run right" to be a bit more likely next time in this situation. Calculus can say, for every one of the 75,000 dials, which way to turn it to raise that probability, and how strongly each dial matters. That list of directions is the gradient. The learner turns every dial a small amount along it; the learning rate sets how small. If the outcome was worse than expected, it turns them the other way. Try it above: press learn and watch both weights move toward "jump", the one on the bigger input moving more.

Why the steps are small

One rollout is 2,048 steps of a noisy game. Some of what looked good was luck. If the network took a big step after each rollout it would swing wildly, and it would forget how to play the start of the stage while it learned the end. So the steps are small, the learning rate shrinks over the run, and PPO adds a fence: no action's probability may change by more than about 20% in one update. Slow and steady, ten thousand times.

The pieces you will meet below

• The actor is the network described above: observation in, ten probabilities out.
• The critic is a second network with one output: its guess at how many points are still to come. It exists so the learner can tell a good outcome from a good outcome for this spot.
• The advantage is what happened minus the critic's guess. Its sign decides which way the dials turn; its size decides how far.
• Entropy is how spread out the ten probabilities are. A small bonus for it keeps them from collapsing onto one button before the network has seen enough.

Everything else in this guide is plumbing that gets the right numbers to this one operation, fast, without lying. If you would rather learn by doing, the eight lessons start with watching a finished policy play.

System map

Two machines, a handful of processes, and the files that connect them. The arrows are what actually moves.

The people and agents

This project is built by a small team of AI sessions with one human owner. Eric owns the goal and tunes. The lead (a Claude session) holds the design, reviews every commit against it, dispatches tasks, and starts runs. Opus (another Claude session) writes the trainer code, test-first. Grok writes machine tooling: scripts for the 5090, the recorder, plots, the browser panel, the emulator bridge. Chief relays between Eric's voice line and the lead through files in comms/. A launchd job wakes the lead every ten minutes so the project moves between conversations.

The components

Each card is one Python module or tool with one responsibility. The 100-foot sections below open each one.

Emulator and RAM

The game runs inside nes-py, a C++ NES core with a Python interface. It gives us three things the loop needs: run one frame with buttons held, read all 2 KB of RAM, and save or restore the whole machine.

Why not just read the screen?

The screen is what a person sees, and Phase 2 will use it. But the game's memory already contains Mario's exact position, speed, and the positions of every enemy, in a form a small network can learn from in minutes instead of days. Phase 1 reads those bytes. The bytes below are addresses in the NES CPU's memory on this exact ROM; a different dump would move them, which is why the ROM's SHA-256 is pinned and any other file refuses to boot.

Two latches that took care to get right

Death fires on the first frame the player state reads dying or Mario drops below the playfield. The obvious alternative, waiting for the lives counter to go down, fires 243 frames later, after the whole death animation, which would make the reward for the death step ambiguous. Clear fires the frame Mario grabs the pole (climbing state with a flagpole object present) or touches the castle axe. The timer-drain latch from the first design fires 159 frames later, after the slide and the walk to the castle; both numbers were measured on the real ROM with a scripted clear.

Buttons

The NES controller is eight bits. This project writes them in the game's own order; nes-py wants them reversed, and the emulator layer translates. Start and Select are masked off outside boot so the agent can never pause. Toggle bits to see the two bytes and which of the ten actions the combination is.

Snapshots and stage starts

Every episode starts by restoring a snapshot of the emulator taken at the first controllable frame of the stage. The snapshot is built once per process by a recipe: power on, tap Start past the title screen, poke the desired world and stage into memory while the game initializes, and stop on the first frame with the timer at 400 and Mario in the normal state. The RAM at that frame is hashed and the hash is pinned, so a changed ROM or a drifted recipe fails loudly instead of training on the wrong level. One subtlety found on the real ROM: nes-py's reset is a soft reset that keeps RAM, so the layer restores a true power-on snapshot instead.

Environment

The environment is the game with a fixed contract: reset() gives an observation, step(action) gives the next observation, a reward, and whether the episode ended.

One step, in order

1. Hold the action's buttons for 4 NES frames.
2. Read RAM and check the death and clear latches first. The flagpole grab puts Mario into an auto-walk state; if busy frames were skipped before checking, the clear would be missed and the next stage's load would look like a warp.
3. If Mario is in a state the player cannot control (pipe transit, auto-walk), run those frames out with no button held. They are not agent steps and pay nothing.
4. Decode everything, detect events, compute the reward.
5. Decide how the step ends.

How an episode ends

The distinction matters. A terminated episode really ended: there is no future to value. A truncated one was cut off by us, so the learner estimates what would have followed. Getting these backwards teaches the agent that a timeout is death.

Sixteen at once

Sixteen environments run in sixteen processes, one emulator each, so the trainer collects 16 steps at a time. When one episode ends, that environment resets immediately and its next observation is already the new episode's first frame; the ended episode's last frame travels alongside in the info so the learner can still value it.

Observation

Thirty numbers describe the moment, 65 with the tile window. The agent never sees a pixel in Phase 1.

A running mean and variance normalize each number before it reaches the network, updated only from training data and frozen during exams. The tile window was added after today's run showed the agent cannot see a pipe: without it, the policy must memorize where obstacles are by x coordinate alone, which worked for 1-1 and failed at a pipe in 1-2.

Reward

The reward is the whole specification of what "playing well" means. Every term here is a decision, and each one has been broken by the agent at least once in the design's history.

The terms

Not potential-based, on purpose

A textbook shaping reward uses a potential function so that any policy optimal under the shaped reward is optimal under the original one. This reward is deliberately not that: progress is undiscounted, clipped, and zeroed on teleports. The code says so in a comment so that nobody "fixes" it into a potential and silently retunes every weight.

PPO: how the numbers change the network

Two small networks. The actor turns the observation into ten action probabilities; the critic guesses the return. Every 2,048 steps both are nudged.

The networks

Each is 30 → 256 → 256 → out, ReLU between, orthogonal initialization. The actor's last layer starts near zero so the first policy is nearly uniform over the ten actions. The critic's output is multiplied by a fixed 100 because returns here are in the thousands and a unit-scale head would take a hundred thousand updates to reach them; that was measured, not guessed. They share no weights, so the +1,500 clear bonus cannot drag the policy's features around.

Advantage: was that better than expected?

For each step the learner asks how much better the outcome was than the critic predicted. GAE blends one-step surprises with longer horizons using two knobs: γ (how much the future counts) and λ (how far the credit spreads). Edit the toy trajectory and move the knobs.

The update

With advantages in hand, PPO raises the probability of actions with positive advantage and lowers the others, but clips the change to a ratio of 0.8 to 1.2 per action, so one lucky rollout cannot overwrite the policy. It also adds a small bonus for entropy, keeping exploration alive, and trains the critic toward the return targets. If the policy moves too far in one update (approximate KL above 0.03) the epoch loop stops early. Advantages are standardized per minibatch; rewards are never normalized, because the weights above were chosen in raw units.

Curriculum

Stages are learned in order. The rules for moving decide how the agent's time is spent; the diagram is the whole rulebook.

When a stage is passed, the actor and its normalizer are saved as teachers/ram/<stage>.pt and never overwritten; Phase 2 will learn from exactly the policy that passed, not from a later one that has moved on to harder stages and plays 1-1 wrong.

Live view and tuning

Training is headless by nature: sixteen processes stepping emulators as fast as they can. The live view opens a window into one of them and a channel back.

Environment 0 copies every frame it renders, plus a small state record, into a shared-memory block. Any number of readers can attach: the native window (tools/live.py), the browser panel (tools/panel.py). The block has one writer per region and a sequence lock, so a reader never sees half a frame. A reader can also write two things back: an override for the buttons, which lets you drive Mario while the other fifteen environments keep training, and a request to save or reload env 0's snapshot.

Tuning goes through runs/<name>/control.json. The trainer reads it at every rollout boundary and applies what changed: entropy coefficient, learning rate and its horizon, any reward weight, the stuck window, pause, slow motion for env 0, checkpoint now. Every applied change is written to events.jsonl with the step it took effect, and marks the CSV row, so a bend in a curve can always be traced to a knob.

Steps you drive are recorded with a human flag and excluded from the learning batch, so driving teaches nothing directly; it lets you put the agent in a situation and watch what it does next.

What happened today

The first real run, 10 million steps on the 5090, 2 hours 3 minutes. It solved 1-1 and then taught us something about the reward.

The diagnosis

Twelve greedy episodes with the final policy all ended stuck at x ≈ 1826 or x ≈ 1874 in 1-2: the base of a short pipe in the corridor after the piranha plants. Sampled play got past the pipe a quarter of the time and then died at the next hazard, x ≈ 2130. Compare the returns:

Under the reward as it stood, camping was worth twice as much as trying. The learner did exactly what it was paid to do. The clawback exists to make suicide at the flag lose to clearing, and it works; but because it grows with progress, a death late in a stage becomes far more expensive than a stall, and stalls were nearly free. 1-1 never exposed this because its hazards come early, when there is little to claw back.

The fix (decision D13): charge the same clawback on a stuck ending. Now camping at the pipe returns about 887 and trying returns about 845: a wash, so exploration decides, and the attempt has the upside of eventually clearing. The reward calculator above has both cases preloaded.

Run 2: the fix, and the rest of the curriculum

Run 2 started from scratch on the Mac Mini with three changes: the clawback on stuck endings (D13), the score-style rewards (D14), and the tile window in the observation. It did not stop at 1-2.

The exam tallies say something about how promotion works: 1-1 passed on its ninth 100k check, 1-2 on its ninth, 2-1 on its sixteenth. Training play was clearing far earlier each time; the exam waits for the most likely action to be reliable on five fixed seeds in a row.

What the red team found

A second session spent an hour on the real ROM trying to earn reward without playing well. Five of its findings became design decisions D16 to D20 before any of them cost a run.

What a reward red team is

The reward is a contract: it says what "playing well" pays. The learner is a very literal reader of that contract, and it will find any clause that pays without the flag. A red team reads the contract the same way, on purpose, before the learner does. It scripts the suspicious move on the real game, runs it through the real reward code, and writes down what it paid next to what an honest clear pays. If a trick pays more than trying, or pays with no risk, that is a finding, and the fix goes into the design before the next run. The first run of the day found one such clause by itself, the pipe stall, and it cost eight million steps. This time the tricks were found in an afternoon by hand.

The five accepted findings

Two more were checked and closed without a fix: the question-block-then-die loop pays about 78 per episode and never beats continuing, so the clawback holds; and piranha plants never show a kill state through three full 1-2 clears, so their in-and-out cycle is not a leak. The whole audit, with the measured tables, is docs/redteam/2026-09-26-reward-audit.md; the code changes are task T13.

Why the warp bootstrap was wrong

Phase 2 preview: pixels

The same game, the same reward, the same stage ladder, but the agent sees the screen instead of the memory. The plan is docs/phase2/PLAN.md; tasks T20 to T25.

What changes is only the front of the network. Every agent step, the four NES frames are turned gray, merged by taking the brightest pixel at each spot (so flickering sprites do not vanish), shrunk to 84 by 84, and stacked with the three previous steps. Four small gray pictures, oldest first, are the whole observation: no positions, no speeds, no tile window. A convolutional network, the same shape used for Atari since 2015, has to learn to see Mario, the ground, the pits and the enemies in those pictures, and to infer speed from how they shift between frames. That is far harder than reading the numbers, so the budget is 50 million steps for 1-1 alone, with a cap of 100 million, and a stop-the-line gate: under 20% greedy clears at 15 million steps means the pipeline is broken, not underfunded. On the 5090 at about 1,200 steps per second, the gate is roughly three and a half hours in and 50 million is a working day.

The Phase 1 teachers make this tractable. While the pixel agent trains, the saved RAM teacher for the current stage watches the same moments through its own 65 numbers and says what it would do. An extra loss pulls the pixel network's ten probabilities toward the teacher's, weighted by a dial β that starts at 0.5 and fades to 0 over the first two million steps on each stage, after which the pixel agent is on its own. You will be able to watch all of it: the four stacked frames drawn beside the real frame in the viewer and the panel, the 32 first-layer filters as a grid image at every checkpoint (edge and ground detectors should appear by about two million steps), the teacher-agreement number rising, and eval videos with the stack composited under the picture so a clear can be seen from both views.

How to run it

Everything runs from ~/mario-rl with uv. The ROM lives at roms/smb1.nes and is never committed.

Watch a trained policy play

uv run python tools/play.py --model teachers/ram/1-1.pt --stage 1-1
uv run python tools/play.py --model runs/p1-1-1-10m/checkpoints/ckpt-0010000384.pt --stage 1-2

Train on the Mini with the live window

uv run python -m mario_rl.train --phase 1 --stage 1-1 --steps 10000000 --envs 16 --live --obs-tiles --out runs/live-1
# in another Terminal:
uv run python tools/live.py --run runs/live-1
uv run python tools/panel.py --run runs/live-1      # then open http://localhost:8765

Read a run

uv run python tools/plot_run.py runs/live-1      # PNGs in runs/live-1/plots/
uv run python tools/alerts.py runs/live-1        # the failure-mode checks, one line each

Unattended runs on the 5090

scripts/sync-5090.sh                              # copy the tree, run the tests there
scripts/start-5090-run.sh p1-x --phase 1 --stage 1-1 --steps 10000000 --envs 16
scripts/watch-5090.sh p1-x                        # pulls metrics and plots every 30 min
scripts/pull-run.sh p1-x --videos 6 --ckpt latest --teachers

Check the machine before a long run

uv run pytest -q                                  # the whole suite, about 25 s
uv run python tools/boot_asserts.py               # five checks on the real ROM

Every knob, reward, and chart, explained

The same text the panel shows behind each help icon, from one file: docs/help.json. Switch on plain words for a first-time reader.

Decision log

Every place this design departs from the first draft, with the reason. The full text is in DESIGN-v2.md.

File index