Eight hands-on lessons

With the default settings the tool plays greedy: the single most likely action every step, on seed 1000. That is exactly the exam the teacher passed, five lives in a row on seeds 1000 to 1004, so it should clear. It mostly holds run-right and run-jump-right; the guide's action chart says the 1-1 teacher spends about two thirds of its steps on those two.

It does not go out of its way for coins. A coin pays 5 and at most 20 coins count per life, while running pays 1 per pixel and the flag pays 1,500 plus a time bank of 3 per second left. Detours lose. The mushroom pays 150, which is worth a small hop but not a long walk back.

--sample draws the action from the ten probabilities instead of taking the top one, which is how training plays. Expect more deaths: sampling adds mistakes, and a policy that is 90% sure at a pit still jumps early one time in ten. --seed 1005 only changes how many idle frames pass before the first input (0 to 8), yet that is enough to shift every enemy's timing, which is why the exam uses five seeds.

It does not look like a person. It never waits to see what an enemy does; it has learned timings, not intentions. The tile window lets it see a pipe or a pit one tile ahead, and that is the whole of its vision.

The network starts with its last layer near zero, so the ten probabilities are almost equal at 0.10 each. Mario jitters: a jump, a step left, a step right, a crouch. Four of the ten actions include Right and three include Left, so even random play drifts slowly right, which is why the first deaths happen at the first Goomba rather than at the spawn.

The reward does the rest. Every step to the right pays a pixel; every step alive costs a quarter point; a death costs 100 plus half the progress so far. Within about 100,000 steps, after roughly fifty updates, the right-moving actions have been paid more often than the others and their bars grow. This is the "right-hold" habit, and the alerts tool watches for it becoming the only habit: right-hold above 70% with jumps under 2% after a million steps is a warning sign, because a policy that never jumps cannot pass a pit.

Entropy falls slowly, not in a cliff. A cliff would mean the policy collapsed onto one button before it had seen enough, which the entropy bonus exists to prevent. The second run of the day passed 1-1 at step 901,120; its first training clears came well before that, because training play samples and sometimes gets lucky, while the exam wants the most likely action to clear five times in a row.

The time line drops to zero at the marker; nothing else changes for a while. Rewards already collected keep their old values, and the network only moves a little per update, so behaviour lags a knob by hundreds of thousands of steps. This is the most important lesson about tuning: the term you changed moves first, then behaviour, then the rest.

The behaviour change is smaller than most people predict. The time term was tiny: a sprint step pays 10 to 14 and the time cost was a quarter point, so it was never what made Mario hurry. Progress still pays a pixel per pixel, and the stuck rule still ends a life after 120 steps without new ground, now with the same clawback as a death. What the time term did was make standing still strictly negative. With it gone, waiting is free, so over time the agent hesitates longer in front of hazards, lives get longer, and more of them end by the in-game timer running out, which counts as a death. In a long run this can turn into the stall the first run of the day showed at a pipe in 1-2.

Every knob change is an event with a step number, so a bend in any curve can be traced back to what you did. Restoring −0.25 is another event.

At 50 a coin, twenty coins pay 1,000: two thirds of a clear bonus, for a stretch of stage that costs nothing to reach. The rule of thumb in the knobs help is to keep any side term's average per episode well under 30% of the clear bonus, and this breaks it on purpose. Over the following updates the agent learns that the blocks are worth a jump, and the coin line climbs.

The cap is the flat top on that line. After the twentieth coin in a life, further coins pay nothing, so no single life can earn more than 1,000 from coins. That is the cap doing its job. It cannot do the whole job, because coin blocks come back when Mario dies. At 5 a coin, a life of coins pays 100, less than the death it takes to reset them, so the die-and-repeat loop loses. At 50 a coin the loop pays 1,000 against a death of 100 plus half the progress, and near the start of the stage the progress is small. The loop wins, and the agent can learn to collect the early coins and then die on purpose, which shows as deaths at low x and a falling clear rate.

That is why both the weight and the cap matter, and why the clawback is charged on every ending that is not a clear. The first design paid coins nothing for exactly this reason. Setting the weight back to 5 makes the loop unprofitable again, but the habit takes time to unlearn.

The learning objective includes a bonus for entropy: the more evenly spread the ten probabilities, the larger the bonus, and this coefficient scales it. Raising it pays the network to hedge, so the tallest bars shrink and the small ones grow within a handful of updates. The entropy number climbs; 2.30 nats is ten equally likely actions and it cannot go higher.

Mario gets worse for a while, because a spread-out policy takes the wrong action more often. That is the trade the dial makes: more exploration now for a better chance of finding something new. It is the tool to use when the policy has collapsed onto one button too early, or after a demotion. The opposite move, lowering it, is for the end of a stage, when you want training play to match the greedy exam. Too high for too long and the agent never commits.

The approximate KL and clip fraction spike together at the change and then settle. That pair spiking after any knob change means the change was large for the learning rate.

This pipe is where the first run of the day spent eight million steps. Its policy had no tile window, so it could not see the pipe; it had to remember by x coordinate where to jump, and it never did. Instead it learned that standing there until the stuck rule ended the life was worth more than trying and dying, because dying late in a stage clawed back half the progress and a stall cost 20. Two fixes followed: the clawback is now charged on stalls too, and the observation gained a 7 × 5 window of solid tiles centered one tile ahead of Mario, which is on by default in the start script.

What your policy does depends on how far along it is. A young one hesitates and samples; you will see different outcomes from the same save, because training play draws the action from the probabilities. A trained one jumps the same way every time. Either way, nothing you did with the controller changed the network: steps taken while HUMAN is showing are flagged and dropped from the learning batch. The takeover is a probe, not a lesson for the agent.

Save and load only touch env 0. The episode continues from the loaded frame with its bookkeeping reset as if a new life began, which is the same mechanism every episode uses to start at the stage's first frame.

The shell. As the reward stood when the audit ran, the second stomp on the same Koopa paid a second kill, because the detector counted every change of the enemy's state into "stomped". The audit measured it on 1-2 at x ≈ 655: state 0 to 4, kicked to 0, then 0 to 4 again. A policy that does it on purpose repeats it about every three seconds, and the per-life cap of 30 kills is the only limit, so the trick is worth up to 600 a life, more with the 1-ups the shell combo eventually pays. The fix, D17, pays one kill per enemy for as long as that enemy exists, and lets the farm lock, which already zeroed score points earned while standing still, zero kills and 1-ups too.

The pipe. This one surprised the red team in the other direction. With the shortcut term capped at 800, the pipe route paid 1,093.5 for the trip and walking the same span paid 1,661.4, so the pipe was worth 568 less. The agent was not exploiting anything; it was correctly refusing a bad deal, which is why no teacher ever takes the pipe. The leak was subtler: the shortcut was not counted as progress, so the clawback on a death did not halve it, and a pipe trip that ends in a death beat a walk that ends in a death by 199. The fix, D18, lets the shortcut pay the full 1,699 pixels it replaces and adds it to the progress the clawback halves. With that, the pipe pays 1,992.5 against 1,661.4 for walking: the pipe wins by 331 plus the time it saves, which is what a shortcut should be worth.

So which pays more? Per trip, the pipe; per life, the shell farm can add up to 600 before the cap. Neither comes near a clear, which pays 1,500 plus a time bank of about 1,000 plus the flag on top of the pixels. That is the standard every finding was measured against: the audit ranked tricks by how likely PPO is to find them times how much they pay against a clear, and only the warp bootstrap and the shell farm were on a scale that changes what the learner does.

About the numbers you see. The scripts run the current reward code. If the hardening task T13 has landed, the shell trial pays one kill and the pipe route prints a shortcut of 1,699; if not, you will see the audit's numbers above. Either way the printed table is the reward's own accounting, term by term, which is the point of the exercise.

Refuses to play is doing the arithmetic right. Under its table a whole level of walking is worth about 320 and one death costs 300. The four ? blocks near the start of 1-1 pay 120 for almost no risk, and a life that then stands still loses only the stuck penalty of 20 and half of the little progress it banked. So the best plan it can see is: hit the blocks, wait. The stuck rate of 80% is not a bug in the learner. It is the reward being obeyed.

What broke the spell was a lucky life, not a weight change. With 16 Marios trying random things, one eventually reached the flag. The clear paid 1,500 plus a time bank near 800, ten times anything it had seen. The value network learned that the far end of the level is worth a lot, the policy followed the gradient toward it, and within 200,000 steps the runs were the majority. The curriculum's exam, five greedy runs on five fixed starts, then passed twice in a row, which a demoted run must do before promotion.

The demotion is the curriculum protecting the experiment. Every 500,000 steps on a stage above 1-1, if fewer than 2% of lives clear and the median life gets less than a quarter of the way, the run is sent to the stage before, seventy percent of its Marios at first, then all of them, until the earlier exam is passed. Both runs started on 1-2 and went back to 1-1 at exactly 501,760 steps. Both came back to 1-2 stronger. A switch to keep the stage is proposed as T17, for experiments that should stay where they were started.

The treasure hunter's greed is progress in disguise. Its items are spread along the level, so chasing them moves Mario right, and at 0.5 a pixel the walk pays too. It passed 1-1 three times faster than the other two. On 1-2 it took six million steps, against two million for the design default in run live-1, because it dies a lot: nine lives in ten. The big item bonuses make risky detours worth it, and the underground is full of them. It still got there, and it is now the panel's default personality because Eric liked what it did.

None of these is the right personality. They are three answers to the question the guide keeps asking: what does the reward make worth doing? A reward you can predict is one you understand. Change one number in the picker, write down what should change, and run it.