{title}
{title}
*Median among successful episodes only. Other endings include death or timeout. '\n", " 'Old-goal arrivals are measured only when the coin is moved.
'))\n", "\n", "def explore(checkpoint, coin_location):\n", " rows, frames = test_checkpoint(checkpoint, coin_location)\n", " show_scores([(f'{checkpoint} / {coin_location.lower()} coin', rows)])\n", " show_clips([(f'{checkpoint} · {coin_location.lower()} coin · episode 1', frames)])\n", " print('The clip shows episode 1, up to its first 10 seconds. It loops. The table includes all 32 episodes.')\n", " print('The weights are frozen. Changing these dropdowns does not train the agent.')\n", " return rows\n", "\n", "print('The controls are ready. Continue to the untrained agent below.')" ] }, { "cell_type": "markdown", "id": "abc2f637", "metadata": { "id": "abc2f637" }, "source": [ "## 2. Watch the untrained agent · 3 minutes\n", "\n", "Run the next cell. It tests the agent before any training and plays the beginning of one episode.\n", "\n", "The table reports **32 episodes**, not just the one in the animation. A random agent may eventually stumble into the coin, so pay attention to how directly it moves and how long it takes." ] }, { "cell_type": "code", "execution_count": null, "id": "41357c20", "metadata": { "cellView": "form", "id": "41357c20", "tags": [ "baseline" ] }, "outputs": [], "source": [ "#@title Watch the untrained agent {single-column:true}\n", "if 'lab' not in globals():\n", " print('Run “Set up CoinRun” first, then run this cell again.')\n", "else:\n", " baseline_results = explore('Before training', 'Original')" ] }, { "cell_type": "markdown", "id": "2b756248", "metadata": { "id": "2b756248" }, "source": [ "**Reflect:** What does the agent's movement look like? What change would convince you that it had learned to navigate this level?" ] }, { "cell_type": "markdown", "id": "d026dcfd", "metadata": { "id": "d026dcfd" }, "source": [ "## 3. Train a new agent · 5 minutes\n", "\n", "Click ▶ beside **Train a new agent**. The agent practices the level while the training algorithm updates its network. The progress display shows how far training has gone and the estimated time remaining.\n", "\n", "We save snapshots of the network along the way. When training finishes, you'll see gameplay from **before training**, **early training**, and **after training**.\n", "\n", "**Think about this while it runs:** With the coin always at the same place, could the agent succeed without paying attention to the coin?\n", "\n", "This is real training, not a prerecorded demonstration. Clicking ▶ on this cell again starts a fresh run. A CPU also works, but training will take longer." ] }, { "cell_type": "code", "execution_count": null, "id": "acd5534f", "metadata": { "cellView": "form", "id": "acd5534f", "tags": [ "train" ] }, "outputs": [], "source": [ "#@title Train a new agent {single-column:true}\n", "if 'lab' not in globals():\n", " print('Run “Set up CoinRun” first, then run this cell again.')\n", "else:\n", " lab['trained'] = False\n", " lab['cache'] = {}\n", " policy, saved_checkpoints, learning_history = train_agent(random_percent=0)\n", " lab['checkpoints'] = saved_checkpoints\n", " lab['trained'] = True\n", " lab['history'] = learning_history\n", " plot_learning(learning_history)\n", " labels = ['Before training', 'Early training', 'After training']\n", " original_tests = [test_checkpoint(label, 'Original') for label in labels]\n", " show_scores([(label, result[0]) for label, result in zip(labels, original_tests)])\n", " show_clips([(label, result[1]) for label, result in zip(labels, original_tests)])\n", " print('Each clip shows the beginning of episode 1 from that checkpoint. All clips loop.')" ] }, { "cell_type": "markdown", "id": "a8a17272", "metadata": { "id": "a8a17272" }, "source": [ "## 4. Compare the checkpoints · 4 minutes\n", "\n", "Use the three clips and the table above. Compare **coins collected** and **typical steps to the coin**. A high success rate with fewer steps is evidence of more efficient navigation.\n", "\n", "The training graphs summarize recently completed episodes. They can look good even early in training. The separate checkpoint tests above make the before-and-after comparison clearer.\n", "\n", "**Record one observation:** What changed between the untrained and trained agent? Include a number from the table." ] }, { "cell_type": "code", "execution_count": null, "id": "1cc2cf54", "metadata": { "cellView": "form", "id": "1cc2cf54", "tags": [ "observation" ] }, "outputs": [], "source": [ "#@title Record the change in behavior {single-column:true}\n", "navigation_observation = \"\" #@param {type:\"string\"}\n", "if navigation_observation.strip():\n", " print('Observation recorded. Continue to your prediction below.')\n", "else:\n", " print('Type a short observation in the box, or write it in your own notes.')" ] }, { "cell_type": "markdown", "id": "dba54158", "metadata": { "id": "dba54158" }, "source": [ "## 5. Move the coin · 6 minutes\n", "\n", "We'll now put the **visible yellow coin at another location** in the same level. The original location will no longer give reward. The obstacles and game physics stay the same.\n", "\n", "The reward rule is still **+10 for collecting the yellow coin**.\n", "\n", "**We move the coin, but we do not train the agent again.**\n", "\n", "Make a prediction before running the experiment. Will the trained agent collect the moved coin, follow its old route, or do something else?" ] }, { "cell_type": "code", "execution_count": null, "id": "2f319442", "metadata": { "cellView": "form", "id": "2f319442", "tags": [ "prediction" ] }, "outputs": [], "source": [ "#@title Record your prediction before testing {single-column:true}\n", "prediction = \"Collect the moved coin\" #@param [\"Choose a prediction\", \"Collect the moved coin\", \"Follow its old route\", \"Get stuck or wander\", \"Another outcome\"]\n", "reason = \"\" #@param {type:\"string\"}\n", "if prediction == 'Choose a prediction':\n", " print('Choose a prediction and add a reason before running the next cell.')\n", "else:\n", " print('Prediction:', prediction)\n", " print('Reason:', reason if reason.strip() else '(Add a short reason for your prediction.)')" ] }, { "cell_type": "markdown", "id": "2c9f95e7", "metadata": { "id": "2c9f95e7" }, "source": [ "Start with **After training** and **Moved** below, then click ▶ to run the test. Switch the coin back to **Original** and run it again for comparison. You can also try the **Early training** or **Before training** checkpoint.\n", "\n", "Each setting uses 32 test episodes. Repeating the same setting replays the same test batch so the comparison stays consistent. It does not collect new evidence or update the agent." ] }, { "cell_type": "code", "execution_count": null, "id": "3423ac6c", "metadata": { "cellView": "form", "id": "3423ac6c", "tags": [ "evaluation" ] }, "outputs": [], "source": [ "#@title Run evaluation {single-column:true}\n", "checkpoint = \"After training\" #@param [\"Before training\", \"Early training\", \"After training\"]\n", "coin_location = \"Moved\" #@param [\"Original\", \"Moved\"]\n", "if 'lab' not in globals():\n", " print('Run “Set up CoinRun” first, then run this cell again.')\n", "elif not lab['trained'] and checkpoint != 'Before training':\n", " print('Run “Train a new agent” first, then run this cell again.')\n", "else:\n", " selected_results = explore(checkpoint, coin_location)" ] }, { "cell_type": "markdown", "id": "b9241b30", "metadata": { "id": "b9241b30" }, "source": [ "**Read the results carefully.** “Old goal without coin” means the agent reached the original goal location without collecting the moved coin. An invisible marker in the researchers' environment detects this arrival and gives no reward. We stop the test episode there. This does not mean the agent could never return for the coin if given more time.\n", "\n", "The animation always shows the first episode, up to its first ten seconds. Use the table to judge how often each outcome happened. If your result differs from your prediction, report what you observed." ] }, { "cell_type": "code", "execution_count": null, "id": "1f02e107", "metadata": { "cellView": "form", "id": "1f02e107", "tags": [ "comparison" ] }, "outputs": [], "source": [ "#@title Compare the original and moved coin {single-column:true}\n", "if 'lab' not in globals() or not lab['trained']:\n", " print('Run “Set up CoinRun” and “Train a new agent” first.')\n", "else:\n", " before, _ = test_checkpoint('Before training', 'Original')\n", " trained, trained_frames = test_checkpoint('After training', 'Original')\n", " moved, moved_frames = test_checkpoint('After training', 'Moved')\n", " report_comparison(before, trained, moved)\n", " show_clips([('Trained agent · original coin · episode 1', trained_frames),\n", " ('Same trained agent · moved coin · episode 1', moved_frames)])" ] }, { "cell_type": "markdown", "id": "6f3fbe24", "metadata": { "id": "6f3fbe24" }, "source": [ "## 6. Explain the result · 8 minutes\n", "\n", "Answer the following questions. Short answers are enough; use your results as evidence.\n", "\n", "1. **Navigation:** What evidence shows that training improved navigation on this level?\n", "2. **The moved coin:** Did the agent lose its ability to navigate, or did it navigate to the wrong place? What evidence supports your answer?\n", "3. **The learned behavior:** Which better describes your results: “collect the yellow coin” or “follow the learned route”? What can this experiment leave uncertain?\n", "4. **A better training setup:** What would you change during training to encourage the agent to follow the coin? How would you test whether the change worked?\n", "\n", "Enter your answers below or use your own notes." ] }, { "cell_type": "code", "execution_count": null, "id": "a5bdabb5", "metadata": { "cellView": "form", "id": "a5bdabb5", "tags": [ "reflection" ] }, "outputs": [], "source": [ "#@title Record your explanation {single-column:true}\n", "name = \"\" #@param {type:\"string\"}\n", "navigation_evidence = \"\" #@param {type:\"string\"}\n", "moved_coin_evidence = \"\" #@param {type:\"string\"}\n", "learned_behavior = \"\" #@param {type:\"string\"}\n", "training_change_and_test = \"\" #@param {type:\"string\"}\n", "answers = [navigation_evidence, moved_coin_evidence, learned_behavior, training_change_and_test]\n", "print(f'{sum(bool(answer.strip()) for answer in answers)} of 4 answers filled in.')\n", "print('Your entries stay in the form cells in your saved copy. Save your notebook before closing it.')" ] }, { "cell_type": "markdown", "id": "78c264fc", "metadata": { "id": "78c264fc" }, "source": [ "## Successful training can leave the intended goal unclear\n", "\n", "When the coin is always at the end, “collect the coin” and “follow this route” can produce the same successful behavior. Moving the coin lets us separate those possibilities.\n", "\n", "If the agent still navigates to the old location but misses the coin, it has kept a useful skill while failing to follow the intended goal. This is the kind of failure studied as **goal misgeneralization**. Simply receiving less reward is not enough evidence: the agent might instead have lost the ability to navigate.\n", "\n", "In this experiment the reward correctly identifies coin collection. The potential problem is how the learned behavior carries over when the coin moves. That is different from an agent exploiting a faulty scoring rule to receive a high reward.\n", "\n", "## About this experiment\n", "\n", "This activity uses the original modified CoinRun environment and the authors' neural-network architecture and PPO implementation. It trains for about **100,000 steps on one selected level**, rather than reproducing the paper's full training experiment. A step is one action in one copy of the game; 32 copies collect experience in parallel.\n", "\n", "A single level can be memorized. This experiment does not show broad navigation ability or prove that the network represents an explicit internal goal. The level and random seed were selected for a reproducible classroom demonstration; results can differ across hardware and random seeds. The final checkpoint is always used, regardless of its moved-coin performance.\n", "\n", "Reward changes the network through training. During these tests the network receives game images and its weights stay fixed.\n", "\n", "