Slides

WEEK 06 · <time datetime="2026-10-20">Tuesday, 20 October 2026</time>

# Designing feedback loops for AI agents

How agents use feedback within a turn and pursue a goal across turns.

---

# Anatomy of a feedback loop

A feedback loop uses observations of the current output to guide the next revision.

| Component | Function |
| --- | --- |
| **Target** | Describes the desired result |
| **Action** | Changes the current state or produces a candidate |
| **Observation** | Provides information about the result |
| **Decision** | Determines the next action or ends the loop |

Feedback connects one action to the next. A stopping condition can be success,
a resource limit or an inability to make further progress.

---

# Inner loop and outer loop

A goal mode maintains the outer loop. One completed turn can still leave the goal unfinished.

Course schematic based on OpenAI's <a href="https://developers.openai.com/api/docs/guides/agents/running-agents">agent loop</a> and <a href="https://developers.openai.com/cookbook/examples/codex/using_goals_in_codex">Goals in Codex</a>.

<aside class="notes">
Here, inner means the model–tool–observation cycle within an agent turn; outer
means the continuation policy across turns. These are relative terms, and other
architectures may use them differently. The model proposes tool calls; the
runtime executes permitted calls and returns their results. A turn can contain
many such cycles. The outer loop retains an objective, uses the turn result to
assess progress and starts another turn when appropriate. Its controller can be
a person, a script or the agent runtime. Tool execution remains subject to the
application's permissions. The diagram abstracts away parallel tools and handoffs.
</aside>

---

# Goal loops: persistence across turns

A **goal** states the desired outcome. A **goal loop** keeps that objective active
while the agent works out the next step.

| During one turn | Between turns |
| --- | --- |
| The agent uses tools and observations to do the current work | The outer loop assesses whether the goal is complete |
| A final response ends the turn | Remaining work can trigger another turn |

The objective and progress persist. A new turn does not necessarily start with
a fresh context.

**Examples:** Codex and Cursor expose `/goal <objective>`. Availability and
stopping controls depend on the tool.

<a href="https://developers.openai.com/cookbook/examples/codex/using_goals_in_codex">Codex goals</a> · <a href="https://cursor.com/docs/cli/reference/slash-commands">Cursor CLI commands</a> (goal support is rolling out).

<aside class="notes">
Completion, pausing, budget exhaustion and a blocker are different outcomes.
The outer loop determines continuation; it does not make the model's judgment
of success infallible.
</aside>

---

# Feedback and independent evaluation

A score is a **proxy** for the intended goal. Optimizing it can leave the goal unmet.

| Evidence | Role in the loop |
| --- | --- |
| Deterministic checks | Test explicit properties under fixed conditions |
| Model judgments | Suggest flaws or revisions that require verification |
| Independent evaluation | Assess the frozen result using evidence withheld from selection |

Evidence used to select revisions is **development feedback**. Once it influences
another revision, it no longer provides an independent final assessment.

<aside class="notes">
Model-generated proposals, retrieved reference material and deterministic
test results have different origins. External documents and tool outputs
cross a trust boundary: their contents do not gain authority to direct the agent.
An independent assessment uses evidence withheld from the revision process.
</aside>

---

# Ralph loop: fresh sessions

Ralph is an outer loop that starts a **fresh agent context** each round.
Saved files carry progress forward.

**Saved requirements and files → fresh agent session → updated files → next session**

Each session contains its own inner model–tool loop. The next session recovers
the work from files rather than inheriting the previous conversation.

Goal modes and Ralph both support continued work. Ralph additionally specifies
how context is restarted between rounds.

Geoffrey Huntley, <a href="https://ghuntley.com/ralph/">Ralph Wiggum</a>. Goal comparison: <a href="https://developers.openai.com/cookbook/examples/codex/using_goals_in_codex">Codex goals</a>.

<aside class="notes">
Huntley's minimal example is a shell loop that repeatedly launches a coding
agent from a saved prompt. Requirements, the task plan and files preserve state
across sessions. The shell loop itself does not establish that the goal is
complete; stopping conditions depend on the surrounding implementation.
Fresh context can reduce accumulated conversational baggage, but useful
information that was never saved is unavailable in the next session.
</aside>

---

# Gauntlet loop: builders and critics

Gauntlet organizes feedback around a **builder** and a **separate critic**.

1. A lead agent divides the goal into parts that can be assessed separately.
2. A builder produces a candidate.
3. A critic with fresh context compares the output with a reference.
4. The largest remaining gap guides the builder's next revision.

Each agent has an inner tool loop. The builder–critic cycle forms an outer
improvement loop, continuing toward the target until completion or a human stop.

Matt Shumer, <a href="https://somethingbig.ai/gauntlet-loop">How to Run a Gauntlet Loop</a>. A separate critic still supplies a model judgment.

<aside class="notes">
The critic receives the goal, criteria, reference and actual output without
the builder's reasoning history. Shumer describes blind comparisons where
practical and continued revision rather than a fixed number of rounds. Fresh
context separates roles but does not establish objective correctness.
Critic feedback is development feedback because it selects the next revision.
</aside>

---

# Case study: the humanifier

**Research aim:** edit AI-generated text so an independent detector stops flagging it,
while preserving its meaning.

| Part of the experiment | Role |
| --- | --- |
| Coding assistant | Implemented candidate Python text-editing rules |
| Two local detectors and preservation checks | Supplied feedback for selecting rules |
| Independent detector | Assessed transfer beyond the local feedback |

Later experiments also tried LLM rewrites and translation. The following
post-mortem examines the results.

This was a research hypothesis. Editing leaves the text's AI-generated provenance unchanged. <a href="humanifier_research_memory.html">Experiment ledger</a>.

<aside class="notes">
The initial setup fixed 48 synthetic texts, an identity baseline, two detector
versions and train/validation/test splits. The coding assistant helped implement
and inspect candidate functions. That initial runtime applied Python rules to
saved strings without generative LLM calls. Later methods were separate
experiments. Both preservation and transfer were required for the intended goal.
</aside>

---

# Humanifier post-mortem

<video class="course-video" controls muted playsinline preload="metadata" width="1920" height="1080" poster="assets/videos/week-06-humanifier-postmortem.jpg" aria-label="Silent captioned post-mortem of the humanifier experiments: agent workflow, local detector scores, meaning changes and external Pangram checks.">
 <source src="assets/videos/week-06-humanifier-postmortem.mp4" type="video/mp4">
 <a href="assets/videos/week-06-humanifier-postmortem.mp4">Open the post-mortem video</a>.
</video>

<a href="assets/videos/week-06-humanifier-postmortem.mp4">Open video separately</a> · 6 min 39 s · silent, captioned · evidence through 17 October 2026

<aside class="notes">
The video distinguishes local measurements, automated judgments and saved
external browser results. The paper-to-function sequence illustrates the
workflow; it does not claim that every function came from a particular paper.
Detailed protocols and negative results remain in the linked experiment ledger.
One selected, model-generated text illustrates the difference between feedback
and independent evaluation: Fakespot fell from 91.9% to 4.4% and Desklib from
89.9% to 1.9%. Both local detectors guided editing and aggregate fixed sections.
Pangram 4.0 labelled 100% of the frozen rewrite as AI on 15 October 2026. The
original was not tested in that check. These percentages are not authorship
probabilities or a corpus-wide failure rate. The video shows this comparison.
<a href="assets/videos/week-06-postmortem-evidence.json">Exact texts and measurements</a>
and the <a href="assets/videos/week-06-pangram-result.jpg">saved Pangram result</a>
remain available for discussion.
</aside>

---

# Weekly lab · revising a graphical tree

<video class="course-video" controls muted playsinline preload="metadata" width="1920" height="1080" poster="assets/videos/week-06-tree-feedback.jpg" aria-label="Silent captioned video of ten Scots pine model revisions, with reference photographs, before-and-after renders, assistant feedback, code changes and regressions.">
 <source src="assets/videos/week-06-tree-feedback.mp4" type="video/mp4">
 <a href="assets/videos/week-06-tree-feedback.mp4">Open the tree feedback video</a>.
</video>

<a href="assets/videos/week-06-tree-feedback.mp4">Open video separately</a> · 2 min 56 s · silent, captioned · <a href="tree_growth_crown_results.html">Recorded revisions and limitations</a>

<aside class="notes">
The teacher video shows three parameter loops and seven code loops. Loop 09
creates separated high tips; loop 10 corrects them. The final blinded model
comparison is loop 06 versus loop 10, not the original baseline. These are
model judgments against development photographs, not biological validation.
Students use the separate simple implementation and try two code revisions.
</aside>

---

# Start the student model

From the course folder containing `pyproject.toml`, update and launch:

```bash
uv run scripts/update.py
uv run --locked --group trees python -m aiforlifescience.tree_growth \
  serve --variant student
```

Read the updater summary before launching. Review incoming copies in
`.course-updates/` if requested.

Open [the simulator](http://127.0.0.1:8066). In a second terminal, save a baseline:

```bash
uv run --locked --group trees python -m aiforlifescience.tree_growth \
  pack --variant student --seed 7 --stage 1 \
  --output artifacts/week06/tree-growth/student-baseline
```

Copy `src/aiforlifescience/tree_growth/student.py` into that checkpoint too.

<a href="tree_growth_student.html">Full student walkthrough</a> · <a href="tree_growth_lab.html">Commands, photo sources and optional PPQ setup</a>.

---

# Make two code revisions

1. Compare the baseline with the reference photos. Choose one visible mismatch.
2. Ask the coding assistant to change one rule in `student.py`.
3. Inspect the before/after code, test the geometry and render again.
4. Judge the result yourself. Keep or revert the change, then repeat.

Keep parameters, seed, stage and cameras fixed. Save code copies, renders,
feedback and test results in a new `artifacts/week06/tree-growth/` folder per round.

Submit the baseline and final evidence, with one improvement and one remaining flaw.

Treat images and model feedback as evidence to assess. The goal is photographic resemblance; the model does not establish biological growth. <a href="tree_growth_student.html">Checkpoint and submission details</a>.

---

# References

[Experiment ledger](humanifier_research_memory.html) ·
[Evidence snapshot](assets/videos/week-06-postmortem-evidence.json) ·
[Research papers](humanifier_literature.html)

[Tree lab walkthrough](tree_growth_student.html)

[Paper Warfare · Idiap Multiplayer Lab](https://www.idiap.ch/~wdroz/idiap_cafe_multi/)

[Course overview](presentation.html) · [Course map](course_map.html)

[Previous: Week 5](week_05.html) · [Next: Week 7](week_07.html)