Assignment 2 — Build an agent harness (10%)

Out Mon Sep 21 · due Sun Oct 4, 11:59pm · all five assignments

You will build madsLoop, a minimal coding-agent harness, and test it with SWE-bench Pro. An agent is a model in a loop. It proposes tool calls, and your harness executes them and feeds the results back. This task aims to build everything you need beyond the model API.

The model API

We use the course proxy at api.cs2680.com, an OpenAI-compatible endpoint, to serve the requests.

Export your key in the shell you run from:

export CS2680_API_KEY="hyi-..."     # your key

import os
from openai import OpenAI

# Read the key from the environment. Never hardcode it and never commit it.
# Your agent runs inside the task container, so you have to pass it 
# (e.g. `docker run -e CS2680_API_KEY=...`, see 2.3).
CS2680_API_KEY = os.environ["CS2680_API_KEY"]

client = OpenAI(
    base_url="https://api.cs2680.com/v1",
    api_key=CS2680_API_KEY,
)
resp = client.chat.completions.create(
    model="qwen3.6-35b-a3b",
    messages=[{"role": "user", "content": "hello"}],
    tools=[...],         
)

In this assignment, qwen3.6-35b-a3b will serve as the backbone model and should be used for all evaluations.

Set up the environment

The agent you designed will runs inside Docker containers: each task ships as a prebuilt image with the repository and its dependencies, your agent runs inside that container, and the official evaluation runs in another one too. Make sure Docker is installed and that you can run it without sudo; the ten provided task images are about 1.6–1.8 GB each.

• An x86-64 Linux host is the easy path. The images are built for linux/amd64 and run natively.
• Apple Silicon (M1–M4): the images run under emulation. Set PLATFORM_FLAG="--platform linux/amd64" in the shell you run the scripts from. run_task.sh passes it to docker pull and docker run. Expect slower runs.
• Windows: use WSL 2 with Docker Desktop's WSL integration, and run everything from the Linux side.
• Prefer a Linux machine? You can use the course cluster, see the cluster instructions.

If you encounter difficulty setting up the environment, please reach out to the TFs.

Part 0 — The interface and logging

0.1 The interface

python madsLoop.py -p "<problem statement>" [--log] <workdir>

• -p — the issue text: the problem_statement field of the task's entry in evaluation_scripts/agent_task_input.json. This is the task description your agent receives.
• <workdir> — path to the repo checkout. Your agent may read, search, edit, and run commands only inside this directory. We will use /app as an example.
• --log — trace logging, off by default. When set, your agent writes its run trace into a madsLoop_logs/ directory in the current working directory (create it if it doesn't exist), see schema in 0.3. We pass --log when we evaluate your agent.
• When the agent finishes, the fix exists as unstaged changes in <workdir>, and later patches will be generated by git diff.

0.2 Developer logs (developer_logs/)

Your submission includes a developer_logs/ directory documenting how you built madsLoop with a coding agent. The logs should be in .jsonl format. If you use a coding agent through a terminal or IDE, you can typically find the session logs in the following locations:

0.3 Agent logs (madsLoop_logs/)

When run with --log, your agent should write its trace into ./madsLoop_logs/ as JSONL (one file per run, e.g. madsLoop_logs/run1.jsonl): one JSON object per line, one line per event, in order, from which the whole run can be reconstructed. Every line carries timestamp (ISO-8601) and event; the remaining fields depend on the event type:

Example lines:

{"timestamp": "2026-09-12T14:03:07+00:00", "event": "run_start", "model_id": "qwen3.6-35b-a3b", "workdir": "/app"}
{"timestamp": "2026-09-12T14:03:11+00:00", "event": "api_request", "iteration": 1, "prompt_tokens": 1834, "completion_tokens": 96, "total_tokens": 1930}
{"timestamp": "2026-09-12T14:03:11+00:00", "event": "tool_call", "iteration": 1, "tool_name": "bash", "arguments": {"command": "grep -rn 'version_check' qutebrowser/utils/"}}
{"timestamp": "2026-09-12T14:03:12+00:00", "event": "tool_result", "iteration": 1, "tool_name": "bash", "result": "qutebrowser/utils/qtutils.py:91: ...", "is_error": false}
{"timestamp": "2026-09-12T14:12:40+00:00", "event": "run_end", "reason": "done", "num_iterations": 14}

Please stick to exactly these fields. During development, you may use stderr for print-based debugging as needed. However, make sure that the output follows this schema when using --log.

Part 1 — What the harness must contain

1.1 Tool definitions

This is a list you tell the model which tools are available to use: each has a name, a natural-language description, and a JSON Schema for its parameters.

A workable minimal tool set, and you may design a different one:

1.2 The agentic loop

The while-loop that makes it an agent rather than a chatbot:

messages = [system prompt, user message containing the problem statement]
loop:
    response = model(messages, tools)
    if response contains tool calls:
        execute each requested tool
        append the assistant message AND one tool-result message per call
        continue
    else:
        break   # model produced a final answer

It is important to fix failures you introduced and pre-existing failures unrelated to your change can be ignored . You can compare against the untouched code if unsure.

1.3 Conversation state management

The model is stateless: every request must resend the entire history. Your harness owns that history:

• Maintain the messages array: system prompt, initial user message, then alternating assistant turns (including their tool calls) and tool-result messages. Every tool call the model makes must be paired with exactly one result message referencing its call ID — an unpaired call is an API error.
• Write a system prompt that tells the model its role, the interface ("you are fixing an issue in the repo at ... ; make minimal changes; do not modify tests"), the reproduction protocol ("before editing, write reproduction checks under /tmp from the requirements and register each with repro_check;
• Context growth: tool outputs accumulate. At minimum, truncate large outputs before appending them.

That's the whole minimal harness: tool definitions (what the model can ask for), the loop + tool executors (your code doing what it asked), and conversation state (the memory tying the turns together).

Part 2 — Running in the container

Each task entry in agent_task_input.json specifies a prebuilt Docker image (docker_image) containing the target repository. The repository is checked out at the designated base_commit in /app, with all required dependencies preinstalled. To run your agent, first launch the corresponding container, then execute the agent inside the running container using /app as the working repository.

2.1 Pull the image (once per task)

The image names for the other tasks are in the docker_image field of their evaluation_scripts/agent_task_input.json entries. On an x86-64 host, the image runs natively:

# the first task's image (jefzda/sweap-images:qutebrowser.qutebrowser-…)
IMG=$(python -c "import json; print(list(json.load(open('evaluation_scripts/agent_task_input.json')).values())[0]['docker_image'])")
docker pull "$IMG"

2.2 Set up the container environment: docker_env.sh

The task images don't have the openai package installed. Your repo must include a docker_env.sh at the root that installs whatever your agent needs. It will run inside the container before the agent runs. The minimal version:

#!/bin/bash
# docker_env.sh — container setup, runs before madsLoop.py
pip install -q openai

If your agent needs anything else (e.g. pinning an openai version), add it in docker_env.sh to modify the container environment. Make sure your agent run on Python 3.10+ with openai and the standard library.

2.3 Run your agent inside it: run_task.sh

# the first task: its instance_id, and its problem statement (the -p payload) as a file
IID=$(python -c "import json; print(list(json.load(open('evaluation_scripts/agent_task_input.json')))[0])")
python -c "import json; print(list(json.load(open('evaluation_scripts/agent_task_input.json')).values())[0]['problem_statement'])" > problem.txt

docker run --rm \
  --entrypoint bash \
  -v "$PWD:/madsLoop" \
  -e CS2680_API_KEY="$CS2680_API_KEY" \
  "$IMG" \
  -c 'bash /madsLoop/docker_env.sh && \
      cd /tmp && \
      python /madsLoop/madsLoop.py -p "$(cat /madsLoop/problem.txt)" --log /app && \
      cp -r madsLoop_logs /madsLoop/ && \
      git -C /app diff' \
  > "model_patch_$IID.diff"

The script above identifies the ID and problem statement for each task, mounts the corresponding codebase in a container, passes your CS2680_API_KEY through to it, and sets up the environment. It then runs madsLoop. After the agent finishes modifying the repository, the script exports the madsLoop logs and generates a diff patch. Feel free to modify the run_task.sh, just make sure run_all.sh can call it.

2.4 Evaluate your patch

Evaluate it for real with the official SWE-bench Pro evaluation script (scaleapi/SWE-bench_Pro-os), the same tool we grade with. It applies your diff to a fresh checkout, restores the held-out test files, and runs the actual tests. Bundle your patches into a patches.json as in make_patches.py:

python - <<'EOF'
import json, pathlib
tasks = json.load(open("evaluation_scripts/agent_task_input.json"))
predictions = [{"instance_id": iid,
                "model_patch": pathlib.Path(f"model_patch_{iid}.diff").read_text(),
                "prefix": "madsLoop"}
               for iid in tasks]
json.dump(predictions, open("patches.json", "w"), indent=2)
EOF

and run the evaluation:

bash evaluation_scripts/evaluate.sh

It writes pro_eval/eval_results.json, mapping each instance_id to true/false (resolved or not), plus per-instance test logs under pro_eval/<instance_id>/ — that's your ground truth for whether a task is actually solved, and can be used to iterate on your agent . Patches are graded by applying them to a fresh checkout, restoring the held-out test files, and running hidden tests — any edits your agent made to existing test files are overwritten, so editing tests cannot help you.

2.5 Provided evaluation scripts (evaluation_scripts/)

We provide an evaluation_scripts/ folder as the evaluation entry point. Copy the folder into your repo root as-is. It contains:

Run them from your repo root (the directory containing madsLoop.py); if you keep the folder elsewhere, set MADSLOOP_REPO to your repo root. The task JSONs are always read from inside evaluation_scripts/ itself, and all outputs (patches, patches.json, madsLoop_logs/, pro_eval/) land in the repo root:

bash evaluation_scripts/run_task.sh 0     # one task, or run_all.sh for all four
python3 evaluation_scripts/make_patches.py
bash evaluation_scripts/evaluate.sh       # -> pro_eval/eval_results.json

What you may change in evaluation_scripts/. You are free to edit run_task.sh, agent_task_input.json and task_test.json to test things locally — point them at your own tasks, add debugging, adjust how the container is set up. run_all.sh, evaluate.sh and make_patches.py are what we keep: we run our own copies of them unchanged.

2.6 Build your own tasks (mytest/)

Beyond the tasks we give you, you build four tasks of your own. Do not take them from SWE-bench Pro, or from any other agent benchmark. The point is to see whether your harness holds up on a task nobody has tuned it for. Source each of the four from one of:

• an open-source repository — a real issue, together with the commit or pull request that closed it, or
• one of your own past projects — a bug you actually fixed, or a feature you actually added.

Building a task means producing three things.

1. One paragraph of description. This is everything your agent is told. It takes the place of problem_statement, requirements and interface in the tasks we gave you, and it should be a single paragraph, in the problem_statement field. Say what is wrong or missing, how it shows up, and what counts as done. Keep requirements and interface as keys with "" values, so the entry has the same shape as agent_task_input.json and your agent reads it unchanged.

2. Tests that show it is fixed. Record them in mytest/task_test.json, keyed by instance_id, with the same four fields as evaluation_scripts/task_test.json: before_repo_set_cmd, selected_test_files_to_run, fail_to_pass, pass_to_pass. fail_to_pass is at least one test that fails at base_commit and passes once the task is done. pass_to_pass is tests that pass in both states, so that a patch which breaks the repo cannot be scored as a fix.

3. A container. mytest/tasks/<instance_id>/Dockerfile builds an image with the repo checked out at base_commit and every dependency already installed, so the tests run without network access. Tag it, and put that tag in the task's docker_image field — that is the field run_task.sh runs. You do not have to push it anywhere: when the pull fails, run_task.sh falls back to the image of that name in your local Docker daemon.

Alongside the Dockerfile, include mytest/tasks/<instance_id>/gold.diff: the real fix, the upstream commit or whatever you wrote at the time. It is there so the tests can be trusted, meaning applying it must turn every fail_to_pass test green.

Split the four tasks into two files:

• mytest/solved_after_change.json — two tasks your agent that solves the given tasks could not solve at first, and can solve after you changed something in the agent. The change must be in the submitted code. The write-up and the video explain it.
• mytest/unsolved.json — two tasks your agent still cannot solve when you submit. Their gold.diff must still make their tests pass.

Running them. The workflow is the one from 2.1–2.4, with one substitution: the official evaluation in evaluate.sh only knows SWE-bench Pro instances, so it cannot score your images. Write a mytest/verify.sh to do that job instead.

IID=...                                                # one of your four instance_ids
IMG=...                                                # that task's docker_image tag

docker build -t "$IMG" "mytest/tasks/$IID/"            # 1. build the image
TASKS_JSON=mytest/solved_after_change.json \
  bash evaluation_scripts/run_task.sh 0                # 2. agent -> model_patch_$IID.diff
bash mytest/verify.sh "$IID" "model_patch_$IID.diff"   # 3. apply, run tests, report

verify.sh takes an instance_id and a diff, starts a fresh container from that task's docker_image, resets the checkout with before_repo_set_cmd, applies the diff, runs selected_test_files_to_run, and prints RESOLVED only if every fail_to_pass test passes and no pass_to_pass test regresses, the same rule the official evaluation applies. Run it with no diff to get the base_commit baseline.

Save its output under mytest/results/ for each of the four tasks: <instance_id>.before.log (at base_commit, showing fail_to_pass failing) and <instance_id>.after.log (with your agent's patch for the two in solved_after_change.json; with gold.diff for the two in unsolved.json).

Minimal handin under mytest/:

Deliverables

The submission has three parts: the code (a GitHub repo), a write-up, and a video. We will use the same private GitHub repository as in Assignment 1 with code and the writeup under assignment2/. The video should be uploaded to Canvas.

Code

We grade by cloning your repo and invoking your agent automatically, so the layout below is a hard requirement — if madsLoop.py is not at the repo root, our runner cannot call it and the submission scores zero.

Required repository layout:

<assignment2>
├── madsLoop.py                             # the entry point, at the ROOT
├── docker_env.sh                           # container setup (see 2.2), at the ROOT
├── writeup.pdf                             # write up pdf
├── src/                                    # the modules madsLoop.py imports, the harness design
│   ├── agentic_loop.py                     #   (any structure you like — this is one example)
│   ├── tools.py
│   └── prompts.py
├── developer_logs/                         # how you built madsLoop (see 0.2)
│   └── claude_code_session.jsonl           #   logs in jsonl
├── evaluation_scripts/                     # provided by us (see 2.5)
│   ├── agent_task_input.json               #   
│   ├── task_test.json                      #  
│   ├── run_task.sh                         #  
│   ├── run_all.sh                          #   entry point for our grading evaluation, do not change it
│   ├── make_patches.py                     #   
│   └── evaluate.sh                         #   
├── mytest/                                 #  four tasks you build yourself (see 2.6)
│   ├── solved_after_change.json            #   2 tasks: unsolved at first, solved after a change to your agent
│   ├── unsolved.json                       #   2 tasks: still unsolved at the end
│   ├── task_test.json                      #  
│   ├── tasks/<instance_id>/Dockerfile      #   one per task: builds the image named in docker_image
│   ├── tasks/<instance_id>/gold.diff       #   one per task: the real fix, never shown to your agent
│   ├── verify.sh                           #   applies a patch in the container and runs the tests
│   └── results/                            #   <instance_id>.before.log and .after.log per task
└── README.md                               # optional: design notes, tool-set rationale

We evaluate by running run_all.sh, which invokes run_task.sh for each task. Your run_task.sh is expected to invoke your agent with --log, and to produce a patch for each task; run_all.sh then bundles those patches and evaluates them.

Write-up

A one-page (hard limit) document in PDF in the repo with three sections:

1. Which tasks in agent_task_input.json you think (not the agent thinks) are unsolvable, and why. Name the instance_ids and give the evidence from your runs: what the agent tried, where it got stuck, and what about the task (not your agent) makes it unsolvable. You can change the instructions in Parts 0 and 1 in any way you think is appropriate. Failures caused by those instructions should not be considered unsolvable. Instead, identify problems with the tasks themselves that make them impossible to solve.
2. What you did differently from the handout. Use bullet points, one per difference: where your harness departs from the design described in Parts 0–1 and why.
3. How you made the two solved_after_change.json tasks pass. For each of the two: what failed at first, what you changed in the agent, and how that change turns the failure into a pass.

The document must be entirely your own work. No AI-generated text.

Video

Record a 3-minute video. First walk through your code: the entry point, the tools, the agentic loop, and how conversation state is kept. Then show which part of the code you changed to turn the two solved_after_change.json tasks from unsolved to solved, and why that change made the difference. It should be uploaded to Canvas.

Grading

Now let's get started!

Have fun with your agent! The starter code can be found here, under the folder assignment2.