In the final project, you choose a problem, build a system or an experiment to address it, and support one claim with measurements.
In the assignments, the course specifies the problem and the evaluation. In the project, you choose both. You can take one of two tracks: the product track, in which you build a tool that solves a practical problem, or the research track, in which you make an original claim about agents or the systems that run them. Both tracks have the same deadlines, deliverables, and rubric. The research track is held to a higher standard of novelty and evidence.
The project is individual and is worth 20% of the course grade. Each student submits one proposal, presents one poster and demo, and publishes one blog post and repository. You can start before the proposal deadline. Email the instructor if you want feedback on an idea or its scope, need a compute allocation, or want someone to read a draft.
| Due | Deliverable | Weight | How to submit |
|---|---|---|---|
| Wed Oct 28, 11:59pm | Proposal | 2% | One PDF on Canvas |
| Mon Dec 7, 9:00am–1:00pm | Poster and demo, including peer evaluation | 10% | In person, SEC 1.402 |
| Tue Dec 8, 11:59pm | Blog post and repository | 8% | Pull request to the course blog repository |
Choose the track that matches the claim you want to make, and name it in the proposal. You can switch later by emailing the instructor.
Build an AI system that solves a real problem better than the strongest available alternative. The gap can come from missing context, poor reliability, high cost, slow response, privacy constraints, or a workflow that existing tools do not support.
Suitable directions include:
The research track requires an original, falsifiable claim about agents or AI systems. The work must establish a result that the closest published work does not provide, explain why the result occurs, and define the conditions under which it holds. A polished implementation, a reproduction, a benchmark, or a set of measurements does not meet this bar by itself.
You can work on either side of the course:
On either side, the contribution can take one of four forms:
If the main contribution is a useful application, choose the product track and evaluate it rigorously. Choose the research track if you can state the new claim, name the closest published claim, and design a controlled experiment that distinguishes the two.
| Due | Wed Oct 28, 11:59pm |
|---|---|
| Weight | 2% of the course grade |
| Submit | One PDF on Canvas |
| Length | No page limit |
The proposal lets the course staff check that the project is specific, can be completed in five weeks, and will be evaluated against the right baselines before most of the work begins. Include the nine sections below, in this order. Each section lists what it must contain for both tracks and, where they differ, for each track.
The proposal is graded on completeness and specificity: whether all nine sections are present, whether the gap is supported by tests or reproductions you report, and whether the baselines, workload, and metrics are specific enough to run. The track and plan are not binding. If the project changes direction after the proposal, email the instructor with the new track or plan.
Mon Dec 7, 9:00am–1:00pm, SEC 1.402 · 10%
Present the project as a poster at the poster session after classes end. A visitor should be able to find the problem, the gap, the system or experiment, and the main result within a few minutes. Include at least one figure that is legible from two meters away. If a live demonstration suits the project, prepare a version that runs reliably in about two minutes.
The poster session includes peer evaluation, and peer scores count toward the project grade. Each student reviews several other projects, and the reviews are part of the deliverable.
Tue Dec 8, 11:59pm · 8%
Publish the write-up on the course blog by opening a pull request to the course blog repository, following the publishing guide. The deadline is the time the pull request is opened. It does not need to be merged by then.
Write for a general technical audience, using the assignment write-ups as a model. There is no length requirement. The post must cover eight items:
The repository must let another person reproduce the main result. Include an installation and execution guide, the raw measurements behind each figure, the scripts that generate the figures, and the hardware, software versions, and workload used for every reported number. Label all axes and units, and keep figures legible in print and in grayscale.
The proposal is graded as described in Proposal, and the poster session is partly scored by peers. The blog post and repository are graded on the four dimensions below, weighted equally. Both tracks use the same dimensions. Research projects are held to a higher standard of novelty, experimental control, reproducibility, and engagement with published work because they claim to add to the published record.
| Dimension | What strong work shows | Warning sign |
|---|---|---|
| Problem and motivation | A clear problem or question, a thorough review of what already exists, and a testable gap | An existing tool already solves the problem, or a paper already reports the answer |
| Technical execution | A substantial, technically sound implementation or measurement | Code that works only on one machine, one input, or one run |
| Evaluation | Appropriate baselines and metrics, controlled comparisons, and conclusions supported by the experiments | Single-run numbers, a weak baseline, or a comparison between your run and an unreproduced number from a paper |
| Reporting | Clear claims, reproducible evidence, and an accurate account of limitations and negative results | A figure or claim that the underlying measurements do not support |
A project whose main idea does not work can still earn nearly full credit if the blog post explains what you expected, what you measured, and why the two differ. A claim that your measurements do not support loses credit whether the result is favorable or unfavorable.
Before building, find out what already exists and state its limitation in one testable sentence. Do this before the proposal, so that you can change the project early if an existing tool or paper already covers it.
Run the closest commercial tools, open-source systems, and research prototypes on your own task. Also give the same input directly to a frontier model; this counts as a baseline even if it seems too simple. Record the specific failure you observe. For example, “It cannot read my cluster's job queue” is a testable gap, whereas “Nothing like this exists” is not.
Read the closest published work on the agent side or the systems side and identify the claim it leaves unsettled. Research gaps commonly come from three sources. First, nobody has measured the behavior on your workload, model, or hardware. Second, a published result depends on conditions that should change the answer under yours. Third, two published results disagree. A plan to try an idea and see whether it works is not yet a research question; state the result you expect and why.
Reproduce the closest result first. A number reported in a paper was measured with a different model, prompt, task set, GPU, software version, or workload. Comparing your run against that number confounds your change with those differences, so reproduce the closest result in your own setup before you try to improve on it. If the published artifact cannot be reproduced, document why and build the closest comparison you can justify.
| Track | Baseline 1 | Baseline 2 |
|---|---|---|
| Product | The strongest existing system, run on your tasks | The current by-hand workflow, timed before your tool changes it |
| Research | The unmodified agent or system | The strongest published method or result, reproduced in your setup |
If you introduce a mechanism, isolate it with an ablation or another controlled comparison. Report the configurations that perform worse as well as those that perform better, and include the strongest comparison even if your system loses to it.
A research result must hold beyond the run that produced it. Before the main experiment, state the hypothesis and fix the protocol. Then run repeated trials, report variation or uncertainty, isolate the proposed cause, test at least one boundary condition, and keep the artifacts needed to reproduce the result.
For agent-side research, fix the model version, prompts, tools, task set, and success criteria. Use held-out tasks if you tuned the method on examples, and separate capability from token cost, latency, and run-to-run variation. For systems-side research, fix the hardware and software stack, describe the workload, measure under controlled load, and report both central and tail behavior where applicable.
A single benchmark score, one favorable prompt example, or one throughput number does not meet this bar.
Agent projects may need success rate, latency, cost, token usage, number of steps, resource use, and human effort. Serving projects may need throughput, time to first token, per-token latency, tail latency, memory use, cache hit ratio, and prefill work avoided. A latency result must state the request rate at which it was measured.
Measure one level deeper. Break the top-line result into the components that explain it. For example, “The agent took 60 seconds” reports what happened, while “Tool calls took 40 seconds, including one 30-second call that could run in parallel” shows what to change. Ousterhout's Always Measure One Level Deeper explains this practice.
Fix the workload, task set, and metrics before the main experiment. For the product track, time the by-hand workflow before your tool changes it. For the research track, do not choose the workload after seeing which one gives the strongest result.
Run the complete path once and take the first measurement before building more infrastructure. If the project needs a GPU, include queue time and reruns in the plan.
The project overlaps with Assignment 4 (due Tue Nov 10) and Assignment 5 (due Tue Nov 24). The poster session is Mon Dec 7 and the blog post is due Tue Dec 8, so start the poster and the blog post before the final week.
The computing setup page has access instructions. Choose the simplest resource that can run the experiment. Most projects fit on a laptop, model API access, or a small number of GPUs. Contact the instructor early if the project needs hardware that is not listed here or a large amount of compute.
| Resource | Use it when | Plan for |
|---|---|---|
| Course model API | You need a hosted frontier model but do not need to control the serving stack | Request cost and data leaving your machine |
| Course HPC | You need CPU or RTX PRO 4500 Blackwell GPU resources; this is the default for most serving projects | A three-day CPU job limit, a 14-day GPU job limit, and a shared GPU pool |
| AWS | You need cloud resources covered by an announced course allocation | Do not depend on credits until the allocation is confirmed |
| FASRC | You already have or can obtain access to research CPU nodes or GPUs such as H100 and H200 | Access approval and queue time |
| CloudLab | You need bare metal, kernel or network access, or isolation from a shared cluster | Environment setup and experiment reproducibility |
Start with the least expensive model and the smallest complete experiment. Set a spending cap before each API run, stop cloud jobs when they finish, and run a small sweep before the full one. A late GPU allocation is not grounds for an extension, but the course staff can help you debug access and adjust the scope.