CS2680 Modern AI Systems: Agents and System Optimizations
HPC: CPU and GPU jobs on the course cluster

The course cluster

Harvard FAS ATG runs a Slurm cluster for this course, with the GPU nodes Assignments 4 and 5 use. Students enrolled on Canvas already have an account. There are two ways in:

  • The Open OnDemand portal, from anywhere. Its Clusters menu opens a terminal on the login node in a browser tab, and its file browser moves single files. ATG's OnDemand documentation covers the portal.
  • SSH, from the Harvard network or the Harvard University VPN only, because the login host's address is private to Harvard. SSH also carries rsync, VS Code and port tunnels.

If the portal does not let you in, email the course staff. For Assignment 4: do the first-time setup, start a session on a GPU node as in serving a model, and reach the server with a tunnel, or through your laptop if your agent runs on CloudLab or AWS.


How the cluster works

A terminal on the cluster first lands on the login node, which the whole course shares. The hardware you compute on is the compute nodes, and Slurm decides who uses them. You ask Slurm for a job (for example, 4 cores and 16 GB for 2 hours); when the hardware is free, it starts your job on a compute node, and the job ends when its time runs out or you exit it. hostname tells you which machine a prompt belongs to.

Your home directory is shared: the login node and every compute node see the same files, so anything you install once, such as claude, is there in every job.

Do not run an agent or a model on the login node. Every process there is killed after 15 minutes of CPU time, with the message CPU time limit exceeded. Use the login node to edit files, install tools, move data and start jobs.

First-time setup

  1. Open a terminal on the login node: in the OnDemand portal, the shell under the Clusters menu. Run whoami and note your cluster username.
  2. Register an SSH key, to connect from your laptop's terminal (Terminal on macOS, PowerShell on Windows). On your laptop, create a key if you do not have one, and print its public half:
    ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 # skip if the file exists; a passphrase is optional cat ~/.ssh/id_ed25519.pub
    Copy the one line it prints. (The file without .pub is your private key, which never leaves your laptop.) In the OnDemand terminal, add the line to ~/.ssh/authorized_keys, replacing the example with your own line:
    mkdir -p ~/.ssh && chmod 700 ~/.ssh echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... you@laptop' >> ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys
    In the browser terminal, paste with Ctrl+Shift+V (Cmd+V on a Mac), and keep the key on one line.
  3. Add the SSH config. On your laptop, add this to ~/.ssh/config (C:\Users\you\.ssh\config on Windows), with your username:
    Host cs2680 Hostname academ-acade-iL73aitWT6xF-c83014867702e61e.elb.us-east-1.amazonaws.com User YOUR_USERNAME IdentityFile ~/.ssh/id_ed25519 IdentitiesOnly yes ServerAliveInterval 30
    Then, from the Harvard network or VPN, run ssh cs2680. The first time, answer yes to the host-key question. A prompt on the login node means the key works; if not, see troubleshooting.
  4. Install Claude Code, if you will run it on the cluster:
    curl -fsSL https://claude.ai/install.sh | bash echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc claude # sign in, then /exit
    Over SSH or in the browser terminal, the login gives you a code to paste back; the Claude Code page walks through it. When youarchive your session files, include the cluster's ~/.claude/projects/.

Interactive jobs

From a terminal on the login node:

srun -p general -c 4 --mem=16G -t 2:00:00 --pty bash

When the prompt comes back, you are on a compute node: work there (for example, run claude), and exit when you are done. Slurm powers down a node after 10 idle minutes, so a job can wait about 4 minutes while one boots; squeue -u $USER shows it as CF meanwhile. If your connection drops, the job ends; start a new one, and claude --continue picks up the conversation. To keep work running while you are away, use a session that survives a disconnect.

Flag Meaning
-p general The partition, the pool of nodes: general for CPU work, or a GPU partition.
-c 4 CPU cores. 4 is plenty for an agent; ask for 8 on a GPU node.
--mem=16G Memory Slurm sets aside for the job. Slurm does not enforce it here, so stay below it: going over can get your process, or a neighbor's, killed when the node runs out.
-t 2:00:00 Time limit, HH:MM:SS. The job ends when it is reached. Always set it.
--gres=gpu:<type>:1 One GPU, on a GPU partition. The table below gives each partition's type.
--pty bash Gives you a shell in the job.

GPUs (Assignments 4 and 5, and the project)

Students may use these partitions. Each GPU node has one GPU.

Partition GPU Node Longest job Request
gpu-g7-reserved RTX PRO 4500 Blackwell, 32 GB 8 cores, about 30 GB 14 days srun -p gpu-g7-reserved --gres=gpu:rtxpro4500:1 -c 8 --mem=28G -t 4:00:00 --pty bash
general none 48 cores, about 90 GB 3 days srun -p general -c 4 --mem=16G -t 2:00:00 --pty bash

Assignment 4 uses the RTX PRO 4500. Other GPU partitions on the cluster need a TF's approval before you use them for course work.

In a GPU job, echo $CUDA_VISIBLE_DEVICES should print a device number. If it prints nothing, the job has no GPU: you left out --gres. (nvidia-smi lists the card either way, so it does not tell you.)

Serving a model on a GPU node

A model server takes minutes to start and runs until you stop it, so run it in a session that survives a disconnect, in the background, with its output in a log file. From the login node:

sbatch -p gpu-g7-reserved --gres=gpu:rtxpro4500:1 -c 8 --mem=28G -t 4:00:00 -J work -o /dev/null --wrap 'screen -dmS work; while screen -ls work | grep -q "\.work"; do sleep 30; done' squeue -u $USER --name=work # wait for state R, and note the JOBID and the NODELIST srun --jobid=<JOBID> --overlap --pty screen -r work

Then, in the screen session on the GPU node, start the server (your assignment gives the command) with an API key, for example:

(umask 077; [ -f ~/serve.key ] || openssl rand -hex 16 > ~/serve.key) # a password for your server, once python -m sglang.launch_server ... --host 0.0.0.0 --port 30000 --api-key $(cat ~/serve.key) > server.log 2>&1 & tail -f server.log # Ctrl-c stops watching; the server keeps running
  • Always set an API key. --host 0.0.0.0 lets everyone on the cluster reach your server, and the key keeps them off your GPU.
  • Keep model weights out of your home directory. Use the copy your assignment provides, or keep weights in /scratch/$USER (export HF_HOME=/scratch/$USER/hf) and copy them to the node's local disk (/local_scratch/$USER) before loading, which is much faster than loading them from /scratch.
  • Use the environment your assignment provides. Otherwise, install the engine in a virtualenv in your home directory, which every node sees.

Reaching the server from another machine

Compute nodes cannot be reached from outside the cluster, so you forward a local port through the login node with an SSH tunnel. On the machine that needs the server (it must be on the Harvard network or VPN, and have the cs2680 config; for a CloudLab or AWS machine, see below), with the node name from squeue, run this and leave the terminal open:

ssh -N -L 30000:<node-name>:30000 cs2680

In a second terminal, curl -s -H "Authorization: Bearer $(ssh cs2680 cat serve.key)" localhost:30000/v1/models should list the model. The tunnel stops working when the job ends, so open a new one for each job.

Reaching the cluster from CloudLab or AWS

The login node answers only on the Harvard network and the Harvard University VPN, so a CloudLab or AWS machine cannot run ssh cs2680 or open the tunnel above by itself. If your agent runs on such a machine, as it may in Assignment 4, it can reach your server in one of two ways:

  1. The machine joins the VPN. Install the VPN client that HUIT's VPN page names (Cisco AnyConnect) on that machine and connect it as HUIT describes. Then create an SSH key there, register it as in the key step, and add the cs2680 config. The commands on this page and on the assignment page then work there as written. However, if the VPN client sends all of the machine's traffic through Harvard, your own SSH connection to the machine can stop working when the VPN connects; the second way avoids this.
  2. A machine on the VPN keeps the tunnel, usually your laptop. The laptop forwards a port to the GPU node through the login node, as above, and a second SSH connection from the laptop carries that port to the CloudLab or AWS machine. Nothing is installed on that machine, but every model call passes through the laptop, so the laptop must stay awake and on the VPN for the whole run.

The second way takes three steps. First, once on the CloudLab or AWS machine, let its SSH server bind a forwarded port to the Docker bridge address. By default it binds every port a client forwards to it to 127.0.0.1, which containers cannot reach:

echo 'GatewayPorts clientspecified' | sudo tee /etc/ssh/sshd_config.d/cs2680-tunnel.conf sudo systemctl reload ssh || sudo systemctl reload sshd sudo sshd -T | grep -i gatewayports # should print: gatewayports clientspecified docker network inspect bridge -f '{{(index .IPAM.Config 0).Gateway}}' # the bridge address, usually 172.17.0.1; use yours in place of it below

clientspecified binds a forwarded port only to the address the client names. (yes would bind it to every address, which puts your server's port on the public internet.)

Second, on the laptop, with the node name from squeue and your SSH config's name for the CloudLab or AWS machine in place of agentbox, run each line in its own terminal and leave both open:

ssh -N -o ExitOnForwardFailure=yes -L 30000:<node-name>:30000 cs2680 ssh -N -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -R 172.17.0.1:30000:127.0.0.1:30000 agentbox

The first line is the tunnel above. The second makes port 30000 at the machine's Docker bridge address lead to port 30000 on the laptop, and so to your server. When your server's job ends, open the first one again to the next job's node; the second can stay open.

Third, on the CloudLab or AWS machine, your server is now at http://172.17.0.1:30000, the address the Assignment 4 page points the harness at, so skip that page's own ssh -L line. That machine cannot read your server's key with ssh cs2680 either, so copy the key there from the laptop (scp -3 copies between two remote machines through the laptop), and use the file in place of that page's ssh cs2680 cat a4/serve.key:

scp -3 cs2680:a4/serve.key agentbox:a4-serve.key # on the laptop export CS2680_API_KEY=$(cat ~/a4-serve.key) # on the CloudLab or AWS machine curl -s -H "Authorization: Bearer $CS2680_API_KEY" 172.17.0.1:30000/v1/models # should list the model

The same holds for every other copy between the two machines. The starter's README.md copies your submission's engine/ files to ~/a4/engine on the cluster, and back before you zip the submission; run those copies on the laptop too, with the path of your course repository on the CloudLab or AWS machine in place of <repo>:

scp -3 agentbox:<repo>/assignment4/starter_code/engine/engine.json agentbox:<repo>/assignment4/starter_code/engine/launch.sh cs2680:a4/engine/ # to the cluster scp -3 -r 'cs2680:a4/engine/*' agentbox:<repo>/assignment4/starter_code/engine/ # back to the repository

Keeping a session alive when you disconnect

A shell from srun --pty ends, with its job, when your connection drops. To keep work running, start a job that holds a screen session, and attach to it. (The compute nodes have screen but not tmux.) From the login node:

sbatch -p general -c 4 --mem=16G -t 4:00:00 -J work -o /dev/null --wrap 'screen -dmS work; while screen -ls work | grep -q "\.work"; do sleep 30; done' squeue -u $USER --name=work # wait for state R, and note the JOBID srun --jobid=<JOBID> --overlap --pty screen -r work

For a GPU, use the partition and --gres from the GPU table in the first line. In the session:

  • Ctrl-a d detaches. Everything you started keeps running, and the third line above attaches again, from any login shell.
  • Ctrl-a c opens another window in the session, for example to watch a log; Ctrl-a n switches between windows.
  • exit in the last window ends the session, and with it the job. The job also ends at its time limit, whatever is running.

Optional: SSH straight to a compute node

To land in a CPU job directly from your laptop, for example to run VS Code's Remote-SSH there instead of on the login node, add this to ~/.ssh/config below the cs2680 entry:

Host cs2680-cpu Hostname academ-acade-iL73aitWT6xF-c83014867702e61e.elb.us-east-1.amazonaws.com User YOUR_USERNAME IdentityFile ~/.ssh/id_ed25519 IdentitiesOnly yes HostKeyAlias cs2680-cpu ProxyCommand ssh -q cs2680 exec /shared/courseSharedFolders/176914outer/176914/cpu-tunnel.sh cpu

ssh cs2680-cpu then starts a job on general (4 cores, 16 GB, 4 hours) and connects to it. For VS Code, set "remote.SSH.connectTimeout": 600 in its settings, since a node can take minutes to boot. Every connection is a new job, so check squeue -u $USER and scancel the ones you no longer use.


Unattended runs

For work you do not need to watch, write a batch script, for example run.sh:

#!/bin/bash #SBATCH -p general #SBATCH -c 4 #SBATCH --mem=16G #SBATCH -t 2:00:00 #SBATCH -J analysis #SBATCH -o logs/%x-%j.out source ~/venv/bin/activate python3 analyze.py --in /scratch/$USER/data --out /scratch/$USER/results

Then mkdir -p logs and sbatch run.sh, from the same directory. sbatch prints the job id and returns; the output goes to logs/analysis-<jobid>.out. Slurm does not create logs/, and a job whose log it cannot open fails at once. Anything that needs Docker, such as the Assignment 3 and 4 harness, cannot run on the cluster.


Files and storage

Where What to keep there
~ (your home directory) Code, configs and installed tools. Every node sees it.
/scratch/$USER Large files: model weights, datasets, results. Private to you, and every node sees it.
~/176914 The course folder, with what the course provides. Read-only.
/local_scratch/$USER on a compute node Fast space for a running job, on the node's own NVMe disk (about 400 GB). Nodes started after the cluster change on October 29, 2026 have it. Use it only if ls -ld /local_scratch/$USER shows you as the owner. Then run chmod 700 /local_scratch/$USER, so that only you can read it. If the folder is missing or belongs to someone else, use /var/tmp and tell the course staff. Files there are not deleted when the job ends. They stay until the node shuts down and take space on a disk that other people's jobs on that node also use, so delete them before you end the job.
/tmp and /var/tmp on a compute node Scratch space for a running job, on the node's small root disk (about 66 GB free), which every job on the node shares. Use /local_scratch/$USER for large files when the node has it. Files there are not deleted when the job ends. They stay until the node shuts down and take disk space from the next person on that node, so delete them before you end the job.

Move code with git. Move other files from your laptop with rsync (create the target directory first), or with scp on Windows, which has no rsync:

ssh cs2680 mkdir -p /scratch/YOUR_USERNAME/data rsync -av ./data/ cs2680:/scratch/YOUR_USERNAME/data/ # upload rsync -av cs2680:/scratch/YOUR_USERNAME/results/ ./results/ # download

Sharing the cluster

The whole class shares these nodes, the course pays for every GPU-hour, and there is no per-student limit. We may cancel a job that holds far more than its share, especially before a deadline.

  • Ask for what you need, and give every job a time limit with -t.
  • End a job when you stop working (exit, or scancel <jobid>). An idle job still holds its node. Delete your files in /local_scratch/$USER and /var/tmp first. For Assignment 4, run bash $A4/bin/cleanup.sh.
  • Keep the login node for light work.

Command reference

Command What it does
squeue -u $USER Your jobs, with id, state (PD waiting, CF node booting, R running), time used, node, and why a job waits.
sinfo -p <partition> The partition's nodes: idle free, idle~ powered down, alloc or mix in use, # booting, down or drain unavailable.
scancel <jobid> Ends one job.
sacct -j <jobid> --format=JobID,State,Elapsed,ExitCode What happened to a finished job; TIMEOUT means it reached its time limit.
hostname Whether you are on the login node or a compute node.

Troubleshooting

Symptom Cause and fix
ssh cs2680 hangs or times out. You are not on the Harvard network. Connect to the Harvard University VPN, or use the OnDemand portal. From a CloudLab or AWS machine, see reaching the cluster from CloudLab or AWS.
Permission denied (publickey), or SSH asks for a password. The cluster does not have your public key, or has it split across lines. In the OnDemand terminal, compare cat ~/.ssh/authorized_keys with your laptop's ~/.ssh/id_ed25519.pub, and redo the key step, including both chmod lines.
The job waits for minutes, or never starts. A few minutes (state CF) is a node booting. Longer, squeue -u $USER gives the reason: usually you asked for more than a node has. Compare your flags with the partition table.
Invalid partition, or not permitted to use this partition. A typo, or a partition students cannot use. Use one from the partition table.
CPU time limit exceeded, or a program killed on the login node. The login node's 15-minute CPU limit. Run it in a job.
A job's program was killed without a message. sacct -j <jobid>: TIMEOUT is the time limit. Otherwise the node probably ran out of memory; ask for more --mem and use less.
claude: command not found. The export PATH line from first-time setup is missing from ~/.bashrc.
VS Code Remote-SSH keeps disconnecting. It is running on the login node and hits the CPU limit. Connect to cs2680-cpu instead.
ssh cs2680-cpu hangs. Check that ssh cs2680 works, then run ssh cs2680 /shared/courseSharedFolders/176914outer/176914/cpu-tunnel.sh cpu to see the errors the ProxyCommand hides. A few minutes is normal while a node boots.
Out of disk while downloading a model. Your home directory is full, usually with ~/.cache/huggingface. Keep weights in /scratch/$USER.
Something else. Bring the command, the job id and the error message to office hours or the course forum.