Harvard FAS ATG runs a Slurm cluster for this course, with the GPU nodes Assignments 4 and 5 use. Students enrolled on Canvas already have an account. There are two ways in:
rsync, VS Code and port tunnels.If the portal does not let you in, email the course staff. For Assignment 4: do the first-time setup, start a session on a GPU node as in serving a model, and reach the server with a tunnel, or through your laptop if your agent runs on CloudLab or AWS.
A terminal on the cluster first lands on the login node, which the whole course shares. The hardware you compute on is the compute nodes, and Slurm decides who uses them. You ask Slurm for a job (for example, 4 cores and 16 GB for 2 hours); when the hardware is free, it starts your job on a compute node, and the job ends when its time runs out or you exit it. hostname tells you which machine a prompt belongs to.
Your home directory is shared: the login node and every compute node see the same files, so anything you install once, such as claude, is there in every job.
CPU time limit exceeded. Use the login node to edit files, install tools, move data and start jobs.
whoami and note your cluster username..pub is your private key, which never leaves your laptop.) In the OnDemand terminal, add the line to ~/.ssh/authorized_keys, replacing the example with your own line:
Ctrl+Shift+V (Cmd+V on a Mac), and keep the key on one line.
~/.ssh/config (C:\Users\you\.ssh\config on Windows), with your username:
ssh cs2680. The first time, answer yes to the host-key question. A prompt on the login node means the key works; if not, see troubleshooting.
~/.claude/projects/.
From a terminal on the login node:
When the prompt comes back, you are on a compute node: work there (for example, run claude), and exit when you are done. Slurm powers down a node after 10 idle minutes, so a job can wait about 4 minutes while one boots; squeue -u $USER shows it as CF meanwhile. If your connection drops, the job ends; start a new one, and claude --continue picks up the conversation. To keep work running while you are away, use a session that survives a disconnect.
| Flag | Meaning |
|---|---|
-p general |
The partition, the pool of nodes: general for CPU work, or a GPU partition. |
-c 4 |
CPU cores. 4 is plenty for an agent; ask for 8 on a GPU node. |
--mem=16G |
Memory Slurm sets aside for the job. Slurm does not enforce it here, so stay below it: going over can get your process, or a neighbor's, killed when the node runs out. |
-t 2:00:00 |
Time limit, HH:MM:SS. The job ends when it is reached. Always set it. |
--gres=gpu:<type>:1 |
One GPU, on a GPU partition. The table below gives each partition's type. |
--pty bash |
Gives you a shell in the job. |
Students may use these partitions. Each GPU node has one GPU.
| Partition | GPU | Node | Longest job | Request |
|---|---|---|---|---|
gpu-g7-reserved |
RTX PRO 4500 Blackwell, 32 GB | 8 cores, about 30 GB | 14 days | srun -p gpu-g7-reserved --gres=gpu:rtxpro4500:1 -c 8 --mem=28G -t 4:00:00 --pty bash |
general |
none | 48 cores, about 90 GB | 3 days | srun -p general -c 4 --mem=16G -t 2:00:00 --pty bash |
Assignment 4 uses the RTX PRO 4500. Other GPU partitions on the cluster need a TF's approval before you use them for course work.
In a GPU job, echo $CUDA_VISIBLE_DEVICES should print a device number. If it prints nothing, the job has no GPU: you left out --gres. (nvidia-smi lists the card either way, so it does not tell you.)
A model server takes minutes to start and runs until you stop it, so run it in a session that survives a disconnect, in the background, with its output in a log file. From the login node:
Then, in the screen session on the GPU node, start the server (your assignment gives the command) with an API key, for example:
--host 0.0.0.0 lets everyone on the cluster reach your server, and the key keeps them off your GPU./scratch/$USER (export HF_HOME=/scratch/$USER/hf) and copy them to the node's local disk (/local_scratch/$USER) before loading, which is much faster than loading them from /scratch.Compute nodes cannot be reached from outside the cluster, so you forward a local port through the login node with an SSH tunnel. On the machine that needs the server (it must be on the Harvard network or VPN, and have the cs2680 config; for a CloudLab or AWS machine, see below), with the node name from squeue, run this and leave the terminal open:
In a second terminal, curl -s -H "Authorization: Bearer $(ssh cs2680 cat serve.key)" localhost:30000/v1/models should list the model. The tunnel stops working when the job ends, so open a new one for each job.
The login node answers only on the Harvard network and the Harvard University VPN, so a CloudLab or AWS machine cannot run ssh cs2680 or open the tunnel above by itself. If your agent runs on such a machine, as it may in Assignment 4, it can reach your server in one of two ways:
cs2680 config. The commands on this page and on the assignment page then work there as written. However, if the VPN client sends all of the machine's traffic through Harvard, your own SSH connection to the machine can stop working when the VPN connects; the second way avoids this.The second way takes three steps. First, once on the CloudLab or AWS machine, let its SSH server bind a forwarded port to the Docker bridge address. By default it binds every port a client forwards to it to 127.0.0.1, which containers cannot reach:
clientspecified binds a forwarded port only to the address the client names. (yes would bind it to every address, which puts your server's port on the public internet.)
Second, on the laptop, with the node name from squeue and your SSH config's name for the CloudLab or AWS machine in place of agentbox, run each line in its own terminal and leave both open:
The first line is the tunnel above. The second makes port 30000 at the machine's Docker bridge address lead to port 30000 on the laptop, and so to your server. When your server's job ends, open the first one again to the next job's node; the second can stay open.
Third, on the CloudLab or AWS machine, your server is now at http://172.17.0.1:30000, the address the Assignment 4 page points the harness at, so skip that page's own ssh -L line. That machine cannot read your server's key with ssh cs2680 either, so copy the key there from the laptop (scp -3 copies between two remote machines through the laptop), and use the file in place of that page's ssh cs2680 cat a4/serve.key:
The same holds for every other copy between the two machines. The starter's README.md copies your submission's engine/ files to ~/a4/engine on the cluster, and back before you zip the submission; run those copies on the laptop too, with the path of your course repository on the CloudLab or AWS machine in place of <repo>:
A shell from srun --pty ends, with its job, when your connection drops. To keep work running, start a job that holds a screen session, and attach to it. (The compute nodes have screen but not tmux.) From the login node:
For a GPU, use the partition and --gres from the GPU table in the first line. In the session:
Ctrl-a d detaches. Everything you started keeps running, and the third line above attaches again, from any login shell.Ctrl-a c opens another window in the session, for example to watch a log; Ctrl-a n switches between windows.exit in the last window ends the session, and with it the job. The job also ends at its time limit, whatever is running.To land in a CPU job directly from your laptop, for example to run VS Code's Remote-SSH there instead of on the login node, add this to ~/.ssh/config below the cs2680 entry:
ssh cs2680-cpu then starts a job on general (4 cores, 16 GB, 4 hours) and connects to it. For VS Code, set "remote.SSH.connectTimeout": 600 in its settings, since a node can take minutes to boot. Every connection is a new job, so check squeue -u $USER and scancel the ones you no longer use.
For work you do not need to watch, write a batch script, for example run.sh:
Then mkdir -p logs and sbatch run.sh, from the same directory. sbatch prints the job id and returns; the output goes to logs/analysis-<jobid>.out. Slurm does not create logs/, and a job whose log it cannot open fails at once. Anything that needs Docker, such as the Assignment 3 and 4 harness, cannot run on the cluster.
| Where | What to keep there |
|---|---|
~ (your home directory) |
Code, configs and installed tools. Every node sees it. |
/scratch/$USER |
Large files: model weights, datasets, results. Private to you, and every node sees it. |
~/176914 |
The course folder, with what the course provides. Read-only. |
/local_scratch/$USER on a compute node |
Fast space for a running job, on the node's own NVMe disk (about 400 GB). Nodes started after the cluster change on October 29, 2026 have it. Use it only if ls -ld /local_scratch/$USER shows you as the owner. Then run chmod 700 /local_scratch/$USER, so that only you can read it. If the folder is missing or belongs to someone else, use /var/tmp and tell the course staff. Files there are not deleted when the job ends. They stay until the node shuts down and take space on a disk that other people's jobs on that node also use, so delete them before you end the job. |
/tmp and /var/tmp on a compute node |
Scratch space for a running job, on the node's small root disk (about 66 GB free), which every job on the node shares. Use /local_scratch/$USER for large files when the node has it. Files there are not deleted when the job ends. They stay until the node shuts down and take disk space from the next person on that node, so delete them before you end the job. |
Move code with git. Move other files from your laptop with rsync (create the target directory first), or with scp on Windows, which has no rsync:
The whole class shares these nodes, the course pays for every GPU-hour, and there is no per-student limit. We may cancel a job that holds far more than its share, especially before a deadline.
-t.exit, or scancel <jobid>). An idle job still holds its node. Delete your files in /local_scratch/$USER and /var/tmp first. For Assignment 4, run bash $A4/bin/cleanup.sh.| Command | What it does |
|---|---|
squeue -u $USER |
Your jobs, with id, state (PD waiting, CF node booting, R running), time used, node, and why a job waits. |
sinfo -p <partition> |
The partition's nodes: idle free, idle~ powered down, alloc or mix in use, # booting, down or drain unavailable. |
scancel <jobid> |
Ends one job. |
sacct -j <jobid> --format=JobID,State,Elapsed,ExitCode |
What happened to a finished job; TIMEOUT means it reached its time limit. |
hostname |
Whether you are on the login node or a compute node. |
| Symptom | Cause and fix |
|---|---|
ssh cs2680 hangs or times out. |
You are not on the Harvard network. Connect to the Harvard University VPN, or use the OnDemand portal. From a CloudLab or AWS machine, see reaching the cluster from CloudLab or AWS. |
Permission denied (publickey), or SSH asks for a password. |
The cluster does not have your public key, or has it split across lines. In the OnDemand terminal, compare cat ~/.ssh/authorized_keys with your laptop's ~/.ssh/id_ed25519.pub, and redo the key step, including both chmod lines. |
| The job waits for minutes, or never starts. | A few minutes (state CF) is a node booting. Longer, squeue -u $USER gives the reason: usually you asked for more than a node has. Compare your flags with the partition table. |
Invalid partition, or not permitted to use this partition. |
A typo, or a partition students cannot use. Use one from the partition table. |
CPU time limit exceeded, or a program killed on the login node. |
The login node's 15-minute CPU limit. Run it in a job. |
| A job's program was killed without a message. | sacct -j <jobid>: TIMEOUT is the time limit. Otherwise the node probably ran out of memory; ask for more --mem and use less. |
claude: command not found. |
The export PATH line from first-time setup is missing from ~/.bashrc. |
| VS Code Remote-SSH keeps disconnecting. | It is running on the login node and hits the CPU limit. Connect to cs2680-cpu instead. |
ssh cs2680-cpu hangs. |
Check that ssh cs2680 works, then run ssh cs2680 /shared/courseSharedFolders/176914outer/176914/cpu-tunnel.sh cpu to see the errors the ProxyCommand hides. A few minutes is normal while a node boots. |
| Out of disk while downloading a model. | Your home directory is full, usually with ~/.cache/huggingface. Keep weights in /scratch/$USER. |
| Something else. | Bring the command, the job id and the error message to office hours or the course forum. |