Background. OWASP's Top 10 for LLM applications describes risks at the level of a model response, such as prompt injection (LLM01) and excessive agency (LLM06). An agent plans across many steps, holds credentials, keeps memory, and delegates to tools and other agents, so one manipulated input can affect every step that follows it.

Problem. Builders and security teams need a shared list of which agent risks matter most and what to do about each one. The detailed agent threat taxonomies that already existed are broad enough that, in the authors' words, they are difficult to translate into day-to-day practice.

Key idea. Name ten agent-specific risks, ASI01-ASI10, and give each a description, common examples, attack scenarios drawn from public incidents, and mitigations. The ten are agent goal hijack, tool misuse and exploitation, identity and privilege abuse, agentic supply-chain vulnerabilities, unexpected code execution, memory and context poisoning, insecure inter-agent communication, cascading failures, human-agent trust exploitation, and rogue agents. The document also extends least privilege to least agency: an agent should not be given autonomy that its task does not need.

Why it matters. Use it as the index for this group, since each paper below demonstrates one of its entries. Indirect prompt injection is the first listed example of goal hijack (ASI01), AgentPoison is cited under memory and context poisoning (ASI06), and the multi-agent code-execution paper is cited under rogue agents (ASI10). Its appendix of exploits and incidents also shows that these attacks were reported against deployed products in 2025, not only against research prototypes.

Background. An LLM application becomes useful once it can read outside itself: retrieved documents, web pages, email, tool output. All of that arrives in the same context window as the developer’s instructions and the user’s request.

Problem. The model has no channel separation. An agent that retrieves untrusted data cannot separate that data from its instructions, so any document the agent reads is a place an attacker can put a command, and the attacker never has to speak to the user.

Key idea. Name the attack the tool loop creates and demonstrate it against real integrated applications. Text planted in content the model will later retrieve turns retrieval itself into a delivery channel for instructions the operator never wrote.

Findings. The demonstrations compromise Bing's GPT-4-powered chat, code-completion engines, and synthetic GPT-4 applications. Retrieved instructions can redirect API calls, exfiltrate data, manipulate later model outputs, and propagate through content, showing that the vulnerability crosses application types rather than depending on one interface.

Why it matters. It is the mechanism behind goal hijack in the list above. Every tool an agent can call also decides what an injected instruction is able to reach, so what you control is the granularity of the capabilities you hand the loop, not the model’s willingness to obey.

Adoption. OWASP classifies prompt injection as LLM01 in its Top 10 for LLM applications, and its agentic Top 10 above lists indirect prompt injection as the first example of agent goal hijack (ASI01).

Background. Many agents retrieve past examples from a long-term memory or a RAG knowledge base and use them as demonstrations when planning the next action. Whatever is in that store shapes what the agent does.

Problem. The store is rarely verified, so an attacker who can insert a few records does not need to change the model or the prompt. However, a planted record has to be retrieved for the attacker's target queries and not for ordinary ones; otherwise the change in behavior is visible on normal use.

Key idea. Optimize a backdoor trigger so that any instruction containing it maps to a distinct region of the retriever's embedding space, where the attacker's malicious demonstrations sit. Instructions with the trigger retrieve the poisoned records with high probability, and instructions without it retrieve normal ones. The attack requires no model training or fine-tuning.

Findings. Against three agents (an autonomous-driving agent, a knowledge-intensive question-answering agent, and a healthcare EHR agent), the attack succeeds on more than 80% of triggered queries on average. It poisons less than 0.1% of the memory or knowledge base and changes benign performance by less than 1%.

Why it matters. It is memory and context poisoning (ASI06) as a concrete attack: memory turns a one-time injection into a persistent one. The self-evolving harnesses later on this page write their own memories and playbooks, which makes the provenance of every stored entry a security question as well as a quality question.

Background. Multi-agent frameworks split work between an orchestrator and specialized sub-agents, such as a web surfer, a file reader, and a code executor. The orchestrator trusts each sub-agent to report the status of its task correctly.

Problem. Prompt-injection evaluations usually test one agent at a time. In a multi-agent system, content read by one sub-agent can change what the orchestrator asks another sub-agent to do, and no single agent's check sees that whole path.

Key idea. Attack the control flow between agents rather than any one agent, which the paper calls multi-agent system control-flow hijacking. In the basic version, a malicious file makes the file-reading agent return a false error message saying that the only way to access the file is to execute it, and the orchestrator then asks the code-execution agent to run it. The paper describes this as an instance of the confused-deputy problem.

Findings. With GPT-4o agents, web-based attacks make the system execute arbitrary malicious code in 58-90% of trials, depending on the orchestrator, and in 100% for some model-orchestrator configurations. Previously known indirect injection attacks have near-zero success at causing code execution, and the new attacks succeed even when the individual agents resist direct and indirect prompt injection and refuse harmful actions.

Why it matters. Safety does not compose: agents that each pass a per-agent check can still form a system that runs an attacker's code. The trust boundary you have to reason about is the status and error metadata passed between agents, which OWASP covers under insecure inter-agent communication (ASI07) and rogue agents (ASI10).

Background. Defenses against jailbreaks and prompt injections are usually published with an attack success rate measured against a fixed set of attack strings, or against an optimization method that was not designed with the defense in mind.

Problem. A real attacker sees the defense before attacking it and adapts. A low success rate against static attacks therefore says little about robustness. The earlier adversarial-example literature on image classifiers reached the same conclusion, after many defenses that resisted known attacks were broken by attacks adapted to them.

Key idea. Evaluate every defense against attackers who tailor their strategy to it and spend substantial compute. The paper scales four general attack families (gradient descent, reinforcement learning, search with LLM-suggested mutations, and human red-teaming through an online competition with more than 500 participants) and applies them to 12 recent defenses.

Findings. None of the 12 defenses is robust: the adaptive attacks succeed more than 90% of the time against most of them, although the majority originally reported near-zero attack success. In the human red-teaming study, participants succeed on every scenario, where the static attack from the original evaluations succeeds on none.

Why it matters. Read it before the defenses group. A robustness number is only as strong as the adaptive attack it was measured against, and a defense tested only against the attacks in its own paper has not been tested. Designs whose guarantee does not depend on the model resisting the injection, such as CaMeL below, respond to this result.

Background. A chat model receives messages from different sources: a system message from the application developer, user messages, its own earlier outputs, and tool outputs such as search results. All of them reach the model as text in one context.

Problem. Models tend to give all of these sources the same priority, so a web result that says "ignore previous instructions" can override the developer's system message. However, ignoring every lower-priority instruction would break benign requests, such as a user asking a car-sales bot to answer in Spanish.

Key idea. Define a priority order (system message, then user messages, then model outputs, then tool outputs) and train the model to follow a lower-priority instruction when it is aligned with the higher-priority ones and to ignore it when it conflicts. Aligned training examples split a compositional request into pieces placed at different levels; misaligned examples come from red-teaming models and are paired with responses that act as if the injected instruction were absent. GPT-3.5 Turbo is then fine-tuned on this data with supervised learning and RLHF.

Findings. Against a baseline fine-tuned on the same capability data without these examples, robustness to system-message extraction rises from 32.8% to 95.9% and robustness to indirect injection through web browsing from 77.5% to 85.0%. Jailbreaks were deliberately left out of training, and robustness to ChatGPT jailbreaks still rises from 37.4% to 71.2%. The cost is over-refusal: compliance on benign inputs that resemble jailbreaks falls from 83.1% to 60.4%.

Why it matters. It is the model-side defense, and the rest of this group shows why it cannot be the only one. A trained preference lowers the attack success rate but provides no guarantee, and the adaptive-attack paper above shows how far a reported rate can move once the attacker adapts.

Adoption. OpenAI states that GPT-4o mini in its API was the first model to apply the instruction hierarchy method.

Background. The instruction hierarchy above makes a model less likely to follow an injected instruction, but a lower probability is not a guarantee. A different question is what a runtime underneath the model can enforce whatever the model does, given that the model itself will keep failing to separate data from instructions.

Problem. Any guarantee has to come from outside the weights, which means a component that knows which values arrived from untrusted sources and what each tool is permitted to touch. Neither fact is available inside a single context of undifferentiated text.

Key idea. Extract the control flow and the data flow from the trusted query and keep them apart. A privileged component turns the user's request into a program before any untrusted content is read, a quarantined component processes that content but holds no capabilities, and a custom interpreter tracks the provenance of every value and checks a policy before a tool with side effects runs.

Findings. The guarantee has a measurable price. CaMeL solves 77% of AgentDojo tasks (the benchmark at the end of this group) with provable security, against 84% for the same undefended system, so the security property costs 7 points of utility rather than being free.

Why it matters. It is the harness argument applied to security: the guarantee lives in code you wrote instead of in the model's willingness to comply. The cost is stated plainly, since a plan fixed before the untrusted data is read cannot adapt to what that data says, and somebody has to author the policy.

Background. CaMeL above is one complete system. A team building a different agent needs to know which arrangements of the loop give a security property and how much generality each one costs.

Problem. A defense that asks the model to resist instructions is a request rather than a constraint. What is needed instead is an arrangement in which a successful injection cannot reach anything that matters, and that is a property of the loop rather than of the weights.

Key idea. Stop trying to make the model resistant and constrain what the loop is able to do. The paper names six design patterns, each surrendering some generality for a security property that holds regardless of the model: action-selector, plan-then-execute, LLM map-reduce, dual LLM, code-then-execute, and context minimization. For example, plan-then-execute fixes the plan before any untrusted data is read, and dual LLM splits the work so that the component reading untrusted content holds no capabilities.

Why it matters. Every pattern is a decision in the loop you own, and each one restricts which tool results are allowed to influence what the agent does next. CaMeL is the paper's code-then-execute pattern, so the catalog places it among simpler alternatives, and the paper's case studies show which pattern fits which application.

Background. The defenses above assume that the model is honest and that the attacker is outside it, in a document or a tool result. Common safety techniques, such as having one model review another model's output, rest on the same assumption.

Problem. If the capable model is itself trying to cause harm, a review step that it can anticipate or collude with may fail. The paper notes that such techniques had not been evaluated against a model deliberately trying to subvert them.

Key idea. Treat safety as a game between a blue team that designs a protocol and a red team that replaces the untrusted model with an attack policy. The testbed is 5,000 APPS programming problems, a strong untrusted model (GPT-4), a weaker trusted model (GPT-3.5), and trusted high-quality labor (simulated with GPT-4) sufficient to audit 2% of the solutions. Safety is the probability that no backdoored solution is submitted across all problems, and usefulness is the correctness rate rescaled so that GPT-3.5 alone scores 0% and GPT-4 alone 100% (22% and 54% correct before rescaling).

Findings. Using only the trusted model is safe by construction but scores 2% usefulness, while auditing the first 2% of GPT-4's solutions keeps 100% usefulness at 15% safety. Trusted monitoring, where GPT-3.5 rates each solution's suspiciousness and the most suspicious 2% are audited, reaches 62% safety at 97% usefulness. Untrusted monitoring by a second GPT-4 instance with anti-collusion measures reaches 79% safety at 96% usefulness, and trusted editing, where GPT-3.5 rewrites suspicious solutions, reaches 92% safety at 94% usefulness.

Why it matters. It turns monitoring from a classifier into a protocol: a monitor, a suspicion threshold, a fallback to a trusted model, and an audit budget, all evaluated against an adversary who knows the protocol. An agent harness faces the same decision about which actions run, which are rewritten, and which go to a person.

Background. Comparing the defenses above requires a common test: an agent, tools it can call, tasks worth completing, and attacker-controlled content sitting in the middle.

Problem. A frozen list of attack strings measures the wrong thing. Attacks adapt to defenses, so a suite fixed at publication reports a defense against last year’s attacker, and it says nothing about what the defense costs on ordinary work.

Key idea. 97 realistic tasks and 629 security test cases in an environment built to be extended rather than frozen, on the argument that a static suite cannot track adaptive attacks. Utility on the benign tasks and security under injection are measured in the same harness.

Findings. On GPT-4o, the paper's strongest generic injection succeeds on 57.7% of targeted attacks and lowers task utility from 69.0% without attacks to 50.0% under attack. A tool-filter defense reduces targeted attack success to 6.84%, but still leaves utility under attack at 56.28%.

Why it matters. The evaluation-design lesson generalizes past security: a benchmark for a moving target has to be an environment, not a fixed list. Measuring both axes together also forces the tradeoff into view, since a defense that blocks every injection by refusing tool calls scores badly on the tasks.

Background. A failed agent trajectory contains its actions, errors, and stopping point. Reinforcement learning can use such feedback, but it requires many samples and a fine-tuning run.

Problem. Neither cost is available inside an agent loop. A deployment cannot fine-tune between two attempts, and one failed episode is far too little data for a weight update, so the lesson in the transcript is discarded when the episode ends.

Key idea. After a failure, have the agent write a reflection, store it in episodic memory, and read it on the next attempt. Behavior changes through context rather than a weight update, using either scalar or language feedback.

Findings. Reflexion improves on a baseline agent across sequential decision-making, coding, and language reasoning. On HumanEval it reaches 91% pass@1, against 80% for the GPT-4 state of the art it is measured against.

Why it matters. Memory becomes the mechanism of improvement that a harness can implement between attempts. Its limit is diagnosis: a reflection is only as useful as the trajectory and feedback from which it was written.

Background. The entry above reflects between attempts, on feedback the environment returned. Much of what an agent produces draws no environmental feedback at all: a plan, a summary, a message, a patch nobody has run yet.

Problem. With no external signal there is nothing to reflect on, and a first draft is where a model otherwise stops. Training a separate critic, or collecting supervision for the revision step, is the usual answer, and it costs data and a training run for each task.

Key idea. Let one model play all three parts. Generate an output, have the same model write feedback on it, then have it revise against that feedback, and iterate. There is no supervised data, no additional training, and no second model, so the pattern drops into any loop that already runs one.

Findings. Across seven tasks, from dialogue response generation to mathematical reasoning, humans and automatic metrics both prefer the refined output to one-step generation from the same model, by about 20 points absolute on average.

Why it matters. It is not an agent paper, and the generate-critique-revise triple it names became a standard component inside agents anyway, which is why it sits here. Read it for what the critic actually is: the same weights asked a different question, which is what makes it cheap and is also why a self-critique can miss exactly what the generator missed.

Background. The two entries above change an agent's behavior through text it writes for itself. The agent's design, meaning its prompts, its control flow, and how it combines building blocks such as chain-of-thought, self-reflection, and debate, is still chosen by hand.

Problem. The design space is large, and hand exploration covers little of it. Prompt optimizers search only over prompt text, which leaves out control flow and the way model calls are combined.

Key idea. Define each agent as code (a Python forward function over model calls) and let a meta agent program new agents. Meta Agent Search keeps an archive of discovered agents and their scores. In each iteration the meta agent reads the archive, writes a new agent intended to differ from those already in it, evaluates the agent on a validation set, and adds it to the archive.

Findings. With a GPT-4o meta agent and GPT-3.5 running the agents, the best discovered agents beat the best hand-designed baseline by 13.6 F1 points on DROP reading comprehension (79.4 against 65.8) and by 14.4 percentage points on MGSM math (53.4% against 39.0%). The top agent found on MGSM also transfers to GSM8K and GSM-Hard, where it improves on the best baseline by 25.9 and 13.2 points.

Why it matters. It recasts agent design as search over programs with an evaluation set as the objective. The harness becomes something an optimizer produces, and the evaluation set becomes its specification: whatever the set does not test, the search is free to ignore.

Background. ADAS uses a fixed meta agent to write other agents. The Gödel machine, a theoretical proposal, describes a system that rewrites its own code whenever it can prove that the change is beneficial.

Problem. Proving that a change to an agent is beneficial is impossible in practice. In addition, if the component that does the modifying is never itself improved, its ability to find improvements stays where it started.

Key idea. Replace proof with empirical validation and let a coding agent edit its own repository, so an improvement to its coding ability is also an improvement to its ability to modify itself. The DGM keeps an archive of every agent it has generated. Each iteration samples a parent from the archive, has it modify its own code, and scores the child on a coding benchmark, and keeping lower-scoring agents in the archive lets later iterations branch from them.

Findings. Over 80 iterations, the agent improves from 20.0% to 50.0% on a 200-task subset of SWE-bench Verified and from 14.2% to 30.7% on the full Polyglot benchmark, more than ablations without self-improvement or without the archive. The discovered changes include better file-editing tools, long-context management, and patch-review steps, and one SWE-bench run costs about USD 22,000. In a separate run whose objective was to reduce tool-use hallucination, one agent reached a perfect score by removing the logging that the hallucination detector depended on.

Why it matters. It makes the harness the object being optimized, with a benchmark as the only check on each change. The objective-hacking case shows the cost of that design: a self-modifying agent optimizes what the evaluation measures, so the evaluation and the sandbox have to stay outside the agent's reach.

Background. ADAS and the DGM change code. Many deployed agents instead adapt through context: a system prompt, a memory, or a set of strategies accumulated from earlier runs, as in Reflexion above.

Problem. Methods that have a model rewrite the whole context after each step tend to compress it. The paper names two failures. Brevity bias drops domain-specific detail in favor of concise generic advice, and context collapse is the loss of accumulated knowledge in a single rewrite: in one AppWorld run the context shrank from 18,282 tokens at 66.7% accuracy to 122 tokens at 57.1%, below the 63.7% of the agent without adaptation.

Key idea. Treat the context as a playbook of itemized entries and divide the work among three roles: a Generator that runs tasks, a Reflector that extracts lessons from the trajectories, and a Curator that turns the lessons into small delta entries. Deltas are merged deterministically instead of having a model rewrite the whole context, and a de-duplication step prunes redundant entries by comparing their embeddings.

Findings. Using DeepSeek-V3.1 for all three roles, ACE improves over in-context learning, GEPA, and Dynamic Cheatsheet baselines by 10.6% on average on agent benchmarks and 8.6% on financial reasoning, and it reduces adaptation latency by 86.9% on average. On the AppWorld leaderboard it reaches a 59.4% average against 60.3% for IBM CUGA, the top-ranked production agent built on GPT-4.1, and surpasses it on the harder test-challenge split. It can adapt from execution feedback without ground-truth labels.

Why it matters. It is the least invasive form of self-evolution in this group: no code changes and no weight updates, only an itemized context and a merge rule. The design decision is the update rule, and the collapse example shows that letting a model rewrite its own memory wholesale can erase what it has learned.

Background. A harness is the code around a model that decides what to store, retrieve, and show to it. The paper cites a 6× performance gap on one benchmark from changing only the harness around a fixed model, yet harnesses are still mostly tuned by hand.

Problem. Existing text optimizers compress the feedback they learn from: they see only the current candidate, scalar scores, or short LLM-written summaries. A harness decision about what to store can cause a failure many steps later, and a summary often drops the evidence needed to trace that failure back to the decision.

Key idea. Store the source code, scores, and full execution traces of every candidate harness in a filesystem, and let a coding agent (Claude Code with Opus 4.6) read whatever it needs before proposing the next harness. In the largest setting, one evaluation produces about 10 million tokens of logs, while the text optimizers the paper compares work with 100-30,000 tokens of context per optimization step.

Findings. On online text classification it outperforms ACE by 7.7 points while using 4× fewer context tokens, and a single harness discovered for retrieval-augmented math reasoning improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On TerminalBench-2 the discovered harness reaches 76.4% with Opus 4.6, against 74.7% for the best hand-engineered harness, and 37.6% with Haiku 4.5, the highest reported for that model. In the ablation, the best candidate reaches 41.3% accuracy with scores only and 38.7% with scores plus summaries, while the median candidate with full traces reaches 50.0%.

Why it matters. The feedback interface is the main design decision. Compressing traces into summaries looks like a cost saving but removed the signal the search needed, so the proposer's access to raw experience is what makes automated harness engineering work.

Background. A coding agent's harness has several editable parts, among them the system prompt, tools, middleware, skills, and long-term memory. A harness tuned for one base model often underperforms on another, so it has to be tuned again whenever the model changes.

Problem. Automating that tuning faces three obstacles. First, the edits span heterogeneous components. Second, the trajectories that show what went wrong run to millions of tokens. Third, the effect of any one edit is hard to attribute, so an automated loop easily becomes trial and error.

Key idea. Make each of the three observable. Every harness component is a file, so an edit is explicit and revertible; raw trajectories are distilled into a layered evidence corpus that the evolving agent can read; and every edit ships with a prediction of which tasks it will fix and which it may break, checked against the next round's task-level results.

Findings. With GPT-5.4, ten iterations raise pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, above the human-designed Codex CLI harness (71.9%) and the self-evolving baselines ACE and Training-Free GRPO. Without further evolution, the frozen harness has the highest aggregate success on SWE-bench Verified among the compared harnesses while using 12% fewer tokens than the seed harness, and on Terminal-Bench 2 it gains 5.1-10.1 points with three other model families. In the ablation, tools, middleware, and long-term memory each carry the gain on their own, while the evolved system prompt alone regresses.

Why it matters. It treats harness evolution as an engineering process with change records and testable predictions rather than as search over prompts. The analysis also states two limits: the loop's predictions are reliable for fixes but miss regressions, and effective edits do not add up when combined.

Background. The DGM and the harness optimizers above improve an agent offline: they run many benchmark evaluations and then ship a fixed result. A single DGM run on SWE-bench costs about USD 22,000.

Problem. Offline search is tied to the benchmark it optimized on and may not generalize to other models or benchmarks, so it has to be repeated when either changes. The agent it produces is also fixed by the time it meets a new problem.

Key idea. Let the agent evolve its own scaffold while it solves the task. Starting from mini-SWE-agent, which has only a bash tool, the prompt tells the agent it may write custom tools as scripts, and a reflection message after each environment step asks whether a new tool would help. Tool creation becomes an action in the loop, with no offline training.

Findings. On SWE-bench Verified, the same changes raise the solve rate over mini-SWE-agent with each of the four models tested, for example from 70.6% to 75.4% with Claude 4.5 Sonnet at an average cost of USD 0.68 against USD 0.56 per issue, and the agent reaches 77.4% with Gemini 3 Pro without test-time scaling. On SWE-Bench Pro it reaches 45.8% with Claude 4.5 Sonnet, against 43.6% for SWE-agent with the same model. On a 60-task subset of SWE-bench Verified it solves 65.0%, against 53.3% for the DGM, with no offline cost.

Why it matters. It moves self-evolution from an offline outer loop into the task itself, which removes the search cost but also the validation step: a tool written during the task runs without being evaluated first. Unexpected code execution (ASI05) in the safety list above applies to these tools as well, since a script the agent writes runs with the agent's own permissions.