Adaptive defense middleware for LLM agents

CAITLYN

Autonomous defense synthesis for emerging LLM-agent injection attacks.

An agent-agnostic defense middleware that turns prompt-injection misses into verified, reusable defense skills.

Paper Code Results

Tier-0 scripts Tier-1 LLM classifier System-II synthesis

Zi Liang^1† Xiaoyu Xu^1† Yanyun Wang² Minxin Du¹ Qingqing Ye¹ Haibo Hu^1*

¹The Hong Kong Polytechnic University ²The Chinese University of Hong Kong

^†Equal contribution ^*Corresponding author

CAITLYN character visual inspired by a cybernetic marksman

caitlyn.scan

threat score 0.87

$ ingest external_content.json

TIER-0 executable skills: miss

TIER-1 classifier: malicious

SYSTEM-II counterexample queued

VERIFY deploy synthesized skill

library 24 seed skills mode fail-open daemon evolve CEGIS loop

24seed defense skills

100%AgentDojo detection TPR

~40ppEmerging ASR reduction

4verified evolved skills

Overview

Agents now read the attack surface

LLM agents now consume webpages, local files, API payloads, user replies, and MCP tool outputs. These inputs can carry indirect prompt injections that ask the underlying model to execute harmful actions outside its benign scope.

CAITLYN protects this boundary by treating defenses as executable, self-extending skills. Runtime misses become counterexamples, and counterexamples become specifications for synthesizing stronger defenses.

defense.trilemma

{
  "regex_guard": "fast but brittle",
  "llm_judge": "accurate but costly",
  "offline_retrain": "adaptive but slow",
  "caitlyn": "runtime gate + synthesis"
}

System

Two-tier scanning plus lifelong synthesis

External content

Search results, files, webpages, user replies, API payloads, and MCP observations enter the middleware.

System I · Tier-0

Preloaded executable skills block high-confidence malicious content with no token cost.

System I · Tier-1

Merged-pair LLM calls classify uncertain inputs through compact status-and-score contracts.

System II

Counterexample-guided synthesis mutates, reviews, verifies, and deploys new defense skills.

Benchmark

Standard benchmark results

CAITLYN is evaluated on AgentDojo-S250, ASPI-S, and SafeClawBench-S240 across OpenCode, Codex, Pi, Hermes, and OpenClaw. It drives end-to-end attack success close to zero while keeping AgentDojo benign FPR at 0.0%.

Agent	AgentDojo ASR	ASPI ASR	SafeClawBench ASR	Best reported latency
OpenCode + CAITLYN	0.0%	2.2%	2.9%	7.2s on SafeClawBench
Codex + CAITLYN	0.0%	2.2%	4.6%	5.3s on SafeClawBench
Pi + CAITLYN	0.0%	2.2%	2.1%	5.4s on AgentDojo
OpenClaw + CAITLYN	0.4%	1.1%	3.8%	11.7s on ASPI

100% AgentDojo-S250 detection TPR at default threshold

0.00100 USD per inspection for the full System-I ablation run

0.0% AgentDojo benign FPR across all CAITLYN agent runs

Detection ROC and PR curves — Detection quality across the four attack datasets.

Detection latency and cost Pareto frontier — Latency-cost frontier for detection-only experiments.

Emerging

Adapting to attacks absent from the initial library

Emerging embeds indirect prompt injections into local file reads, web search results, and external webpages. Static defenses remain vulnerable, while CAITLYN-evolved converts misses into verified skills and cuts ASR by about 40 percentage points across OpenClaw, Codex, and Hermes.

77-79.5% CAITLYN-static ASR before synthesis on Emerging

38.5-39.5% ASR after adding four evolved skills

38/38 adaptive evasions recovered after one defender update

Emerging benchmark ASR comparison — CAITLYN-evolved breaks out of the static baseline band on Emerging.

Lifelong synthesis curve — Sequential synthesis accumulates four active skills while preserving 1.6% FPR.

Library

Defense and attack entries are filesystem-native

Defense skills

Each skill is a portable directory with natural-language documentation, structured metadata, thresholds, optional prompts, and executable detectors.

README.md · config.yaml · detect.ts

Counterexamples

Runtime misses become synthesis specifications that preserve payload evidence, triggering context, and nearest attack samples.

payload · channel · behavior trace

Verification sandbox

Candidate defenses are tested against adversarial positives and benign constraints before they enter the active library.

fitness · lineage · benign guard

Citation

BibTeX

@misc{liang2026caitlyn,
  title  = {CAITLYN: Can LLM Agents Autonomously Synthesize Defenses against Emerging Injection Attacks?},
  author = {Liang, Zi and Xu, Xiaoyu and Wang, Yanyun and Du, Minxin and Ye, Qingqing and Hu, Haibo},
  year   = {2026},
  note   = {Project paper}
}