Persistent memory. Persistent consequences.

When Routine Chats
Turn Toxic

Unintended Long-Term State Poisoning in Personalized Agents

An ordinary conversation can become a rule for tomorrow. We study how routine interactions reshape persistent agent state—and how to audit those changes before they are saved.

Xiaoyu Xu†, Minxin Du†*, Qipeng Xie◇, Haobin Ke†, Qingqing Ye†, Haibo Hu†*

†The Hong Kong Polytechnic University
◇Hong Kong University of Science and Technology, HKUST (Guangzhou)

* Corresponding authors

350conversation settings
2languages · English & Chinese
7interaction patterns
4backbone models evaluated

The research

Small requests. Lasting changes.

Personalization depends on memory. The question is what gets carried forward.

01 / The risk

Local permission becomes policy

Routine interactions can weaken future confirmation boundaries, broaden tool-use defaults, or increase unchecked autonomy—even without an attacker.

02 / The benchmark

Inspect the state, not just the chat

ULSPB pairs 24-turn routine conversations with single-injection variants across five assistance categories. Harm Score screens the resulting state diffs for predefined risk patterns.

03 / The defense

Check before committing

StateGuard audits proposed persistent-state updates and can roll back risky changes. The explorer below shows the conversations, file changes and recorded audit decisions.

Selected findings

The risk is measurable. So is the reduction.

Two results from the paper: risky state updates during routine interactions, and lower residual HS after writeback auditing.

01 / Routine interactionsTable 3 ↗

Routine chats alone produce non-zero HS.

No explicit injection in this condition. Values are mean Harm Score ± standard deviation.

Mean HS in the Routine condition across four backbone models; bars use a common zero-to-eight scale.
BackboneMean HS · 0–8Mean ± SD
Kimi K2.54.52 ± 0.62
GPT-5.45.47 ± 0.73
MiniMax M2.76.66 ± 0.86
Grok 4.203.90 ± 0.55

Routine condition only; both benchmark languages. Higher HS indicates more matched persistent-state risk patterns.

02 / StateGuardTables 4–5 ↗

Audit the writeback. Reduce residual risk.

Mean HS across all 350 conversation settings, comparing no defense with Targeted-Ensemble StateGuard.

Mean HS before defense and after Targeted-Ensemble StateGuard across the full benchmark.
BackboneNo defenseStateGuard
Kimi K2.54.420.06
GPT-5.45.530.05
MiniMax M2.76.420.14
Grok 4.204.350.03
$0.0036average auditing cost per run
Targeted-Ensemble · reported experimental pricing

HS measures state-diff risk patterns, not the rate of unsafe actions. StateGuard uses conservative auditing and can reject benign updates; lower residual HS alone does not measure retained personalization.

Inside the benchmark

Follow a conversation into memory.

Choose a recorded run, move through the conversation, and inspect what changes.

Recorded trajectories
  1. 1 Choose a model and setting
  2. 2 Step through the interaction
  3. 3 Inspect state diffs and audit votes

Illustrative runs are selected by Harm Score within the available settings. This explorer is not a prevalence estimate. HS flags risk patterns; authorization still requires context.

Loading recorded trajectories…

Conversation replay

Step 1 / 1

A replay of recorded interactions. No messages are sent and no agent actions are executed here.

Persistent-state changes

Compare the protected files before and after the selected interaction.

    
          

    StateGuard audit

    For defended runs, inspect each auditor’s vote, reasoning and final write decision.

    How the Harm Score is calculated

    Inspect the matched dimensions, state-role weights and contribution of each changed file.

    Build on this work

    Citation

    @misc{xu2026routinechatsturntoxic,
      title = {When Routine Chats Turn Toxic: Unintended Long-Term State Poisoning in Personalized Agents},
      author = {Xiaoyu Xu and Minxin Du and Qipeng Xie and Haobin Ke and Qingqing Ye and Haibo Hu},
      year = {2026},
      eprint = {2605.06731},
      archivePrefix = {arXiv},
      primaryClass = {cs.CR},
      url = {https://arxiv.org/abs/2605.06731}
    }