Autonomous agents search for the deferral policy that lets a small, un-fingerprinted model write as much of the answer as possible while matching a large model's quality — and we test whether the large model's watermark survives the dilution.
This is a living report: the numbers and charts below update as the agents run more experiments.
A model fingerprint or watermark lets a provider prove that some text came from their model — to enforce terms of use, detect model theft, or trace a leaked checkpoint. Watermarks perturb generation so outputs carry a statistically detectable signal invisible to a reader; the canonical scheme is the Kirchenbauer et al. red-green watermark
This report studies one such step, deliberately cheap — no fine-tuning, no gradients on the target, no detector queries — just inference-time orchestration of a small model the attacker owns alongside the watermarked target.
Fix a cheap weak model (here Llama-3.2-1B) that we run white-box: we can read not just its per-token output probabilities but its internal activations — a far richer signal the agents have barely begun to exploit (there may be, for instance, a "confidence" direction worth probing). The expensive strong model (Qwen2.5-7B) we treat as a black box: text in, text out. We generate each answer collaboratively — the weak model proposes tokens, and at every step a deferral policy decides whether to keep the weak model's token or hand the next span to the strong model. Which white-box signal that policy reads is exactly what the agents are searching over. Because the two models tokenize differently, the handoff happens at the text level.
Two quantities score a policy. The first is $f_\text{weak}$, the fraction of output characters the weak model produced. The second, measured against the strong model itself, is utility recovery: with $U$ the method's judge-scored utility (an LLM-judge win-rate against the strong model's own outputs, so the strong model scores $U_\text{strong}=0.5$ against itself),
where $\text{recovery}=1$ means the collaboration is as good as the strong model. The objective is to maximize $f_\text{weak}$ subject to $\text{recovery} \ge 0.98$: push as much generation as possible onto the cheap model without losing quality.
We do not hand-design the policy; autonomous Claude agents search for it, using Anthropic's Automated Weak-to-Strong Researcher framework
The current state of that search — live:
Explore the full research feed — every finding the agents have shared →
Nothing in the search above mentions watermarks — the agents optimize only $f_\text{weak}$ and recovery. Separately, we reinterpret the same machinery as an attack. Let the strong model be a watermarked target queried as a black box, and the weak model an un-watermarked model the attacker owns. Then $f_\text{weak}$ is the fraction of the final text that did not come from the watermarked model — the dilution — and the question is whether a high-$f_\text{weak}$, quality-preserving policy washes the watermark out. Crucially, the agents never see the detector: we measure detection post-hoc on the utility-optimal policies, so this is not a policy tuned to evade a watermark.
We test two watermarks, both KGW-family (each seeds a pseudo-random green list from the previous token and detects an excess of green tokens with a z-test) but installed very differently: the original inference-time red-green watermark
This is a living document: the leaderboard, the frontier, and the watermark graph above all update as the agents run more experiments. Once the search settles, we may write up the single best policy found and whether it is a positive or a negative result for fingerprint removal.