My idea would be that the ngram size over which the watermarking works is necessarily limited in order to resist edits better. It might be possible to lead the model to trigger the refusal in the form of these specific ngrams, the completion of which is then more likely flipped to compliance (due to the logit bias introduced by the watermarking), making hazardous requests systematically more likely to be accepted?

Detection of watermarking requires access to the watermarking key, a secret in the current suggested scheme (leaking it would amount to being able to strip the watermark).

So, there will need to be a watermark checking service. The checking service will of course be rate-limited for common folk (and model distillers). OpenAI/Anthropic/Google/other privileged model builders need to filter out AI slop at scale, so need access to others' service without rate-limits (or the watermarking keys need to be shared).

This creates an in-group with pristine datasets, and an outgroup whose models will collapse on the slop outputs with no good ability to filter.

No it's EU law.

Let's say there are four billion possible seeds. There are four billion possible ways we could watermark the generation. We could say "we will choose seed 1, that way we will know exactly what output it produced", we could say "we will choose seed 2, that way we will know exactly what output it produced"... etc etc. Now, if we decide "not to watermark", we STILL must choose a seed. So we are actually still applying one of the watermarks, the only difference is we are not careful to remember which one. Could some seeds give a better or worse answer to some specific prompt? Yes. Could choosing a random "watermark" to apply be better or worse on average than choosing a random seed to apply? No. It's mathematically impossible.

This is like an open source project changing their seed from "12321" to "43", and saying that because we changed the seed, the quality is "necessarily lower".

I just took his text, pasted it to ChatGPT, "rephrase", paste it back in the online checker, 0% AI.

This is because SynthID and similar watermarking methods for LLMs don't just change the random seed. They take additional steps (that I have yet to read about) in order to detect when someone changes a few words of the output, trying to remove the watermark.

The other takeaway is that just by knowing a watermark is being applied gives an attacker an advantage in working around safety features because then they know the output isn't based on true randomness and can take advantage of that in a similar fashion to how breaking cryptography becomes easier when the RNG isn't truly random.