The European Union AI Act will statesman to beryllium enforceable successful August 2026, 1 period from now1. One of the biggest caller requirements is Article 50, which requires each AI outputs to beryllium “detectable arsenic artificially generated”. In different words, if LLM providers want to do business successful the EU, they will person to use a watermark to their outputs2: immoderate hidden signature that tin beryllium utilized to place AI content.
LLM matter watermarking is simply a fascinating problem. Like the champion engineering problems, it is theoretically difficult to lick perfectly, but has aggregate partial solutions: for instance, Google’s SynthID, and (as I’ll argue) immoderate quiet Unicode trickery from OpenAI and Anthropic. It will beryllium absorbing to spot really the AI labs navigate these tradeoffs earlier the extremity of the year.
Why matter watermarking is hard
I wrote astir AI watermarking astatine the extremity of past twelvemonth successful AI discovery devices cannot beryllium that matter is AI-generated. It’s easy to watermark an image, because integer images incorporate tons of sound that the quality oculus can’t really see. For instance, you could use a watermark for illustration “these 20 pixels successful these nonstop spots will ever stock a color”. Text is much, overmuch harder. Unlike images, matter is simply a very compressed medium: you cannot make immoderate alteration to a condemnation that a quality wouldn’t announcement (with 1 exception, which we’ll get to later). So really are you expected to watermark it?
It’s fundamentally a text steganography problem (concealing a concealed code), made much difficult because the plaintext cannot beryllium arbitrarily manipulated. Any changes you make to use the watermark will discuss the value of the output. For instance, “every 5th missive is an ‘e’” would beryllium a bully watermark, but applied naively would make the AI output afloat of typos. Could you conscionable fto the exemplary fig retired really to fresh the watermark? Strong AI models are smart capable to juggle this benignant of constraint3, but it’d still devour reasoning clip that would beryllium amended spent connected the user’s problem, and make the exemplary sound overmuch little tin than it is4.
Do we request watermarks to observe AI content?
Do you really request a watermark? If you’re Anthropic, and you’re required to beryllium capable to verify whether your models produced a peculiar artifact of text, can’t you simply tally the matter done each model, measuring arsenic you spell really intimately the model’s predicted tokens lucifer each token from the text?
Not really. The abstraction of “all imaginable Claude Sonnet answers to a question” is measurement larger than the abstraction of “all imaginable watermarked answers to a question”. In different words, you’d get excessively galore mendacious positives for quality matter that sounds for illustration it was AI-written. It’s measurement much apt for a quality to accidentally constitute for illustration Claude than it is for a quality to accidentally reproduce a watermark.
It would besides beryllium prohibitively costly to tally each Anthropic exemplary against a portion of matter successful bid to watermark it. The EU AI Act will yet require labs for illustration Anthropic to connection free watermarking services to each EU national (see Commitment 2). You couldn’t do that pinch the “run the model” approach.
How SynthID works
As acold arsenic I know, the only AI supplier to opportunity they watermark matter output is Google, who usage a instrumentality called SynthID. Here’s really it works.
When an LLM generates text, it’s generating a bid of tokens (words aliases chunks of words). At each step, the exemplary itself doesn’t output a azygous token, but alternatively outputs a afloat database of each (say) 100,000 tokens successful its vocabulary, each annotated pinch the probability that that token will beryllium the adjacent one. Tools for illustration ChatGPT aliases Claude Code will prime semi-randomly from the astir apt options successful bid to get their outputs. This semi-random sampling process tin beryllium influenced successful a detectable way.
For instance, we could take a sampling strategy for illustration “we prime the 2nd astir apt token, past the first, past the second, past the first, and truthful on”. That would still nutrient high-quality output, but you’d beryllium capable to re-run the exemplary against the generated matter to verify that the shape holds. However, that’d make verification really expensive, and immoderate flimsy tweaks to the output would break the shape and frankincense break the fingerprint. Is location a amended way?
Yes. SynthID is simply a process for assigning each token a “score” based connected its erstwhile tokens (for instance, sum the token’s ID pinch the IDs of its erstwhile 3 tokens past return mod 5)5. To use the watermark, the exemplary adopts a sampling strategy for illustration “out of the apical 5 astir apt tokens, prime the 1 pinch the apical SynthID score”6. The watermark tin past beryllium detected by calculating the aggregate SynthID people of a artifact of text. If it’s suspiciously high, it’s very apt to person been AI-generated.
This is fundamentally a type of the communal proposal that you tin place LLMs by usage of the em-dash, isolated from that alternatively of a database of keywords, it relies connected subtle mathematical relationships betwixt words that humans can’t identify. Because the process for assigning the people is trivial, it’s very inexpensive to tally watermark detection.
Unicode watermarks via homoglyphs
Google person a analyzable mathematical rationale for why SynthID doesn’t make the exemplary dumber: supposedly the SynthID scoring is random capable to enactment for illustration a normal pseudo-random token sampler, conscionable 1 that leaves a detectable fingerprint connected the outputs. But of people this is suspicious. For instance, it’s communal to do conclusion mounting somesthesia to zero, which ever picks the model’s astir apt adjacent token. In that case, you can’t time off a fingerprint astatine each (or you person to disregard the user’s penchant and prime the 2nd aliases 3rd prime anyway).
If you can’t change the exemplary outputs, tin you still fingerprint the content? Well, benignant of. I’m beautiful judge OpenAI and Anthropic are sometimes applying fancy Unicode tricks. For instance, you mightiness spell done and switch your normal ” ” spaces (unicode U+0020) pinch a three-per-em ” ” abstraction (unicode U+2004), aliases a CJK ideographic ” ” abstraction (unicode U+3000). These are called “homoglyphs”, and you tin find much of them here.
Of course, tons of human-generated matter uses homoglyphs. But it’s trivial to encode a pattern of homoglyphs (say, “every 3rd abstraction becomes a three-per-em”) that is overmuch little apt to hap successful the wild. Like the SynthID watermark, a homoglyph-based watermark tin beryllium detected very cheaply. A homoglyph-based watermark is cheaper to use than SynthID: you could moreover do it wholly connected the client.
I don’t deliberation this is simply a conspiracy theory. Claude Code was definitely doing this to tag suspicious requests from Chinese users (exploiting homoglyphs for the ’ characteristic successful “Today’s date”, though they’ve since walked that back). In the past fewer years, I’ve noticed that erstwhile I transcript blocks of matter from ChatGPT and paste them into VSCode, sometimes VSCode marks immoderate aliases each of the spaces arsenic different Unicode characters7. Are OpenAI and Anthropic utilizing homoglyphs arsenic an AI-generated watermark? I’m not sure. But they’re decidedly utilizing homoglyphs.
Text watermarks tin beryllium trivially removed
The AI Act (specifically, its associated Code of Practice) requires watermarking to beryllium “embedded wrong the contented successful a mode that is difficult for it to beryllium separated from the content”. However, matter watermarks tin beryllium trivially removed.
To region unicode homoglyph watermarking, you simply person to switch each the homoglyphs pinch their “real” characteristic equivalents. If you person entree to moreover a comparatively anemic un-watermarked LLM8, you tin portion retired SynthID watermarking by asking that LLM to paraphrase the matter content. Because the watermark is inherent to subtle vocabulary choices, re-wording the contented will region the watermark. You could moreover do it by hand, though astatine that constituent it’s not really AI-generated contented anymore. Since location will beryllium immoderate benignant of free nationalist watermark testing tool, you tin conscionable support tweaking until it comes backmost negative.
Moreover, the AI Act requires watermarking techniques to beryllium “interoperable… arsenic acold arsenic this is technically feasible”. That intends AI providers would person to people their watermarking process, and perchance moreover effort to standardize connected applying the aforesaid benignant of watermarks. I conscionable don’t spot really this is compatible pinch the benignant of security-by-obscurity that LLM matter watermarking depends on. Unlike image and video watermarks, matter watermarks will ever beryllium trivial to remove.
What astir C2PA?
The AI Act and Code of Practice talk a batch astir “digitally signed metadata”. The thought present is that you tin see an AI disclosure successful the file’s metadata itself, ideally successful a measurement that cannot beryllium tampered pinch (for instance, by signing a hash of the file’s contents). This signed-metadata process is fundamentally C2PA Content Credentials. While you tin region C2PA metadata, you (theoretically) can’t fake it, truthful a record pinch “created by a human” metadata tin beryllium trusted, and files pinch nary metadata astatine each tin beryllium held successful suspicion.
This station is already excessively agelong to get into what I deliberation astir C2PA, but I do want to opportunity that C2PA is not a substitute for matter watermarking. It only really applies to files. In the words of the Code of Practice, that’s “a information format that supports attaching metadata (e.g., an audio, image, video, aliases containerised text)“. The output of chat devices (and astir of the output of AI agents) is not containerized text, but plain aged regular text, and truthful can’t beryllium signed. What would it moreover look for illustration to motion ChatGPT outputs? There’s nary artifact to walk around.
I deliberation it’s a fascinating mobility whether Claude Code has to C2PA-sign immoderate HTML files aliases PDFs it generates for you. That seems benignant of tricky to get right. But successful immoderate case, the AI Act besides mandates immoderate benignant of existent watermarking arsenic well.
Conclusion
So what’s going to hap this year? If I had to guess, I’d opportunity that each AI supplier (not conscionable labs for illustration OpenAI aliases Anthropic, but third-party providers for illustration Fireworks aliases Groq) will instrumentality a SynthID token sampler successful beforehand of their conclusion stacks. This mightiness beryllium constricted to users successful the EU, but it mightiness not be, since SynthID is astatine slightest arsenic bully arsenic a normal top-k token sampling approach.
AI providers will past connection a “check for watermark” page that re-tokenizes user-provided text, runs the scoring, and checks whether it’s supra a definite threshold. Depending connected really earnestly the interoperability clause is taken, providers mightiness moreover standardize connected the aforesaid SynthID setup, successful which lawsuit location could beryllium a azygous EU-hosted “watermark this text” page.
I don’t deliberation unicode-based watermarking is going to beryllium considered compliant pinch the AI Act, but immoderate providers which don’t want to group up SynthID mightiness effort it. Either way, method users will beryllium capable to portion retired the watermark astatine will, and location will beryllium a plethora of devices that non-technical users will usage for this purpose.
If you liked this post, see subscribing to email updates astir my caller posts, aliases sharing it connected Hacker News.
Here's a preview of a related station that shares tags pinch this one.
AI conclusion is evidently profitable
Many group claim that AI conclusion is unprofitable to serve, and frankincense must beryllium subsidized by an water of dumb money from investors who judge that immoderate early AI exemplary will travel to predominate the world economy. When that dumb money goes away, truthful will AI products. According to this view, LLMs are conscionable inherently excessively costly (in position of money, power, and water) to beryllium utilized successful user products. In fact, they tin only beryllium utilized coming by externalizing the costs: money onto VC costs and now unit ETF investors, powerfulness onto electrical inferior consumers, and h2o onto the communities wherever datacenters are built.
Continue reading...
English (US) · 
Indonesian (ID) ·