Smells like AI spirit

Have you ever encountered a line of text or a passage that felt oddly recognizable, like ChatGPT or Claude had written it? Maybe it was a LinkedIn post from a colleague, or maybe even the caption below an Instagram post. You're right to be suspicious. AI adoption has grown rapidly to gain a meaningful share of social media content being generated today. But is that really all that bad?

This week, Anthropic published a document explaining how Claude marks the text it generates. This clarification was triggered by Article 50 of the EU AI Act, which became applicable on 2nd August 2026. The Act requires AI providers to ensure that generative output, including AI-generated text, is marked with machine-readable properties so that it can be detected. How would that work? Anthropic describes two techniques:

• A signed C2PA metadata. This is specifically for filetypes where it is easy to encode additional metadata such as images, videos, etc. A boring cryptographic signature.
• A watermark in text. An exciting new proposition.

Watermarks in AI-generated text

Historically, the technique to encode additional data inside plain text has been to leverage Unicode variation selectors - invisible characters silently inserted within the passage at appropriate positions. But it is a weak solution that can be defeated easily. These hidden characters are easy to detect and sanitize. If you edit one word, some of these characters may be lost. You can retype the passage. You can take a screenshot and pass it through OCR (Optical Character Recognition) to convert the picture back to text. There are many failure modes. So a better approach of encoding a watermark in AI-generated text is to hide it in the choice of words. A statistical mark.

Let's say a Large Language Model (LLM) is generating the text "The city was _". At this point, the LLM has a decision to make. It has at its disposal a probability distribution over the words that could come next. And it samples from it. This is where the watermark comes in.

The watermark algorithm takes the last couple of words, hashes them with a secret cryptographic key to split the set of possible words into two buckets - let's call them red and green sets. Now, it multiplies the probability of every word in the green set by a fixed factor. It's only a gentle nudge towards a preference, so the LLM does not end up choosing a word from the green set every time. But over a long enough passage of AI-generated text, words from the green set will appear more often.

Detection of AI-generated text now becomes a counting task. Anyone with the secret cryptographic key (say, Anthropic's trusted partners) can compute the green set of words at each position, and tally the hits. A passage of text watermarked by AI will therefore have a high hit count of words from the green set, well beyond what you'd expect would occur by chance alone. And the longer the passage, the stronger the confidence of this watermarking detection.

But this only works when the LLM is genuinely undecided.

"Text watermarks generally cannot be embedded or detected reliably when the text has low entropy... It would be hard to distinguish whether the canonical continuation of '1 + 1 =' was generated by GAI (Generative AI). When there are more options for high-quality text, the watermark can be more easily embedded and detected..." - NIST, Reducing Risks Posed by Synthetic Content (2024).

For example, when it comes to "The capital of France is _" the model's probability distribution will put Paris at a high 99 point something percent. Even after multiplying the probability of every choice from the green set by a factor, Paris still wins (assuming Paris did not get to be in the privileged green set). In this case, the position carries no watermark signal. Compare that to the case "The city was _" where plenty of words are suitable options with similar probabilities. There is now an opportunity for the watermark algorithm to tip the scales of probability and therefore encode something. This means that in cases where there is only one right answer - facts, direct quotations, etc - the watermarking algorithm is ineffective. But across a long enough passage of text without fixed answers, the many instances where the model got to chose a word from the green set, can come together to become a high confidence indicator of AI-generated text.

Another obvious limitation was pointed out by NIST.

"... higher entropy also allows paraphrasing."

Passages of text with many plausible next-word choices are also typically the ones that are easiest to paraphrase - restating it with your own words while keeping the same meaning. That's a bummer. The same phrases with many possible word choices that encode watermarking strongly, are also the ones where paraphrasing is most effective to remove it.

That adds to the growing list of limitations in text watermarking. But here is a fun one demonstrated by Riley Goodside (who was Scale AI's first Staff Prompt Engineer). Ask the model to emit an emoji (or any junk token) after every word as it generates. Then delete the emojis from the output. Now you have clean AI-generated text with no watermarks.

There is also an obvious tradeoff between the effectiveness of watermarking and model performance. The watermarking algorithm is effectively messing around with the probabilities of words that the model should emit next. So if you crank the wheels and boost the factors by which it does so, you get a strong watermark signal, but at the cost of output quality. Research published by Google on SynthID (it's own watermarking mechanism) has noted 0.57% increase in latency, and a difference of 0.01 - 0.02% in user-rated quality feedback from roughly 20 million live Gemini responses - deeming this concern statistically insignificant.

An alternative approach - Pangram

Pangram is a startup from Brooklyn NY that recently raised $9 million funding, has come up with an alternative approach to detection of AI-generated text.

They built a classifier. It needs no keys or special access to any model. It reads text and infers whether a machine wrote it. Their central technique is synthetic mirroring, i.e., for every human writing sample, generate an equivalent AI sample (same topic, same format), and then make the classifier learn _how_ they differ. Their reported false positive rate is 0.0041%, or roughly 1 in 24,000. Surely, that sounds promising?

Arvind Narayanan - Professor of Computer Science at Princeton university and the co-author of the book AI Snake Oil - explains this best in his LinkedIn post. He says, a typical student may submit somewhere between 500 to 1,000 pieces of work across a degree. So if you do the math, about 5-10% of students may get falsely accused at least once.

Alex Danco from a16z published an essay this week called "This essay is 10% AI-generated", in which, he claims Pangram scored several of his blog posts from 2019 and 2020 at 70-75% AI-generated (and consequently goes on to sarcastically gloat that AI may have been trained preferentially on his writing style).

Unlike watermarks in text, a classifier can not verify provenance from built-in signals. Therefore it tends to find false positives in text that was written before LLMs became mainstream, or people who have somehow internalized writing like LLMs (which is likely for AI-pilled engineers like me who are talking to AI chat interfaces all day all night). Pangram's own documentation concedes that it:

"... does not account for people who intentionally write like LLMs or who have absorbed elements of LLM style into their writing." - Pangram 4 Technical Report (Page 25, last few lines)

That was funny. Why would someone intentionally write like LLMs?

Who wrote the Odyssey

Did you have a chance to watch Nolan's epic portrayal of The Odyssey, written by the ancient Greek poet Homer, at the biggest IMAX screen near you? I hope you did. But did you know that there is unresolved scholarly debate over whether Homer was actually a single historical person, or a group of authors? Stylometric and linguistic studies have supported that there may have been a single mastermind author, or maybe a close-knit guild of authors residing during the same era. Or maybe Homer was never a person at all, just a label attached to a bundle of stylistically similar pieces of work. We don't really know. But it sure does not make The Odyssey any less of an epic.

I think the deeper issue is not whether the words are AI-generated. To me, it does not matter whether the words came out of a machine, as long as behind those words there is authentic human thinking.

When I see people calling out passages of text that sound like AI, I think what they are really complaining about is that they fear the author is being inauthentic. These people know how little it takes to have AI generate paragraph after paragraph of meaningless slop disguised as prose. So they fear the author is being lazy. Such accusations may not always hold true. AI has opened the doors to many non-native English speakers to express themselves, albeit in words chosen by AI models. So I try to hold my judgement when I read something that "smells" like AI. But please don't share AI-generated videos of cats with fingers dancing to K-Pop. You're better than that.

Stylometry for AI-generated text is uniquely interesting, and next week I will delve (iykyk) into the top features with examples so you can start sniffing out AI-generated prose. Until then, park this at an accessible corner of your human brain: The next time you encounter AI-generated text on social media, think twice about what's actually bothering you.