IEEE Spectrum tests point to limits for OpenAI’s textGrain watermark in the EU

Tests reported by IEEE Spectrum indicate that OpenAI’s textGrain watermark may not survive additional editing and performs worse on texts under 150 words. On questions from the ELI5 database, the detector identified watermarks in about 80% of texts of roughly 150 words, compared with 95% in texts of about 300 words. According to OpenAI, replacing 10% of words with synonyms reduced detection from 92% to 66%; replacing up to 25% cut the rate to 17%.

The transparency obligations in Article 50 of the European Union’s AI Act took effect on 2 August and require synthetic text, images, video and audio to be marked in a machine-readable format. There is still no clear legal benchmark for measuring watermark effectiveness. OpenAI says textGrain is meant to complement other provenance tools, rather than determine on its own whether content was generated by AI.