Keeping AI crawlers away from paid WordPress content

AI companies crawl the web to train models and to answer questions with live pages. What they can take from a membership site depends on where the paywall runs.

The paywall is on the server

Many paywalls send the whole article and hide it with JavaScript or CSS. A crawler that doesn’t run the script, or simply reads the HTML, gets everything. RedeemID removes locked text on the server, before WordPress sends the page. Every visitor without access, crawler or person, gets the teaser and the wall, and the locked text isn’t in the HTML, the feeds, the REST API, search results, oEmbed or the sitemap. The plugin’s leak tests check each of those on every change.

So a crawler can read and quote your teaser, the same as a search engine. It can’t reach the rest without a member’s account.

What robots.txt adds

To keep AI crawlers off the public parts too, add their user agents to robots.txt (each company publishes its crawler names). Well-behaved crawlers respect it; others don’t, which is why the paywall itself doesn’t depend on it. RedeemID doesn’t block crawlers by user agent: anyone can fake one, and blocking by name would also catch the search crawlers you want.

Agents acting for a member

An AI agent signed in as a member, or using a member’s browser, reads what that member can read, because it is that member as far as the site can tell. RedeemID’s Abilities and MCP support is the supported way for agents to work with the site: staff abilities need an administrator, and member abilities only return the member’s own record.

What no plugin can promise

A member can still copy what they read. The paywall decides who gets the text; it doesn’t control what they do with it afterwards.

Written for: Publishers