How to check if your site is blocking AI crawlers (robots.txt won't tell you)

Related stories

• The SaaSpocalypse that wasn’t, with Atlassian CEO Mike Cannon-Brookestheverge.com
  The Verge · 0 points · 3 days ago
• Can you forget how you feel about Meta?theverge.com
  The Verge · 0 points · 9 days ago
• Can Cloudflare CEO Matthew Prince save the web from AI?theverge.com
  The Verge · 0 points · 5 days ago
• Can John Ternus find Apple’s next big thing?theverge.com
  The Verge · 0 points · 10 days ago
• Show HN: Training a model to identify AI web content from structure alonearxiv.org
  Hacker News · 73 points · 9 days ago
• Does AI need an antitrust exemption so it doesn’t kill everyone????theverge.com
  The Verge · 0 points · 12 days ago

Related topics