With Robots.txt, Websites Halt AI Companies' Web Crawlers - IEEE Spectrum

https://spectrum.ieee.org/web-crawling