Some websites are now blocking AI agents from crawling their sites

The internet has long operated on an assumption of openness: that public websites are fair game for automated systems to visit, read, and index. This foundation is now in question. Understanding what's happening requires learning how the web's foundational rules work, where the tension originates, and why stakes have risen. How the web was designed to be readable The modern web was built on a simple principle: computers should be able to talk to computers as easily as humans talk to websites. In the early days, search engines needed a way to automatically visit millions of pages, read them, and decide what content was relevant to user queries. To manage this, website owners adopted a common agreement: they could use a file called robots.txt to communicate with automated visitors about which parts of a site should be visited and which should be avoided. This system assumed good intent. A search engine crawler would identify itself honestly, follow the rules left by site owners, and index pages to make them more discoverable. The owner benefited from traffic; the search engine got content; the user got a useful directory of the web. It was a bargain built on transparency and mutual interest. The shift from indexing to training data For decades, the rules of web crawling remained relatively stable. Then the nature of what automated systems were doing began to change. Rather than indexing pages to make them searchable, companies started collecting vast amounts of online text, images, and code to train machine learning models. The purpose was no longer to help users find a website; it was to feed data into systems that would generate new content, summarize information, or perform tasks. This distinction matters because training a model feels different from indexing, even though the technical process of visiting a website and reading its contents can be identical. A search engine crawler makes a website more visible. A training crawler extracts patterns and knowledge from a website's text to build a product that might compete with, replace, or profit from that content without attribution or permission. The question of ownership and value Web content sits in an interesting legal and philosophical space. A published webpage is publicly accessible, which historically meant it was available for automated reading. But accessibility and ownership are not the same thing. Someone wrote that content, paid for hosting it, built an audience for it. The question that now divides the internet is whether automated collection and use of that work for commercial purposes requires explicit permission. Some website operators argue that their content has value, and that training commercial AI models on their work without permission or compensation amounts to taking. Others point out that humans read, share, quote, and learn from web content all the time without asking permission, and that automated reading should operate under similar rules. This debate touches