Policy

Meta Removes Web-Scraping Language From Terms After Legal Pushback

Meta has excised controversial contract language that experts argued permitted unrestricted AI training on news site content, following criticism from legal scholars and publishers.

Last verified:

Meta Removes Controversial Web-Scraping Clause

Meta has deleted language from its commercial terms of service that legal experts and publishers interpreted as authorizing the company to scrape news sites and other web properties for artificial intelligence model training without restriction. According to The Verge AI, the removed clause permitted Meta to use “any content from such domains, web pages, or URLs” to develop and maintain its products—a formulation that critics worried would override robots.txt files that publishers use to signal crawling preferences.

James Grimmelmann, a professor of digital and information law at Cornell University, and outlets including Nieman Lab flagged the problematic phrasing as a potential mechanism for Meta to harvest publisher content without consent or compensation. The language suggested Meta could supersede existing web-blocking standards to feed its AI systems training data scraped from news and other protected sites. The interpretation gained traction after the clause appeared in Meta’s publicly available commercial terms, prompting pushback from the publishing industry and legal scholars.

Meta’s Clarification and Course Correction

According to The Verge AI, Meta’s vice president of communications Andy Stone initially pushed back against criticism, stating there was “no change to how Meta uses web crawlers.” However, the company subsequently revised its position. Daniel Roberts, Meta’s spokesperson, issued a more detailed statement clarifying that the consolidated crawling terms applied only to ads and business products, not general AI training. Roberts added that Meta remains bound by robots.txt preferences and that “these updates don’t give us new rights that supersede those preferences.”

Industry Context and Broader Implications

The reversal underscores ongoing tension between AI companies’ data appetite and publishers’ intellectual property rights. News organizations have grown increasingly vocal about the use of their content to train large language models without licensing agreements or compensation. Meta’s decision to remove the ambiguous language sidesteps potential litigation but leaves unresolved the deeper question of whether AI training on web-scraped content requires explicit permission or constitutes fair use.

Why This Matters

For publishers and legal teams evaluating AI companies’ data practices, this update signals that contractual ambiguity around web crawling will face heightened scrutiny. Organizations relying on robots.txt as their primary defense mechanism can use this clarification as evidence that at least one major AI training operation respects such signals. However, the change does not resolve whether Meta’s existing training datasets include news content harvested before the policy shift, nor does it establish industry-wide norms for AI training on protected content. Legal battles over AI training data remain likely as courts have yet to establish definitive precedent on fair use in this context.

Frequently Asked Questions

What specific language did Meta remove from its terms?

A clause stating Meta 'may use any content from such domains, web pages, or URLs' to train and maintain Meta products. Legal experts interpreted this as permission to scrape news sites for AI training without explicit consent.

Can publishers still block Meta's crawlers?

Yes. Meta clarified that robots.txt preferences remain enforceable. The removed language did not grant Meta the right to override those blocking instructions.

Does this change how Meta trains its AI models?

According to Meta's statement, no. The company says the terms change only clarifies existing policy, not new capabilities. The language was tied to ads and business products, not general AI training practices.

Who raised concerns about the original language?

Legal scholars including Cornell professor James Grimmelmann and outlets like Nieman Lab flagged the language as potentially permitting unrestricted AI training on copyrighted news content.

#meta #content-scraping #ai-training #copyright #terms-of-service