Academic libraries are fighting off swarms of data-scraping bots — that are slowing or crashing sites meant for students, researchers and the community — as AI companies continue hunting for satisfied to nourish their ample tongue models.
One academy says it was overwhelmed by additional than 11,000 searches on Finnish music in fair one day
Kevin Maimann · CBC News
· Posted: Nov 05, 2026 4:00 AM EST | Last Updated: November 5
Listen to this article
Estimated 6 minutes
The audio type of this part is generated by AI-based technology. Mispronunciations can occur. We are operating alongside our partners to continually assessment and enhance the results.
Academic libraries are fighting off swarms of data-scraping bots — that are slowing or crashing sites meant for students, researchers and the community — as AI companies continue hunting for satisfied to nourish their ample tongue models.
University libraries say complete collections are being knocked offline by persistent streams of unruly bots trying to admission their archives, which contain academic, historic and authorities documents.
"They're fair out there on the internet, fair bombarding us alongside trying to catch up as much matter as possible," Jill Strykowski, cataloguing and government documents guide at San Jose State University, told CBC News.
The bots are used by the two AI and third-party companies to scour websites and recover data, frequently to nourish LLMs akin Claude, Gemini and ChatGPT.
Libraries, museums and another traditional bequest institutions are premier targets since they have quality, vetted data that is mostly not hidden rearward paywalls or logins.
The bots are "causing an amazing amount of activity and headache for all institution that does digitization activity and all institution that is trying to create data openly available," Strykowski says.
Researchers found specified reports started appearing in delayed 2024 throughout North America, Europe and Oceania and experts inform CBC News the issue continues to get worse.
LISTEN | Even smarter, alongside equal small guardrails: The Sunday Magazine23:41Inside the earth competition toward an AI that power slay us all
And new challenges keep arising, says Dean Seeman, director of collections and digital infrastructure at University of Victoria Libraries.
"Sometimes the bot traffic gets so bad that it causes the sites to go downward entirely for a small while," he said.
Seeman says this creates a "real problem" for community access.
Regular group using archive websites are now, effectively, in competition alongside infinite traffic from bot farms, says Peter Lewis, Canada Research Chair in Trustworthy Artificial Intelligence at Ontario Tech University.
"This is another case of large, personal companies making use of common-good community involvement institutions in a way that's extremely extractive and fair assumes that that material is accessible and is liberated — and afterward using it to be capable to create extremely business models that afterward they can go and market rear to people," he said.
Workers act data from an old document belonging to the McMillan Memorial Library during a digitization exercise at the Eastland Library in Nairobi on March 24, 2022. Libraries and another traditional bequest institutions are premier targets for AI bots. (Patrick Meinhardt/AFP/Getty Images)
Good bots and bad bots
Eric Hellman, administrator director of Project Gutenberg — a U.S.-based unpaid institution that digitizes and archives traditional plant and distributes eBooks — wrote in a blog final year that these bots, which he characterizes as poorly programmed "monstrosities," are "destroying open access" on the internet.
He tells CBC News that "everything's gotten worse" since he wrote that.
Hellman says the issue comes downward to fine bots and bad bots.
He explains that hunt engines akin Google, and filing sites akin Internet Archive, use bots that prosecute robots.txt — voluntary sets of instructions on websites that inform them what they can and can't do, so they don't disrupt the site's functioning during gathering information.
AI abrasion bots, on the another hand, sometimes disregard these rules and overwhelm websites — in several cases requesting thousands of documents a second and downloading them in all accessible format.
WATCH | Young AI researchers in Ontario: This is what young group studying synthetic intellect are thinking — as the forthcoming of AI grabs headlines
As an example, the University of North Carolina at Chapel Hill reported final twelvemonth that it received additional than 11,000 searches for Finnish music from thousands of distinct net addresses on the identical day. The university stated its archive catalogue was periodically shutting out students, department and personnel because of bot traffic.
And since one business or bot ranch can activity from thousands of IP addresses, Hellman says the old fix of blocking problematic IPs is no longer useful.
Michael Weinberg, co-director of GLAM-E, a collaborative attempt focused on open-access policies for traditional institutions, is operating to defend community admission to information. (Submitted by Michael Weinberg)
He says Project Gutenberg is blessed for its materials to be used as AI training data and equal offers a one-click download of its complete archive of books, "but the bots are too foolish to use it."
"Our matter is nearly all community domain material. It's exactly the kind of matter that they have to be training on," he said. "We fair desire they would do it in a respectful way that doesn't obtain downward websites."
Open admission a 'huge benefit'
Some libraries have set up CAPTCHA challenges or another verification systems to weed out bots.
The issue is, those measures can additionally weed out the fine bots, says Michael Weinberg, co-director of GLAM-E, a collaborative academic initiative at NYU Law School and the University of Warwick focused on open-access policies for traditional institutions.
Some libraries have additionally begun to necessitate login credentials, a arguable move that those who said to CBC News say they desire to evade since it puts barriers on individual admission to things they accept have to be effortlessly accessible.
And as the abrasion bots get additional advanced, experts say they could additionally study to activity about these strategies.
The Anthropic logo is displayed at during the Dreamforce innovation acme in San Francisco, on Sept. 17. AI firms have broadly asserted that abrasion publically accessible data is permissible under fair use and in the public's finest interest. (Carlos Barria/Reuters)
Major AI firms have asserted that abrasion publically accessible data to train their models is permissible under fair use and in the public's finest interest.
But faced alongside general criticism and copyright lawsuits, Anthropic, OpenAI, Google and Microsoft expressed assistance final week for honouring a single, standardized opt-out indication in a website's robots.txt document that could obstacle their web crawlers from harvesting data.
Compliance would be voluntary, however, and the accord leaves out problematic bots from third-party companies and smaller AI startups.
Used bookstores navigate 'suspicious' bulk orders amid AI book-shredding fears
How a small Canadian publishing company is resisting the AI publish wave
AI is making it harder for voters to get exact information: Elections Canada chief
Weinberg says the digitization of precious data has been a "fantastic" betterment for humanity — giving researchers, teachers and inquisitive citizens uncomplicated admission to resources that, 20 years ago, "would have required driving for flying or a particular approval to get access."
Organizations akin GLAM-E are trying to fig out long-term solutions to maintain that admission for humans and helpful bots alike.
Right now, the bots create "a disincentive for libraries and another institutions to create parts of their gathering accessible online," Weinberg said. "I think it's been a huge advantage to community that libraries and museums and another traditional institutions have put a huge amount of activity into making their collections available."
People appearance at their laptops and in the chief study area of the New York Public Library on Oct. 5, 2016, in New York City. Increased bot traffic method regular group using archive websites are efficiently competing alongside infinite traffic from bot farms, says one expert. (Dominick Reuter/AFP/Getty Images)
ABOUT THE AUTHOR
Kevin Maimann is a elder author for CBC News based in Edmonton. He has covered a broad range of topics for publications including VICE, the Toronto Star, Xtra Magazine and the Edmonton Journal. You can attain Kevin by email at [email protected].
