{"id":15524,"date":"2025-04-18T06:47:36","date_gmt":"2025-04-18T06:47:36","guid":{"rendered":"https:\/\/fusionmindlabs.com\/blogs\/wikipedia-offers-dataset-to-thwart-scraper-bots\/"},"modified":"2025-04-18T06:47:36","modified_gmt":"2025-04-18T06:47:36","slug":"wikipedia-offers-dataset-to-thwart-scraper-bots","status":"publish","type":"post","link":"https:\/\/fusionmindlabs.com\/blogs\/wikipedia-offers-dataset-to-thwart-scraper-bots\/","title":{"rendered":"Wikipedia offers dataset to thwart scraper bots"},"content":{"rendered":"\n<\/p>\n<div>\n<h2>Wikipedia&#8217;s Struggle with AI Crawlers<\/h2>\n<p>Wikipedia has been facing challenges due to the impact of <a data-i13n=\"cpos:1;pos:1\" href=\"https:\/\/www.engadget.com\/ai\/wikipedia-is-struggling-with-voracious-ai-bot-crawlers-121546854.html?_fsig=Wr5Dq_GeIVF_s2qPwjs2Ig--%7EA\" data-ylk=\"slk:struggling;cpos:1;pos:1;elm:context_link;itc:0;sec:content-canvas\" class=\"link \" rel=\"nofollow noopener\" target=\"_blank\"><\/a> AI crawlers \u2014 bots that scrape text and multimedia from the encyclopedia to train generative artificial intelligence models \u2014 on its servers, resulting in increased costs and slower load times for human users in some cases. To mitigate this issue, the Wikimedia Foundation, which manages Wikipedia&#8217;s data, is offering AI developers a dataset they can freely use, potentially reducing the strain on the public Wikipedia website and bandwidth consumption.<\/p>\n<p><h3>Collaboration with Kaggle<\/h3>\n<p>The organization has partnered with Kaggle, a data science platform, to release a beta version of a structured dataset in both English and French. <a data-i13n=\"cpos:3;pos:1\" href=\"https:\/\/blog.google\/technology\/developers\/kaggle-wikimedia\/\" rel=\"nofollow noopener\" target=\"_blank\" data-ylk=\"slk:According to Google;cpos:3;pos:1;elm:context_link;itc:0;sec:content-canvas\" class=\"link \"><\/a> \u2014 which owns Kaggle \u2014 the dataset is formatted for machine learning, making it more useful for training, development, and data science applications.<\/p>\n<p><h3>Dataset Details<\/h3>\n<p>Wikimedia Enterprise <a data-i13n=\"cpos:4;pos:1\" href=\"https:\/\/enterprise.wikimedia.com\/blog\/kaggle-dataset\/\" rel=\"nofollow noopener\" target=\"_blank\" data-ylk=\"slk:notes;cpos:4;pos:1;elm:context_link;itc:0;sec:content-canvas\" class=\"link \"><\/a> notes that the dataset includes abstracts, short descriptions, infobox-style key-value data, image links, and clearly segmented article sections. However, it excludes references and other non-prose elements, such as video clips. While the lack of references may raise concerns about attribution, Wikimedia Enterprise assures that the content in the dataset is freely licensed under Creative Commons, public domain, and other licenses, as it is sourced from Wikipedia.<\/p>\n<\/div>\n\n<br \/><a href=\"https:\/\/www.engadget.com\/ai\/wikipedia-offers-ai-developers-a-training-dataset-to-maybe-get-scraper-bots-off-its-back-143255593.html?src=rss\" rel=\"nofollow noopener\" target=\"_blank\">Source Link<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Wikipedia&#8217;s Struggle with AI Crawlers Wikipedia has been facing challenges due to the impact of AI crawlers \u2014 bots that scrape text and multimedia from the encyclopedia to train generative&#8230;<\/p>\n","protected":false},"author":0,"featured_media":15525,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[47],"tags":[64,103,273,63,2932],"class_list":["post-15524","post","type-post","status-publish","format-standard","has-post-thumbnail","category-news","tag-gear","tag-google","tag-internet","tag-news","tag-wikipedia"],"jetpack_featured_media_url":"https:\/\/fusionmindlabs.com\/blogs\/wp-content\/uploads\/sites\/9\/2025\/04\/b0bc0890-1b97-11f0-aefb-32dc82e8871a.jpeg","_links":{"self":[{"href":"https:\/\/fusionmindlabs.com\/blogs\/wp-json\/wp\/v2\/posts\/15524","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fusionmindlabs.com\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fusionmindlabs.com\/blogs\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fusionmindlabs.com\/blogs\/wp-json\/wp\/v2\/comments?post=15524"}],"version-history":[{"count":0,"href":"https:\/\/fusionmindlabs.com\/blogs\/wp-json\/wp\/v2\/posts\/15524\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/fusionmindlabs.com\/blogs\/wp-json\/wp\/v2\/media\/15525"}],"wp:attachment":[{"href":"https:\/\/fusionmindlabs.com\/blogs\/wp-json\/wp\/v2\/media?parent=15524"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fusionmindlabs.com\/blogs\/wp-json\/wp\/v2\/categories?post=15524"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fusionmindlabs.com\/blogs\/wp-json\/wp\/v2\/tags?post=15524"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}