Skip to main content

Internet Archive will ignore robots.txt files to keep historical record accurate

The Internet Archive has announced that going forward, it will no longer conform to directives given by robots.txt files. These files are predominantly used to advise search engines on which portions of the page should be crawled and indexed to help facilitate search queries.

In the past, the Internet Archive has complied with instructions laid out by robots.txt files, according to a report from Boing Boing. However, it has been decided that the way that these files are calibrated is often at odds with the service that the site sets out to provide.

Recommended Videos

“Over time we have observed that the robots.txt files that are geared toward search engine crawlers do not necessarily serve our archival purposes,” stated a blog post that the organization published last week. “Internet Archive’s goal is to create complete ‘snapshots’ of web pages, including the duplicate content and the large versions of files.”

Robots.txt files are increasingly being used to remove entire domains from search engines following their transition from a live, accessible site to a parked domain. If a site goes out of business, and is rendered inaccessible in this way, it also becomes unavailable for viewing via the Internet Archive’s Wayback Machine. The organization apparently receives queries about these sites on a daily basis.

The Internet Archive hopes that disregarding robots.txt files will help contribute to an accurate representation of prior points in the web’s history, removing their capacity to muddy the waters with instructions intended for search engines.

The organization has already ceased referring to robots.txt files on sites and pages related to the U.S. government and the U.S. military, to account for the enormous changes that can be made to domains between one administration and the next. This decision has caused no major problems, so there are high hopes that discontinuing the use of the files more broadly will be helpful.

Brad Jones
Former Digital Trends Contributor
Brad is an English-born writer currently splitting his time between Edinburgh and Pennsylvania. You can find him on Twitter…
Apple is going to make it easier to read on your Mac
Magnifier being used to read the Odyssey.

In honor of Global Accessibility Awareness Day, Apple has announced a litany of new features designed to make its ecosystem more user-friendly. Among these are Accessibility Nutrition Labels in the App Store, Magnifier for Mac, Braille Access, and Accessibility Reader. There are also updates coming to Live listen, visionOS, Personal Voice, and others.

The Accessibility Nutrition Labels provide more detailed information about apps and games in the store, highlighting any accessibility features they offer. This will make it easier for users who need these features to identify games and apps that will assist them. A few of the accessibility features include VoiceOver, Voice Control, Larger Text, and more.

Read more
Looking for a $1,000 gaming laptop? Check out this Gigabyte G6 deal
The Gigabyte G6 gaming laptop on a white background.

Looking for the best gaming laptop deals that you can buy for $1,000? You may want to set your sights on the Gigabyte G6. Its configuration with the Nvidia GeForce RTX 4060 graphics card, which is originally sold for $1,200, is currently available from Best Buy with a $200 discount. We're not sure how much time is remaining on this offer though, so if you're interested, you're going to have to be quick with your purchase and proceed with the transaction right now.

Why you should buy the Gigabyte G6 gaming laptop

Read more
The powerful HP Omen Max 16 gaming laptop with RTX 5070 Ti is $350 off today
The HP Omen Max gaming laptop with Valorant on the screen.

Gamers who want an upgrade should be on the lookout for discounts from gaming laptop deals, as these devices can get pretty expensive. You should check out the HP Omen Max 16, as its configuration with the Nvidia GeForce RTX 5070 Ti graphics card is on sale from HP itself for $2,150 instead of $2,500. That's $350 in savings that you can spend on video games and other accessories, but you're going to have to hurry with your purchase if you want to buy the gaming laptop at 16% off as the offer may be already gone by tomorrow.

Why you should buy the HP Omen Max 16 gaming laptop

Read more