Skip to main content

Internet Archive will ignore robots.txt files to keep historical record accurate

internet archive robots txt server
Internet Archive
The Internet Archive has announced that going forward, it will no longer conform to directives given by robots.txt files. These files are predominantly used to advise search engines on which portions of the page should be crawled and indexed to help facilitate search queries.

In the past, the Internet Archive has complied with instructions laid out by robots.txt files, according to a report from Boing Boing. However, it has been decided that the way that these files are calibrated is often at odds with the service that the site sets out to provide.

“Over time we have observed that the robots.txt files that are geared toward search engine crawlers do not necessarily serve our archival purposes,” stated a blog post that the organization published last week. “Internet Archive’s goal is to create complete ‘snapshots’ of web pages, including the duplicate content and the large versions of files.”

Robots.txt files are increasingly being used to remove entire domains from search engines following their transition from a live, accessible site to a parked domain. If a site goes out of business, and is rendered inaccessible in this way, it also becomes unavailable for viewing via the Internet Archive’s Wayback Machine. The organization apparently receives queries about these sites on a daily basis.

The Internet Archive hopes that disregarding robots.txt files will help contribute to an accurate representation of prior points in the web’s history, removing their capacity to muddy the waters with instructions intended for search engines.

The organization has already ceased referring to robots.txt files on sites and pages related to the U.S. government and the U.S. military, to account for the enormous changes that can be made to domains between one administration and the next. This decision has caused no major problems, so there are high hopes that discontinuing the use of the files more broadly will be helpful.

Brad Jones
Former Digital Trends Contributor
Brad is an English-born writer currently splitting his time between Edinburgh and Pennsylvania. You can find him on Twitter…
Save $400 on this Dell PC with an RTX 4070, 32GB of RAM, 1TB SSD
The Dell XPS desktop on a table.

If you're willing to spend on desktop computer deals for a machine that will offer flagship performance for all of your needs, it's highly recommended that you go with the Dell XPS Desktop 8960. This certain configuration that's on sale from Dell will let you enjoy savings of $400 on its original price of $2,250, so you'll have to pay $1,850 -- it's still not cheap, but it's going to be worth every single penny at this discounted price. You're going to have to be quick though -- this is a clearance sale, so there's no telling when the offer expires and when stocks will run out.

Why you should buy the Dell XPS Desktop 8960
The Dell XPS Desktop 8960 isn't just an excellent PC -- it's our top pick among the best desktop computers for its flexibility to accommodate almost any budget and purpose. This particular model of the PC features the 13th-generation Intel Core i9 processor, the Nvidia GeForce RTX 4070 graphics card, and 32GB of RAM, which will make it more than enough to tackle even the most demanding tasks, while also offering high-end gaming potential to play the best PC games for whenever you need to take a break from work.

Read more
Best color laser printers for 2024: tested and reviewed
A Brother printer on a counter in front of a brick wall.

The best color laser printers can be a great investment, saving you quite a bit of time and money. For shoppers worried about the long-term ink costs, you'll find color laser printers surprisingly affordable. Laser printers use toner, which lasts a very long time, delivering a low cost per page for monochrome documents and fast color prints. The best color laser printers offer quick performance and reliability to help keep your home office or small business productive.

If you need to scan documents for record-keeping and photo capture or want the convenience of a color copier, an all-in-one color laser printer is an essential tool for your small business or personal use. For a small added cost, you get expanded capabilities. That's why every model on this list is an all-in-one from the best printer brands.

Read more
The 5 best Wi-Fi adapters for PC in 2024
The Ugreen AC1300 Wi-Fi adapter in a desktop PC.

Whether you're designing it yourself or getting a pre-built PC, it can be easy to get a computer and realize that it doesn't have a native Wi-Fi adapter. Or, maybe it does, but you're internet speeds are getting faster, game downloads are getting bigger, you've already upgraded your router and need an adapter to match your newfound power requirements. No matter the situation, an external Wi-Fi adapter that you can add to your PC setup or even laptop setup will be worth your time. Here, we investigate the best Wi-Fi adapters for PC use. Most are incredibly affordable and just snap into a free USB port and start working.
The best Wi-Fi adapter for PC in 2024

Buy the

Read more