Skip to main content

Internet Archive will ignore robots.txt files to keep historical record accurate

internet archive robots txt server
Internet Archive
The Internet Archive has announced that going forward, it will no longer conform to directives given by robots.txt files. These files are predominantly used to advise search engines on which portions of the page should be crawled and indexed to help facilitate search queries.

In the past, the Internet Archive has complied with instructions laid out by robots.txt files, according to a report from Boing Boing. However, it has been decided that the way that these files are calibrated is often at odds with the service that the site sets out to provide.

“Over time we have observed that the robots.txt files that are geared toward search engine crawlers do not necessarily serve our archival purposes,” stated a blog post that the organization published last week. “Internet Archive’s goal is to create complete ‘snapshots’ of web pages, including the duplicate content and the large versions of files.”

Robots.txt files are increasingly being used to remove entire domains from search engines following their transition from a live, accessible site to a parked domain. If a site goes out of business, and is rendered inaccessible in this way, it also becomes unavailable for viewing via the Internet Archive’s Wayback Machine. The organization apparently receives queries about these sites on a daily basis.

The Internet Archive hopes that disregarding robots.txt files will help contribute to an accurate representation of prior points in the web’s history, removing their capacity to muddy the waters with instructions intended for search engines.

The organization has already ceased referring to robots.txt files on sites and pages related to the U.S. government and the U.S. military, to account for the enormous changes that can be made to domains between one administration and the next. This decision has caused no major problems, so there are high hopes that discontinuing the use of the files more broadly will be helpful.

Brad Jones
Brad is an English-born writer currently splitting his time between Edinburgh and Pennsylvania. You can find him on Twitter…
Quick! This MSI gaming laptop dropped from $1,500 to $1,000
The MSI Stealth 14-inch gaming laptop at a side angle.

One of the better gaming laptop deals comes via Best Buy with a huge $500 off the MSI Stealth 14-inch gaming laptop. It normally costs $1,500, so with the price down to $1,000, you’re getting a seriously good deal for the latest tech. If you’re keen to know more, keep reading while we take you through everything.

Why you should buy the MSI Stealth 14-inch gaming laptop
MSI is one of the best laptop brands around despite not being quite as big a name as some rivals. Its penchant is gaming hardware and that’s shown in the MSI Stealth 14-inch gaming laptop. It has an Intel Core i7-13620H processor along with 16GB of memory and 1TB of SSD storage, so it has all the key components you could need. Alongside that is an Nvidia GeForce RTX 4060 graphics card with 8GB of dedicated VRAM.

Read more
How to unjam a printer and stop it from happening again
Clearing a jam on the HP Smart Tank 5101 isn't easy.

There are few things as frustrating as a paper jam. Unfortunately, it's all too common for paper jams to happen when you rush to print labels for express shipments or need hard copies for a meeting. That's true on even the best printers.

A printer jam not only stops your progress, but also demands immediate attention because you won't be able to print until it's cleared. It could be a serious mistake to ignore the issue and try printing again since that could damage the printer.

Read more
This Lenovo ThinkPad is usually priced at $2,688 — today it’s $589
The Lenovo ThinkPad L14 Gen 3 laptop on a white background.

If you're looking at laptop deals for an offer that will get you a high-performance device for an affordable price, you should check out the Lenovo ThinkPad L14 Gen 3, which will be yours for $589. Lenovo says it's a $2,099 discount, but we're not sure about the laptop's original price of $2,688. Nevertheless, it's a powerful laptop for that cost, which should catch the attention of a lot of shoppers. If you're interested, you still have time to take advantage of this bargain, but you should hurry because we're not sure when it will end or when stocks will run out.

Why you should buy the Lenovo ThinkPad L14 Gen 3
Lenovo inherited the ThinkPad line from IBM, according to our explainer on the different Lenovo brands, which explains the iconic look, sturdy design, and focus on productivity of the Lenovo ThinkPad L14 Gen 3. The laptop will quickly prove that it's a reliable work-from-anywhere partner with its AMD Ryzen 5 Pro 5675U processor and integrated AMD Radeon Graphics, which are combined with 16GB of RAM that's on the level of top-tier machines, according to our guide on how much RAM do you need.

Read more