Internet Archive will ignore robots.txt files to keep historical record accurate

internet archive robots txt server
Internet Archive
The Internet Archive has announced that going forward, it will no longer conform to directives given by robots.txt files. These files are predominantly used to advise search engines on which portions of the page should be crawled and indexed to help facilitate search queries.

In the past, the Internet Archive has complied with instructions laid out by robots.txt files, according to a report from Boing Boing. However, it has been decided that the way that these files are calibrated is often at odds with the service that the site sets out to provide.

“Over time we have observed that the robots.txt files that are geared toward search engine crawlers do not necessarily serve our archival purposes,” stated a blog post that the organization published last week. “Internet Archive’s goal is to create complete ‘snapshots’ of web pages, including the duplicate content and the large versions of files.”

Robots.txt files are increasingly being used to remove entire domains from search engines following their transition from a live, accessible site to a parked domain. If a site goes out of business, and is rendered inaccessible in this way, it also becomes unavailable for viewing via the Internet Archive’s Wayback Machine. The organization apparently receives queries about these sites on a daily basis.

The Internet Archive hopes that disregarding robots.txt files will help contribute to an accurate representation of prior points in the web’s history, removing their capacity to muddy the waters with instructions intended for search engines.

The organization has already ceased referring to robots.txt files on sites and pages related to the U.S. government and the U.S. military, to account for the enormous changes that can be made to domains between one administration and the next. This decision has caused no major problems, so there are high hopes that discontinuing the use of the files more broadly will be helpful.

Smart Home

Airbnb hosts are offering free rooms to those displaced by California wildfires

Several thousand Airbnb hosts in California are opening their homes to help those displaced by the devastating wildfires in the state. Free accommodation is being offered to those affected through November 29.
Computing

These are the 5 best free antivirus apps to protect your MacBook

Malware protection is more important than ever, even if you eschew Windows in favor of Apple's desktop platform. Thankfully, protecting your machine is as easy as picking from the best free antivirus apps for Mac suites.
News

Zoom in on famous works through the Art Institute of Chicago’s new website

Art lovers, listen up. The Art Institute of Chicago has given its website a serious makeover with new features that let you get up close and personal with more than 50,000 artworks by famous (and not so famous) artists.
Computing

Will Chrome remain our favorite web browser with the arrival of newest version?

Choosing a web browser for surfing the web can be tough with all the great options available. Here we pit the latest versions of Chrome, Opera, Firefox, Edge, and Vivaldi against one another to find the best browsers for most users.
Computing

How to easily record your laptop screen with apps you already have

Learning how to record your computer screen shouldn't be a challenge. Lucky for you, our comprehensive guide lays out how to do so using a host of methods, including both free and premium utilities, in both MacOS and Windows 10.
Gaming

Want to gift a Steam game so you can play with a friend? Here's how to do it

The holidays may have passed, but it's always a good time to give the gift of gaming (especially when there's a Steam sale)! Here's our quick guide on how to give a Steam game as a gift.
Emerging Tech

Microsoft’s friendly new A.I wants to figure out what you want — before you ask

Move over Siri and Alexa! Microsoft wants to build a new type of virtual assistant that wants to be your friend. Already making waves in Asia, could this be the future of A.I. BFFs?
Computing

Multi-monitor issues? Here's how to resolve them

If you're running into multi-monitor problems, you're not alone. Two screens are very useful, but they can present some difficulties. Here are some common multi-monitor problems and how to fix them.
Computing

Capture screenshots with print screen and a few alternative methods

Capturing a screenshot of your desktop is easier than you might think, but it's the kind of thing you'll probably need to know. Here's how to perform the important function in just a few, easy steps.
Emerging Tech

Awesome Tech You Can’t Buy Yet: A.I. selfie drones, ‘invisible’ wireless chargers

Check out our roundup of the best new crowdfunding projects and product announcements that hit the web this week. You may not be able to buy this stuff yet, but it sure is fun to gawk!
Deals

The best MacBook deals for November 2018

If you’re in the market for a new Apple laptop, let us make your work a little easier: We hunted down the best up-to-date MacBook deals available online right now from various retailers.
Deals

Cyber Monday 2018: When it takes place and where to find the best deals

Cyber Monday is still a ways off, but it's never too early to start planning ahead. With so many different deals to choose from during one of the biggest shopping holidays of the year, going in with a little know-how makes all the…
Computing

All the best deals on Surface products for Black Friday

A number of retailers are discounting Surface devices for Black Friday. Be it the Surface Pro 2017, Surface Pro 6, or the Surface Go, here's a look at how (and where) you can save big on Surface this holiday season.
Smart Home

All the best Amazon Black Friday deals for 2018

Amazon may be an online-only retailer, but that doesn’t mean its Black Friday sales are anything to sniff at. In fact, due to its online status, Amazon has huge flexibility with the range of products and deals it can offer. Here's our…