Internet Archive will ignore robots.txt files to keep historical record accurate

internet archive robots txt server
Internet Archive
The Internet Archive has announced that going forward, it will no longer conform to directives given by robots.txt files. These files are predominantly used to advise search engines on which portions of the page should be crawled and indexed to help facilitate search queries.

In the past, the Internet Archive has complied with instructions laid out by robots.txt files, according to a report from Boing Boing. However, it has been decided that the way that these files are calibrated is often at odds with the service that the site sets out to provide.

“Over time we have observed that the robots.txt files that are geared toward search engine crawlers do not necessarily serve our archival purposes,” stated a blog post that the organization published last week. “Internet Archive’s goal is to create complete ‘snapshots’ of web pages, including the duplicate content and the large versions of files.”

Robots.txt files are increasingly being used to remove entire domains from search engines following their transition from a live, accessible site to a parked domain. If a site goes out of business, and is rendered inaccessible in this way, it also becomes unavailable for viewing via the Internet Archive’s Wayback Machine. The organization apparently receives queries about these sites on a daily basis.

The Internet Archive hopes that disregarding robots.txt files will help contribute to an accurate representation of prior points in the web’s history, removing their capacity to muddy the waters with instructions intended for search engines.

The organization has already ceased referring to robots.txt files on sites and pages related to the U.S. government and the U.S. military, to account for the enormous changes that can be made to domains between one administration and the next. This decision has caused no major problems, so there are high hopes that discontinuing the use of the files more broadly will be helpful.

Social Media

Nearly a million Facebook users followed these fake Russian accounts

Facebook purged two separate groups behind more than 500 fake accounts with Russian ties. One group had ties to Russian news agency Sputnik, while the other had behavior similar to the Internet Research Agency's midterm actions.
Social Media

Here’s how to save someone’s Instagram Story to your phone

Curious about how to save someone's Instagram Story to your phone? Lucky for you, it can be done -- but it does take a few extra steps. Here's what you need to know to save Instagram Stories on both iOS and Android.
Computing

Delete tracking cookies from your system by following these quick steps

Cookies are useful when it comes to saving your login credentials and other data, but they can also be used by advertisers to track your browsing habits across multiple sites. Here's how to clear cookies in the major browsers.
Computing

Lost your router? Here's how to find its IP address to help track it down

Changing the login information for your router isn't always easy, that's why so many have that little card on the back. But in order to use it, you need to know where to go. Here's how to find the IP address of your router.
Computing

Faster new PCIe 5.0 standard leapfrogs the best feature of AMD’s Ryzen 3

PCIe 5.0 will bring even faster data transfers, but it may only be found on HPCs and servers initially. The standard is four times faster than your current PC at transferring data, and new devices could appear later this year.
Deals

From Chromebooks to MacBooks, here are the best laptop deals for January 2019

Whether you need a new laptop for school or work or you're just doing some post-holiday shopping, we've got you covered: These are the best laptop deals going right now, from discounted MacBooks to on-the-go gaming PCs.
Product Review

LG Gram 14 proves 2-in-1 laptops don’t need to sacrifice battery for light weight

The LG Gram 14 2-in-1 aims to be very light for a laptop that converts to a tablet. And it is. But it doesn’t skimp on the battery, and so it lasts a very long time on a charge.
Computing

Keep your laptop battery in tip-top condition with these handy tips

Learn how to care for your laptop's battery, how it works, and what you can do to make sure yours last for years and retains its charge. Check out our handy guide for valuable tips, no matter what type of laptop you have.
Computing

Protect your expensive new laptop with the best Macbook cases

If you recently picked up a new MacBook, you’ll want something to protect its gorgeous exterior. Here, we've gathered the best MacBook cases and covers, whether you're looking for style or protection.
Computing

Watch out for these top-10 mistakes people make when buying a laptop

Buying a new laptop is exciting, but you need to watch your footing. There are a number of pitfalls you need to avoid and we're here to help. Check out these top-10 laptop buying mistakes and how to avoid them.
Computing

Don't spend a fortune on a PC. These are the best laptops under $300

Buying a laptop needn't mean spending a fortune. If you're just looking to browse the internet, answer emails, and watch Netflix, you can pick up a great laptop at a great price. These are the best laptops under $300.
Computing

Dell XPS 13 vs. Asus Zenbook 13: In battle of champions, who will be the victor?

The ZenBook 13 UX333 continues Asus's tradition of offering great budget-oriented 13-inch laptop offerings. Does this affordable machine offer enough value to compete with the excellent Dell XPS 13?
Gaming

Take a trip to a new virtual world with one of these awesome HTC Vive games

So you’re considering an HTC Vive, but don't know which games to get? Our list of 25 of the best HTC Vive games will help you out, whether you're into rhythm-based gaming, interstellar dogfights, or something else entirely.
Computing

The Asus ZenBook 13 offers more value and performance than Apple's MacBook Air

The Asus ZenBook 13 UX333 is the latest in that company's excellent "budget" laptop line, and it looks and feels better than ever. How does it compare to Apple's latest MacBook Air?