Internet Archive will ignore robots.txt files to keep historical record accurate

internet archive robots txt server
Internet Archive
The Internet Archive has announced that going forward, it will no longer conform to directives given by robots.txt files. These files are predominantly used to advise search engines on which portions of the page should be crawled and indexed to help facilitate search queries.

In the past, the Internet Archive has complied with instructions laid out by robots.txt files, according to a report from Boing Boing. However, it has been decided that the way that these files are calibrated is often at odds with the service that the site sets out to provide.

“Over time we have observed that the robots.txt files that are geared toward search engine crawlers do not necessarily serve our archival purposes,” stated a blog post that the organization published last week. “Internet Archive’s goal is to create complete ‘snapshots’ of web pages, including the duplicate content and the large versions of files.”

Robots.txt files are increasingly being used to remove entire domains from search engines following their transition from a live, accessible site to a parked domain. If a site goes out of business, and is rendered inaccessible in this way, it also becomes unavailable for viewing via the Internet Archive’s Wayback Machine. The organization apparently receives queries about these sites on a daily basis.

The Internet Archive hopes that disregarding robots.txt files will help contribute to an accurate representation of prior points in the web’s history, removing their capacity to muddy the waters with instructions intended for search engines.

The organization has already ceased referring to robots.txt files on sites and pages related to the U.S. government and the U.S. military, to account for the enormous changes that can be made to domains between one administration and the next. This decision has caused no major problems, so there are high hopes that discontinuing the use of the files more broadly will be helpful.

Smart Home

Airbnb hosts are offering free rooms to those displaced by California wildfires

Several thousand Airbnb hosts in California are opening their homes to help those displaced by the devastating wildfires in the state. Free accommodation is being offered to those affected through November 29.
Computing

These are the 5 best free antivirus apps to protect your MacBook

Malware protection is more important than ever, even if you eschew Windows in favor of Apple's desktop platform. Thankfully, protecting your machine is as easy as picking from the best free antivirus apps for Mac suites.
News

Zoom in on famous works through the Art Institute of Chicago’s new website

Art lovers, listen up. The Art Institute of Chicago has given its website a serious makeover with new features that let you get up close and personal with more than 50,000 artworks by famous (and not so famous) artists.
Computing

Will Chrome remain our favorite web browser with the arrival of newest version?

Choosing a web browser for surfing the web can be tough with all the great options available. Here we pit the latest versions of Chrome, Opera, Firefox, Edge, and Vivaldi against one another to find the best browsers for most users.
Computing

Nvidia admits its 2080 Ti cards have a problem, but isn’t saying what it is

Nvidia has admitted that there is a wider problem affecting its RTX 2080 Ti Founders Edition cards, despite earlier claims to the contrary. It has also promised to continue working with affected consumers to replace dead cards.
Computing

Four fake cryptocurrency apps were listed on the Google Play Store

It is a dangerous time to be going after crytocurrency on Android. Four bogus cryptocurrency apps were spotted on the Google Play Store this week, according to a report from cybersecurity researcher Lukas Stefanko. 
Computing

Turn your iPad into a display for your new Mac Mini with this workaround

The folks at Luna Display have figured out a workaround which lets you get the best of both worlds and use Wi-Fi and an adapter in order to turn your iPad into a display for the 2018 Mac Mini.
Computing

Windows Update not working after October 2018 patch? Here’s how to fix it

Windows update not working? It's a more common problem than you might think. Fortunately, there are a few steps you can take to troubleshoot it and in this guide we'll break them down for you, step by step.
Computing

Microsoft Surface Pro 6: Everything you need to know

The Surface Pro 6 is officially here, though it's not as big of a redesign as you might have hoped. With a new coat of black paint and an 8th-gen processor, this is a small update. If you've been eyeing a Surface Pro, you may want to wait…
Deals

Black Friday 2018: The best deals so far

Black Friday is the biggest shopping holiday of the year, and it will be here before you know it. If you can't wait until November 23 to start formulating a shopping plan, we've got you covered.
Computing

Latest SMS breach could allow hackers access to your online accounts

A new security breach that exposed more than 26 million text messages could be a huge nightmare for users relying on two-factor authentication. Many of the SMS on the database contained security codes and account reset links.
Computing

Microsoft’s Windows 10 Mail client goes freemium with the introduction of ads

Microsoft Windows Insiders are finding a nasty surprise inside the Mail app on the latest Windows 10 preview build in the form of banner ads. These ads will appear in the Mail app regardless of the webmail service you use.
Computing

All the best Apple MacBook deals for Black Friday 2018

Shoppers looking for a new Apple laptop could find huge savings on a new MacBook come Black Friday. Retailers are offering discounts as much as $650 on select MacBook, MacBook Air, and MacBook Pro models this holiday season.
Computing

Apple discontinues AirPort Extreme, Time Capsule as it exits Wi-Fi router business

Apple is now officially no longer in the router business. The company had already stopped selling the AirPort Express, and now its retail stores and websites have stopped offering the AirPort Extreme and Time Capsule.