Armagadd-on explained: What Firefox learned when expired certificates made havoc

firefox chrome back end mozilla symbol hq headquarters building sign convention open source
Mozilla in Europe/Flickr

If you used Mozilla’s Firefox browser last week, you may have noticed that add-ons stopped working all at once. Though the internet technically didn’t break, all of Firefox’s browser add-ons, the equivalent of Chrome’s extensions, stopped working all at the same time, creating a situation that’s been dubbed the “armagadd-on.”

The problem was initially caught by Firefox Ghief Technology Officer Eric Rescorla at around 6 p.m. PT on May 3 and has since been fixed. It took Firefox roughly 12 hours to rectify the situation and get add-ons functioning again. In what appears to be an embarrassing glitch for the browser, Firefox is now using the incident as a learning lesson on what it can do to not break the internet moving forward.

“With that said, obviously, this isn’t an ideal situation and it shouldn’t have happened in the first place,” Rescorla wrote in a detailed postmortem blog post about Firefox’s fix and the length of time it took to issue a patch. “We clearly need to adjust our processes both to make this and similar incidents less likely to happen and to make them easier to fix.”

Firefox’s Armagadd-on

In order to protect users from bad add-ons (there are more than 15,000 third-party add-ons that serve a variety of functions, from blocking ads to managing passwords), Firefox requires the add-ons to be signed in an effort to prevent malicious code from running.

The signing process is complex and is designed to protect users, Firefox said. A pre-installed root certificate is stored in a hardware security module, or HSM, and it is used every year to sign an intermediate certificate that’s kept online. The intermediate certificate is also what is used to sign add-ons. When the add-on is presented, it is issued an end-entity certificate.

Because all the add-ons are signed with the same intermediate certificate, and that certificate was set to expire at 1 a.m. Coordinated Universal Time on May 4, all these extensions started to expire. Since Firefox checks the validity of the certificates every 24 hours, the add-ons didn’t all expire at once and different users were impacted at different times.

Firefox considered several options to remediate the expired certificate. Re-signing every add-on was not possible, Rescorla admitted, given the sheer volume of extensions available for the browser. So Firefox considered a parallel approach of patching Firefox to change the date to ensure that the certificate would be valid again, as well as generating a replacement certificate that was still valid.

Fix deployment

Though internet users were quick to criticize how long it took Firefox to deploy a fix that restored the functionality of add-ons, Rescorla defended the practice, noting that it took some time to get an intermediate certificate issued and that developing a new system add-on takes time.

“As I mentioned above, the Root certificate is in a hardware security module which is stored offline,” he said. “This is good security practice, as you use the Root very rarely and so you want it to be secure, but it’s obviously somewhat inconvenient if you want to issue a new certificate on an emergency basis. At any rate, one of our engineers had to drive to the secure location where the HSM is stored. Then there were a few false starts where we didn’t issue exactly the right certificate, and each attempt cost an hour or two of testing before we knew exactly what to do.”

Moving forward, the company is exploring steps to prevent a situation like this from happening in the future. “First, we should have a much better way of tracking the status of everything in Firefox that is a potential time bomb and making sure that we don’t find ourselves in a situation where one goes off unexpectedly,” Rescorla said. “We’re still working out the details here, but at minimum, we need to inventory everything of this nature.”

Firefox is researching mechanisms to be able to push updates out more quickly. To fix the problem that occurred earlier this month, Firefox used its Normandy Studies mechanism to get the system add-on certificate to its users, but that process has some side effects, like sending unwanted code. The company pledged that it will result more information about the incident next week.

Computing

The new MacBook Pro scored a pathetic 2/10 for repairability. Does it matter?

Apple’s devices aren’t exactly known for their repairability, and the latest MacBook Pro has done nothing to change that, as repair experts iFixit have just given it a lowly 2 out of 10 score for repairability.
Computing

Is your PC slow? Here's how to restore Windows 10 to factory settings

Computers rarely work as well after they accumulate files and misconfigure settings. Thankfully, with this guide, you'll be able to restore your PC to its original state by learning how to factory reset Windows.
Mobile

A.I. photo filters use neural networks to make photos look like Picassos

Artificial Intelligence apps put a novel spin on social network-style photo filters. Instead of just overlaying an effect, these photo filters use machine learning and neural networks to transform an image into a predetermined style.
Deals

Amazon cuts 44% off the Fisher-Price Smart Cycle for Kids this Prime Day

The Fisher-Price Think & Learn Smart Cycle is currently discounted by $48 from its original price of $110. Only for Prime Day, you can take it home to your kids for the amazing price of only $62.
Smart Home

Make Alexa your own: How to change the digital assistant's voice

If you live in a country where English is not the primary language, or you're learning a second language and want to get all the practice you can get, you can set Alexa's default language to one of nine other languages.
Emerging Tech

Buying on a budget? Here's all the best tech you can snag for $25 or less

We live in a world where you can get a cheeseburger for $1, a functioning computer for $5, and thousands of HD movies for $10 -- so it stands to reason that you should be able to pick up some pretty sweet gear for $25.
Computing

Microsoft’s new HoloLens hologram tech can do language translation

Microsoft has further developed its HoloLens hologram technology to include a language translation feature. This new capability was demonstrated this week at the Microsoft Inspire partner conference in Las Vegas.
Photography

Digitize old photos and film with the best photo scanners for 2019

Got a box of old photos to scan, or a batch of fresh film to digitize? The best photo scanners offer high-resolution, colorful scans along with features to make the scanning process less tedious.
Deals

Walmart extends Prime Day hot deals on Apple, Samsung, HP, Dell, and Bose

Walmart doesn't want Prime Day 2019 to stop and it extended its sales event beyond its original July 17 end date. We found great deals on Apple, Samsung, HP, Dell, and Bose products to highlight before they go away.
Computing

Rumored AMD Radeon RX 5600 could compete with Nvidia’s GTX 1660 Ti

AMD may have an RX 5600 RDNA-based graphics card in the works to help it better compete with Nvidia's powerful GTX 1660 Ti. Such a card would need to be faster than an RX 590, though, and with 4GB of GDDR6, that seems suspect.
Deals

Amazon Prime Day deals are ending, but you can still get great bargains

Prime Day 2019 has come to an end for Amazon, but that doesn't mean the Prime Day deals are over. With deals from Walmart, Best Buy, and Amazon still going on, this massive shopping event is continuing on through the week.
Deals

Walmart post-Prime Day sale: 4K TV, Apple, and Nintendo Switch deals

Prime Day 2019 has come to an end for Amazon, but that doesn't mean the deals are over. Walmart's Prime Day sale. Walmart's Prime Day sale lasts all day today, extending a whole extra day beyond Amazon's shopping extravaganza.
Deals

Save up to $300 on the Apple 10.5-inch iPad Pro at Walmart

The 10.5-inch iPad Pro excels in nearly every aspect, making it a worthwhile investment especially for artists and professionals. You can order the 64GB version for only $474 or the 512GB version for $699 on Walmart.
Computing

Change your Outlook password quickly and easily by following these steps

Keeping your digital accounts locked up with strong, unique passwords is important, so learning how to change your Outlook password quickly and easily is one of the best ways to stay safe online.