If you used Mozilla’s Firefox browser last week, you may have noticed that add-ons stopped working all at once. Though the internet technically didn’t break, all of Firefox’s browser add-ons, the equivalent of Chrome’s extensions, stopped working all at the same time, creating a situation that’s been dubbed the “armagadd-on.”
The problem was initially caught by Firefox Ghief Technology Officer Eric Rescorla at around 6 p.m. PT on May 3 and has since been fixed. It took Firefox roughly 12 hours to rectify the situation and get add-ons functioning again. In what appears to be an embarrassing glitch for the browser, Firefox is now using the incident as a learning lesson on what it can do to not break the internet moving forward.
“With that said, obviously, this isn’t an ideal situation and it shouldn’t have happened in the first place,” Rescorla wrote in a detailed postmortem blog post about Firefox’s fix and the length of time it took to issue a patch. “We clearly need to adjust our processes both to make this and similar incidents less likely to happen and to make them easier to fix.”
Firefox’s Armagadd-on
In order to protect users from bad add-ons (there are more than 15,000 third-party add-ons that serve a variety of functions, from blocking ads to managing passwords), Firefox requires the add-ons to be signed in an effort to prevent malicious code from running.
The signing process is complex and is designed to protect users, Firefox said. A pre-installed root certificate is stored in a hardware security module, or HSM, and it is used every year to sign an intermediate certificate that’s kept online. The intermediate certificate is also what is used to sign add-ons. When the add-on is presented, it is issued an end-entity certificate.
Because all the add-ons are signed with the same intermediate certificate, and that certificate was set to expire at 1 a.m. Coordinated Universal Time on May 4, all these extensions started to expire. Since Firefox checks the validity of the certificates every 24 hours, the add-ons didn’t all expire at once and different users were impacted at different times.
Firefox considered several options to remediate the expired certificate. Re-signing every add-on was not possible, Rescorla admitted, given the sheer volume of extensions available for the browser. So Firefox considered a parallel approach of patching Firefox to change the date to ensure that the certificate would be valid again, as well as generating a replacement certificate that was still valid.
Fix deployment
Though internet users were quick to criticize how long it took Firefox to deploy a fix that restored the functionality of add-ons, Rescorla defended the practice, noting that it took some time to get an intermediate certificate issued and that developing a new system add-on takes time.
“As I mentioned above, the Root certificate is in a hardware security module which is stored offline,” he said. “This is good security practice, as you use the Root very rarely and so you want it to be secure, but it’s obviously somewhat inconvenient if you want to issue a new certificate on an emergency basis. At any rate, one of our engineers had to drive to the secure location where the HSM is stored. Then there were a few false starts where we didn’t issue exactly the right certificate, and each attempt cost an hour or two of testing before we knew exactly what to do.”
Moving forward, the company is exploring steps to prevent a situation like this from happening in the future. “First, we should have a much better way of tracking the status of everything in Firefox that is a potential time bomb and making sure that we don’t find ourselves in a situation where one goes off unexpectedly,” Rescorla said. “We’re still working out the details here, but at minimum, we need to inventory everything of this nature.”
Firefox is researching mechanisms to be able to push updates out more quickly. To fix the problem that occurred earlier this month, Firefox used its Normandy Studies mechanism to get the system add-on certificate to its users, but that process has some side effects, like sending unwanted code. The company pledged that it will result more information about the incident next week.
