Google is reporting that a data center serving European customers of Google Compute Engine, located in Belgium, lost data after the local electrical grid was struck by lightning four times in succession. The incident, which took place last Thursday, caused a brief loss of power to some of the company’s storage systems.
Presumably, any employee at the facility who uttered any variant on ‘lightning doesn’t strike twice in the same place’ following the first bolt has spent the seven days since regretting their choice of idiom.
Despite the automatic auxiliary system restoring the building’s power as soon as possible, and battery backup devices that maintained the situation in the interim, some data was lost due to the incident. However, this damage was limited to ‘very few cases,’ with staff intervening manually to save as much information as possible in the immediate aftermath of power being lost. What “manual” intervention means isn’t clear. Perhaps the employees powered the servers with exercise ‘cycles?
Google takes full responsibility for the issue, even though it’s difficult to imagine what more the company could have done, considering the circumstances.
The incident report pertaining to the loss of data reminds customers that GCE instances and persistent disks within a particular zone are housed within a lone data center, and as such events such as this that compromise that facility can result in a loss of data. GCE snapshots and Google cloud storage are recommended as alternatives, due to the fact that they are geographically replicated to defend against such outcomes.
Moving on from the loss, Google has committed to upgrading their hardware at the facility, as well as revisiting the protocols that are in place to deal with such an event. After all, lightning really can strike twice in the same place — so it’s best to be well-prepared if such an eventuality comes to pass.
