A number of Netflix users in the US, Canada and Latin America who on Christmas Eve settled down to enjoy a movie ended up having to look elsewhere for their entertainment after problems at Amazon Web Services (AWS) caused a service failure.
“Terrible timing!” Netflix tweeted at the time, though its customers likely used far more colorful language in commenting on the situation.
On Monday AWS posted an explanation as to what went wrong, and took the opportunity to say sorry to its users.
“We want to apologize,”AWS said, adding “We know how critical our services are to our customers’ businesses, and we know this disruption came at an inopportune time for some of our customers. We will do everything we can to learn from this event.”
The service disruption began at around 12.30pm PT on December 24 and was caused by problems with Amazon’s Elastic Load Balancer (ELB) service, which automatically distributes network traffic to services supporting streaming, such as Netflix.
According to AWS, the problem occurred after ELB data was mistakenly deleted by a developer. “Unfortunately, the developer did not realize the mistake at the time,” Amazon wrote in its explanation. The Seattle-based company said it had already “made a number of changes to protect the ELB service from this sort of disruption in the future.”
In a related post over on the Netflix blog, the company’s cloud architect Adrian Cockcroft said the disruption affected its service in the US, Canada and Latin America, while its service in the UK, Ireland and Nordic countries continued as usual.
‘A slow Netflix night’
Cockcroft pointed out that Christmas Eve is usually “a slow Netflix night” as many people choose to celebrate with family and friends rather than sit in front of the box. Christmas Day, however, usually sees a much bigger demand for its service.
“While we truly regret the inconvenience this outage caused our customers on Christmas Eve, we were also fortunate to have Netflix streaming fully restored before a much higher number of our customers would have been affected,” Cockcroft wrote, adding, “It is still early days for cloud innovation and there is certainly more to do in terms of building resiliency in the cloud.”
Amazon, meanwhile, promised to “do everything we can to learn from this event and use it to drive further improvement in the ELB service.”
AWS has had to deal with a number of service outages this year. In June, for example, a severe storm caused power issues at its Elastic Compute Cloud data center in North Virginia. Netflix was affected by that outage too, as were Instagram and Pinterest.
