Updated OVHcloud has declared a ‘major incident’ at one of its Strasbourg (SBG2) data centres, with the company confirming a fire and telling customers to activate their disaster recovery (DR) plans.
Octave Klaba, founder of OVH, confirmed in a tweet that ‘everybody [was] safe’, and that damage examination and reports in the SBG1, SBG3 and SBG4 sites remained ongoing.
An update was posted on the OVH dashboard at 2342 UTC which read: “We have detected a degradation in service performance. Our teams are fully engaged in fixing the issue and fully restore the service as fast as possible.”
At 0353, an additional comment confirmed the fire. “We are currently facing a major incident in our data centre of Strasbourg with a fire declared in the building SBG2,” the company wrote. “Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4.
“If your production is in Strasbourg, we recommend to activate your disaster recovery plan,” the company added. “All our teams are fully mobilised along with the firefighters.”
At time of publication, Klaba wrote that all servers in SBG3 were ‘okay’, as was SBG1. “They are off, but not impacted,” he wrote of SBG3. “We create a plan how to restart them and connect to the network.” Klaba added four rooms out of 12 in SBG1 were destroyed.
Many commenters, alongside expressing hope and relief that staff were unhurt by the incident, noted the downtime which would result. Animation website Catsuka confirmed it was out of action, adding it will ‘probably take some time’ to restore from backups. Its server was contained entirely in SBG2.
Elsewhere, the #hugops hashtag was used liberally, most notably from rival providers:
OVH has a portfolio of 27 data centres in total, with two thirds located in Europe. The company’s focus on the continent has given it good positioning with regard to initiatives around data sovereignty and privacy.
On the ending of the Privacy Shield agreement last year, the company told CloudTech that it had “built… governance and organisation so as not to be impacted by extra-territorial data sharing with [the] US.” Subsequent business collaborations with Atos, Deutsche Telekom and Google Cloud have solidified this position.
Updated 1200 GMT: OVHcloud pointed CloudTech to an official statement which has just been released and which adds more information to the incident:
At 00:47 on Wednesday, March 10, 2021, a fire broke out in a room in one of our 4 data centres in Strasbourg, SBG2. Please note that the site is not classified as a Seveso site. Firefighters immediately intervened to protect our teams and prevent the spread of the fire.
At 2:54 am they isolated the site and closed off its perimeter. By 4:09 am, the fire had destroyed SBG2 and continued to present risks to the nearby data centres until the fire brigade brought the fire under control. From 5:30 am, the site has been unavailable to our teams for obvious security reasons, under the direction of the prefecture. The fire is now contained. We are relieved that no one was injured, neither among our teams nor among the firefighters and the services of the prefecture, whom we thank for their exemplary mobilisation at our side.
With our 15 data centres in Europe, our technical and commercial teams are fully invested to support our customers, implement solutions and mitigate the impact of the unavailability of our Strasbourg site. Our mission is to provide our customers with the highest quality of services to support their online activities and we know how important this is to them. We sincerely apologise for the difficulties and challenges caused by this fire. We are committed to continue to communicate with the greatest transparency about the cause of the fire and its consequences.
We are currently assessing the impact of this incident and will communicate as transparently as possible on the progress of our analyses and the implementation of solutions. All of our communication channels, starting with our incident tracking platform (travaux.ovh.com), are accessible so that you can stay informed of developments in real-time.
Update 1400 GMT Mar 11: A second OVHcloud statement has outlined the company’s three priorities following the incident. The company has a stock of new servers at its Roubaix and Gravelines sites ready to be delivered to the ‘majority’ of affected customers, as well as working on a plan to relaunch the unaffected datacentres (SBG3 and SBG4) and the partially affected (SBG1) ‘as quickly as possible’. Customers should “exercise caution around the emails they receive” with regard to malicious activity.
Interested in hearing industry leaders discuss subjects like this and sharing their experiences and use-cases? Attend the Cyber Security & Cloud Expo World Series with upcoming events in Silicon Valley, London and Amsterdam to learn more.
