VPSBlocks Incident Report

At 8:36PM on Friday 3rd April the data center (OMNIConnect DC) in which our equipment is located suffered a major power outage. Their incident report can be downloaded at https://www.vpsblocks.com.au/files/IncidentReport-3-4-2020INC14161.pdf

VPSBlocks staff members were onsite at OMNIConnect by 8:45PM, and by 8:55PM we had staff in our office working to bring services back online.

By 9:07PM some services were back online, and this continued throughout the evening. By 10PM more than 70% of services were back online. At that time there were issues with 2 SANs. One we managed to resolve by 2AM. The last SAN had a critical unrecoverable failure. The power outage had damaged controllers in multiple drives meaning they were not detectable by any system. We initially contacted 4 independent data recovery specialist companies, and was advised that any recovery is not possible. This unfortunately was confirmed today.

As a result we started restorations from our emergency SAN volume backups as well as managed backup repositories. We tried to balance these restorations to get clients up and running as quickly as possible, however given the many terabytes of data requiring restoration it took most of the weekend. By Monday morning all services were back online.

We deeply apologise for any loss that occurred to any client as a result of this outage. We will be discussing the data center incident report in detail with them. Please note that VPSBlocks has no interest financial or otherwise in OMNIConnect.

From a VPSBlocks point of view we feel this is a unique situation where having to restore data from emergency backups is the first time we have had to reach to such backups in 8 years of operation in anything other than routine testing. Our staff did a fantastic job of answering nearly every call and every ticket was responded to as quickly as possible. We tried to be as communicative as possible throughout the outage and subsequent restoration period.

This of course has brought up some issues that we will investigate solutions to in the near future. Primarily the backup restoration time was longer than we would have liked and there was an issue with one of the volume backups affecting a very small number of clients.

VPSBlocks prides itself on offering great service, we rarely see any ticket or request go unanswered for more than 10 minutes, and have always been willing to help clients, we believe, well beyond what most hosting companies will do. We appreciate each and every customer, and we hope you will continue to support us well into the future, as we look forward to serving you.

If you have any questions relating to the outage please email support@vpsblocks.com.au and we will reply as quickly as possible.

Thank you,
Will Kruss
Technical Director
VPSBlocks Pty Ltd