A rogue VM had been hacked and a scan rootkit had been uploaded onto the server. Although we have limits for network bandwidth, we believe the rootkit scanner had saturated connection limits within our network.
The VM in question has been decomissioned and the owner alerted. We are investigating the best method to prevent a re-occurrence of this should a similar thing occur again in the future.
Please be aware this is the first time an incident of this nature has occurred in over three years of operation. We do apologise for the downtime and are working hard to find a resolution to ensure it does not reoccur. Since our implementation of bandwidth outbound limitations and inbound DDOS protection our network has been solid, and we expect this to continue into the future.
We have implemented a monitor which picks up large spikes in packets sent on VMs and takes action accordingly. This is being trialed over the next week and will then be put into production. This should ensure that any re-occurrence of the above issue should not impact clients.
All our clients are valued clients, and we would like to take the opportunity to thank you for your continued support.