Category: Status updates

[Resolved] Amsterdam 1 Server Failure

2016-02-15 06:00 GMT: 2 hard drives failed at the same time in a server at our Amsterdam 1 datacenter, causing the RAID array to fail. This has resulted in several clients and some internal infrastructure going offline (including our support helpdesk).

We are currently working hard to restore backups on to another server and will provide regular updates. If you want to contact us you can tweet @anuinternet.

Update 08:55 GMT: our support desk is back online, some virtual machines are already back online, others are still restoring from backup.

Update 10:15 GMT: all servers except the anuhosting.net shared/reseller server are now back online. anuhosting.net has been priority #1 since we started the restore procedure 4 hours ago, unfortunately it is also by far the largest and is taking quite some time to restore. We estimate it may take another 2-3 hours to complete.

Update 13:40 GMT: we are working on recovering services on anuhosting.net, we aim to have mail services running very shortly followed by MySQL and Apache/PHP. Apologies for the ongoing service interruption.

Update 14:00 GMT: DNS and mail on anuhosting.net are operational again. MySQL was unable to recover InnoDB to a functional state, so we are restoring the last consistent database snapshot available which is from 04:00 Sunday. We will make the recovered InnoDB databases from the latest backup available to anyone who wants to try to extract missing data from Sunday. Most of the data is there but we were unable to recover it to a fully functional state, so we made the decision to roll back to a known good copy. We expect PHP/MySQL/Apache services to be back online within 30 minutes.

Update 15:30 GMT: we are running into problem after problem with restoring Web functionality and do not currently have an ETA. We are working as fast as possible to recover MySQL, Apache and PHP services on anuhosting.net. All other services are currently operational. Our sincere apologies for the continued downtime on shared and reseller hosting servers.

Update 01:30 GMT: It’s been a very long day, thankfully at this point we can say our anuhosting.net server is finally operational again. We have spent the past half hour testing as many sites as possible and things seem to be running well. We are concerned there may be a handful of InnoDB tables with errors, if your site is not functioning 100% this may be the cause. Please contact support@anu.net ASAP and we will do what we can to help get you back up and running. We will be on hand tomorrow to answer any questions and help with any remaining issues.

A big thank you to all our customers for their patience, understanding and encouragement throughout this difficult day.

We will of course follow up with a detailed review of our storage systems, redundancy measures, backup and disaster recovery plans.

[Completed] Scheduled maintenance in Amsterdam 2 datacenter, 7-10 November

For safety reasons, some components in the power distribution system supplying our servers in the Amsterdam 2 datacenter require replacement. This will involve 2 short interruptions to our power supply, one affecting our A feeds and one on the B feeds.

The maintenance on the A feed will take place on the 9th of November. As not all equipment connected to the A feed has the capability to automatically fail over to the B feed, we need to migrate some equipment manually prior to the maintenance window. This work will take place on the 7th and 8th of November after 19:00 CET.

Some services in our Amsterdam 2 datacenter may be unavailable for 1-5 minutes during this maintenance.

We will be monitoring our systems carefully throughout the maintenance window, however if one of your system is affected and remains unreachable for more than 5 minutes, please contact support@anu.net so we can investigate and bring you back online ASAP.

The maintenance window on the B feed will take place on the 25th November, but we do not expect any major impact during that time.

Thank you for your understanding.

[Resolved] Packet loss from some locations

As reported by network operations at our Amsterdam 1 colocation facility:

Type: Unscheduled
Start Time: 2015-06-12 10:45 CEST
End Time: t.b.d.
Location: n/a

Impact: We are currently suffering from degraded performance on one of our IP transit providers. The issue is under investigation. Customers might notice packetloss to remote locations.

Update: This seems to be a global issue. To partially mitigate this issue, we have temporarily disabled one of our transit providers. Please note that other ISP's around the world might suffer from the same problem.

This issue also appears to be affecting many customers hosted in our Amsterdam 2 facility. USA, Singapore, Dublin and Munich are not affected.