We encountered errors on the console of the Xen node usa1-xen-1 (Chicago datacenter) this morning. The on-site technician has determined a hardware fault is the root cause. We have migrated all affected VMs to other Xen hosts and service has been fully restored.
Author: Chris Wik
Hosted Email storage upgrade
Update 2013-12-12 22:28 GMT
The migration is complete and all Hosted Email services are now back up and running on the new SSD RAID array.
There was one minor issue that arose during the migration: the time on the new server was running fast by just over 2 hours, and the clock did not sync immediately at startup time, so a few emails were delivered with timestamps in the future. Sorry about that.
2013-12-12 10:00 GMT
We’ll be adding additional storage space to our mail.anu.net Hosted Email server tonight at approx. 22:00 GMT
We are also taking the opportunity to migrate all the mail storage from 7200RPM SATA disks to new SSD drives, bringing big performance increases and lower energy usage.
A short interruption to service is unavoidable but we’ll do our best to keep the downtime to a minimum. We’ll update this post to confirm when we’re done. Apologies in advance for any inconvenience caused.
[Resolved] SpamTitan service interruption
Update 2013-10-23 12:30 UTC
Backlog has been cleared, service back to 100%
Update 2013-10-23 11:45 UTC
We’re still seeing some delays processing incoming mail due to the backlog created by the outage earlier. No mail has been lost, and we expect the backlog to be clear within one hour.
Update 2013-10-23 08:35 UTC
Service fully restored. Apologies to all affected customers for the inconvenience caused.
Update 2013-10-23 08:05 UTC
ETA for fix is 20 minutes.
2013-10-23 06:15 UTC
Unfortunately we currently experiencing a service interruption on our SpamTitan email filtering system. We require assistance from SpamTitan support which come online at 09:00 British time, we expect the problem will be resolved shortly thereafter.