Update 16:54 CET: The network has been stable again for the past 20 minutes so we’re closing this incident. We will continue to monitor the situation carefully and follow up with network operations to obtain a reason for outage.
Update 16:36 CET: Engineers have isolated the fault and are working on a solution. We’re currently still seeing minor packet loss but most services are now accessible again.
As of 16:15 CET we are experiencing packet loss via some routes to our Amsterdam 2 datacenter. Network operations is investigating but we do not yet have an ETA on a resolution.
Impacted services: VMs hosted in ams2, Hosted Email, ams2-c6-1 and ams2-c7-1 DirectAdmin servers.
We will update this blog when we know more.
Please contact email@example.com if you have any queries relating to this incident.
Update 21:55 IST: Maintenance is now complete.
Engineers will be performing scheduled maintenance on-site in our ie1 (Dublin, Ireland) datacenter in order to debug an issue with a remote management system.
The work will take place on 17 September 2020 between 19:00 – 22:00 Irish standard time.
Services may be disrupted briefly while this essential maintenance is carried out.
Please do not hesitate to contact firstname.lastname@example.org if you have any questions.
Update 12:30 UTC: Reason for outage was a large scale Distributed Denial of Service (DDoS) attack on the network. While we were not the target of the attack, our upstream provider’s networks were saturated by the attack and our connectivity was negatively affected. The attack has now been brought under control with a DDoS filtering service. We are continuing to monitor the situation but so far connectivity looks to be stable again.
Update 12:12 UTC: The network is now stable again. We do not yet have a reason for outage. We will continue to monitor the situation and will follow up with network operations to find out exactly what went wrong and what preventative measures are available to reduce the likelihood of a recurrence. Please contact email@example.com if you are still experiencing any difficulties accessing your services.
Update 11:47 UTC: Engineers have identified the fault and are working on a fix. Still no firm ETA available.
As of 11:13 UTC we are experiencing sporadic packet loss of 20-60% on our IP transit connectivity in our Amsterdam 2 datacenter. Network operations team has been activated and is responding to the incident but no further details are available just yet.
Services affected include Hosted Email as well as web hosting accounts and virtual machines hosted out of our Amsterdam 2 datacenter.
We will update this post as soon as we have further information or an ETA on a resolution.