###Incident Summary: On November 1st, 2015 around 11am Eastern we began receiving reports of increased latency and packet loss into and out of our datacenter in Atlanta, Georgia. After ruling out any issues with our own equipment and services, we escalate the issue to our service providor, Zayo. Zayo engineers quickly identified a DDoS attack directed at another customer in the same datacenter and began taking steps to mitigate that attack. After repeated attempts to restore service were met with an evolving attack and changing targets, it was decided that Zayo would remove the targeted customer from their network temporarily. Once this customer was removed around 1:30pm Eastern, traffic normalized, and connectivity was restored.

###Follow-up As DDoS attacks have become more frequent, and their effects have negatively impacted our customers a number of times, we asked Zayo for a detailed explanation of the events, and what was being done to improve the discovery, response, and mitigation of these incidents going forward. We received the following response :

“zColo is in the process of upgrading all core routing gear to carrier class equipment in order to reduce the impact of events such as these. While we cannot eliminate the threat of DDoS totally we can harden the network to prevent it from saturating connections and minimize the impact to other customers in events like these. Not only will this harden the routing infrastructure itself but the modern systems have much more visibility into traffic allowing quicker and more accurate analysis of attack vectors and traffic types. This will allow zColo to not only identify the attack quicker but mitigate it as well. Finally combined with this new infrastructure we are also implementing full out-of-band access to prevent catastrophic events from impeding network management.”

The equipment is already on-site, and is being installed with a projected completion date of the end of November. From our discussions with Zayo, we feel this is an appropriate response and are optimistic that this will help to eliminate most, and at least lessen the impact of the largest of these attacks on their shared network.

We take your connectivity and support seriously, and will notify you of any upcoming maintenance, and upon completion of this implementation. In the meantime, if you have any questions please submit them to support@railsmachine.com, and be sure to follow @RMachineStatus on Twitter, and watch status.railsmachine.com for updates during any future incidents.