Post Incident Report
We wanted to take some time to explain about this morning's outage on server 7, what caused it and what we have done to make sure this does not happen again. Those people who are on server 7 will know we had similar incidents on the following dates and times:
25/07/2011 09:33
15/08/2011 08:28
19/08/2011 20:57
05/10/2011 09:43
Each time the server suffered what was the symptoms of an inbound Denial of Service Attack. In August when this happened we were informed by the system administrators at the data centre that it was another server causing this issue and they had mitigated the attack. This morning the symptoms were identical to those from August and the system admins at the data centre 'null routed' the offending server and everything stabilized again.
We have asked the data centre to move this server to another rack and another switch as there have been 4 outages on this server not directly caused by our server and this is not an acceptable situation.
We will move this server at 4am and we hpoe this is a permanent resolution to this issue.
The downtime this morning on this one server was limited but it is still not something we would have wanted and we are sorry for any inconvenience caused. Hopefully this outage this morning will allow us to get a permanent resolution to this issue that has happened 4 times in 4 months.
=========================================================
5 October 2011: 10.17am
The server has remined stable since the last update and latency is normal. We will continue to monitor
=========================================================
5 October 2011 9.48am
We have brought Apache back on line and are monitoring this. There is still slight latency we are seeing in ping response. We are still working on this issue and thank you for your patience
=========================================================
5 October 2011 9.30am
Unfortunately server 7 was getting a massive inbound widely distributed denial of service attack causing the switch to max out on its bandwidth and causing websites not to load.
We have booted the server into single user mode while we work to mitigate this attack.
Updates will be provided here but unfortunately at this time the server is off line
=========================================================
