Data Centre Networking Outage Affecting a Segment of the DC
-
Tuesday, 27th September, 2016
-
08:45am
Update on Reason for outage - 29 September
Unfortunately IOMART Maidenhead have not issued a Reason for Outage yet for this incident. We have today been told the RFO document is being prepared by the IOMART headquarters in Glasgow and will not be released until next week. As stated earlier once our provider give us reasons for the outage we will pass it along.
1.22pm
IOMART maidenhead working with Cisco still have not resolved the issue in 1UH network location. The final server of ours that is offline we are having them move to our other racks in South Lrw in the DC. That will resolve the networking issue for all our clients as we have moved the servers out of the new location. As mentioned earlier we will email all affected clients the RFO (reason for outage) once the data centre have this issue resolved. We sincerely apologise for the outage and we thank those clients who had to wait longer for understanding.
11.59am
Still waiting an update from IOMART Maidenhead networking team who assure us Cisco and them are actively working on this extended network outage in the 1UH segment. The DC techs have found the time to physically move two of the three affected servers now from our racks in 1UH where the networking issue is happening to our racks in DC5 South Lwr location where there is no networking issue. That leaves just one server off line at this time. We're working to have that resolved as soon as possible. We thank you all for your extended patience and once we have the RFO (Reason for Outage)
10.38am
Still no word on a resolution. There are three servers affected still belonging to us and many servers belonging to other providers still affected. One server is a production live server, one is a server the client is developing on but no live websites pointing yet and one is a backup server. We just spoke to the team at IOMART Maidenhead and requested they move the three servers to another rack we have in a different DC segment. They informed us they were extremely busy and this was not a normal request. We compromised and they have agreed to move the one production server but they cannot move the other two within a timely manner. We're reaching out to all three clients now to explain this logic.
9.49am
IOMART Maidenhead have replied to the ticket we lodged and have told us they will try to bring the systems we highlighted back on line. They also sincerely apologised for the inconvenience. Unfortunately this is a network level event that is completely out of our hands.
9.26am
Three support tickets with highest priority have been opened in IOMART Maidenhead for the three servers still affected by this networking outage as service has not returned as indicated in their update of a few moments ago. We'll continue to push for a speedy resoluition and once the RFO (reason for outage) is issued we will make available to all affected clients.
9.22am
IOMART have just emailed us to inform us there is a 'return of service' for all affected servers. This is not correct. We are openiong individual tickets now for the three servers still showing as offline and we will be calling through as well. Please bear with us and we will have an update asap.
9.15am
Three servers have just come back online so it appears the engineers are working through this issue systematically. We are fully aware of all the other servers that are still showing as down and we are seeking an ETA but none is forthcoming from the data centre as all seniors are working on the resolve which is understandable. The moment we have more information it will be posted here.
9.10am
Just to clarify some chats that have just come in where clients have expressed concern about their data: Please note this is a network level event. Your data is fine. It's just the network that is down.
9.00am
We are aware of an ongoing data centre network issue that is affecting all servers located in the 1UF segment of the data centre. This is affecting the entire data centre segment (ie not just us) and we have been told Cisco Engineers on on site at the moment working on the issue.
We have no update yet but have been promised a complete reason for outage once the issue is resolved. The senior admins at IOMART are working right now on this issue and once it is resolved they will issue a full report which we will pass onto our clients.
This is currently affecting our newest servers located in 1UH location. Any server deployed since June 2016 is currently down. 95% of our servers are unaffected and we apologise if you are one of those clients affected at this time.
This is all we know and although we are on chat and helpdesk we do not have any more information to what is posted here at this time. We will update this announcement regularly so please refresh the page.
