Server 21 Emergency Maintenance
-
Tuesday, 21st March, 2017
-
15:26pm
8.55am
Three members of staff are working through the server now and manually sorting any accounts still off line. If you are still seeing issues please come on live chat and we'll advise you accordingly. Whilst the vast majority of clients are on line we aim to have 100% of clients on this server restored this morning.
7.00am
We're at 92% restored now. Unfortunately these remaining accounts are the largest on the server and will be restored as quickly as we can. We apologise if you are one of the website still off line. Looking at the stats server 21 has a disproportionate high number of massive accounts compared to other shared hosting servers we have. This is the reason for the delay on the remaining accounts.
9.30pm
The restore is 85% completed.
7.55pm
The restore is 77% completed and these websites are now working. The remaining 23% of accounts are the largest accounts on the server so they are restoring more slowly as each one progresses (as our script restores in size order). Both servers are on gigE networking so the data transfer is happening quickly. The cPanel package and restore process is what is taking the time and there is nothing we can do to speed this process up. We'll post an update in 90 minutes.
6.30pm
Unfortunately we're seeing some issues with the data on the main disk array which is unfortunate and we're working on a restore from the daily backup. This was started about 45 minutes ago and we are already seeing some websites come back on line. Our restore script works through accounts in size order from smallest to largest. This will bring the vast majority of websites on line quickly without being slowed down by the bottleneck of a large account. This does mean that those with the most data stored in their accounts will have service restored last. We'll post an update in 60 minutes.
At this time we ask you not to open live chats or tickets and all updates will be posted here. Our team are working through accounts as they are restored to ensure websites are working. We'll happily answer any specific questions after the restore has been completed but at this time we need to focus on getting websites back on line.
4.25pm
We're seeing a failed disk in the array and the server needs a file system check. We're working on this with priority. Updates here asap.
4.05pm
We've rebooted the server but we still are locked out of SSH. We're awaiting our data centre remote hands to hook up a remote stand to the server. Once that is ready we will regain access.
3.30pm
Server 21 is currently showing as unresponsive on some of our monitors. Websites still load but SSH is down and Email is down. We're currently working with our system admins to reboot the machine to regain access. We'll post any updates here.
