Server 120 Emergency Maintenance
-
Wednesday, 23rd November, 2016
-
11:15am
We're been struggling with a load issue in Server 120. Our senior technicians have identified a problem with Atomic Secured Linux (ASL) firewall and this is causing load on the server. We need to reinstall ASL. As the server is currently booted into the Secure ASL Linux Kernel we need to reboot the machine back into the standard CentOS Kernel. There will be a little downtime while we do this followed by elevated load and latency while we reinstall the product.
We'll do all we can limit downtime but it may become necessary to preiodically stop services during this install process and we apologise this is being completed during business hours.
Updates:
11.20am: Server rebooted to prepare for maintenance
11.25am: Server back from reboot technician starting work on Maintenance tasks websites on line
11.32am: Services may be unreliable at this time while our technical team continue working on this issue
11.38am: Websites are back running with higher latency
11.55am: The ASL product has been uninstalled and is being reinstalled now
12.15pm: Sites are still loading normally with increased latency - the ASL product is still installing.
12.30pm: ASL has completed installing we're configuring it now before a final reboot
12.36pm: We are rebooting the server to boot back into the ASL Secure Linux Kernel
12.41pm: Server is back on line in the ASL Kernel. Services Starting up now
2.50pm: Unfortunately the server crashed again. A senior admin is still checking this.
3.00pm: The server has been rebooted and we're still working on this with top priority
3.05pm: The server is back on line we're monitoring very closely
