Database Issues on Multiple Servers

  • Wednesday, 6th November, 2019
  • 11:22am
Good Morning

Last night we had reports that one server had a MySQL database issue and the mysql would not restart.  We worked through this and discovered that an update contained a bug.  This was not a cPanel issue and was in fact a reported bug with MariaDB. 

The issue is only affecting a very small percentage of servers and we are yet unsure of the exact trigger to make this break.  This is happening worldwide on all cPanel Servers running MariaDB database engine. We are continuing to monitor cPanel forums for relevant information.

The good news is there is a solution that takes us about ten minutes to work through once we discover a server with this issue.  For the eagle eyed among you the solution on the forums pasted below would just take 1 minute but we are taking the extra precaution of backing up all relevant database information just to be super cautious (as innodb is a pain if it goes wrong!).

The solution is to downgrade MariaDB on the server to the previous older version (after backing up all database tables and other important database files).  There is no one click option for this so our senior system admins are needing to perform this manually when we discover this.  We have a senior admin whose one task today is to fix these issues as they are reported to us.  If MySQL does not start on your server and your database driven websites are not working then it's likely you are affected and you need to contact us.

This issue is ongoing and so far today we've performed this operation on about 9 servers.  As the issue is reported we have a team ready to perform the fix while we await more information from cPanel.  It is only happening on a small number of servers and we are seeing exactly what others are seeing in the forum posts below.  We are unable as well (like the others below) to determine a trigger.

Link:  https://jira.mariadb.org/browse/MDEV-14021
Link: https://forums.cpanel.net/threads/mysql-r-service-failures-after-update-to-latest-version.661413/
Link: https://forums.cpanel.net/threads/mariadb-error-after-update-10-2-28.662609/

Please be assured this issue is software related to MariaDB and (we assume at this time) cPanel updating to a buggy version.  We will of course edit this thread if we get more confirmed information but for now please know we are working quickly to fix this issue as it occurs across our server fleet.
« Back