Wednesday 1 May 2024: 3.45pm
The issue has been resolved. We apologise it has taken so long to get to the bottom of this issue. A brief explanation follows. When you have spam experts set up any email sent to you is diverted to our Spam Filter Servers in our Maidenhead data centre racks. The spam filter servers check each message and those that are flagged as not spam are immediately released and sent to your server. This morning all email bound for our Northern Ireland data centre was not able to leave the Spam Filter Server. No inbound messages have been lost as they were all queued on the Spam Filter Server. The issue was intermittent and not all clients were affected. All queued messages are now being released gradually and should complete in a few hours.
Initially we were speaking to Spam Experts. We assumed the issue was Spam Experts related. It took 3 hours working with them to determine this was not a Spam Experts issue. We then started working with our data centre network team and discovered that one of our upstream providers had inadvertently blocked our Spam Filter Servers from communicating with our Northern Ireland Data Centre. That upstream provider in the Edinburgh Internet Exchange made a change to a router configuration and email started flowing. A further post mortem has determined that a BGP change was pushed to that upstream provider at 11pm last night causing a bypass to a previous firewall rule. The issue was in fact present from late last night.
We do apologise for this issue. Any clients who we had removed Spam Experts from during the course of today we will revert those changes.
Wednesday 1 May 2024: 2.55pm
We believe we have a solution. Bear with us. We have tested it on 2 clients and it has worked. We will check on a few more than post a full explanation. Really appreciate your patience.
Wednesday 1 May 2024: 1.30pm
We believe the issue is the incoming email filter only. We are actively working through and removing clients from the incoming filter temporarily and we will revert the changes once the issue has a resolution. Please come on chat if you need us to action this change for you as well.
Wednesday 1 May 2024: 12.22pm
We have narrowed down the issue to the Spam Filter server not being able to connect to some of our servers. Other servers work fine. We are still working with Spam Experts to troubleshoot this. We do have the ability to remove clients from receiving email through Spam Experts - that will incolve changing your MX Records but we absolutely can assist with that if you need us to. Just pop on live chat and request that. We will do our best to do this for you as quickly as possible. We've done this for some clients and emails are working again.
Wednesday 1 May 2024: 11.00am
We are receiving reports that those clients who route their emails through Spam Experts are unable to receive incoming emails. The emails are all queued in our Spam Experts Spam Filter Account and the option to release these emails is not working. We have a high priority support ticket opened with Spam Experts (opened at 10.16am). We have tried disabling the Incoming filtering in the Spam Experts GUI but that is not working. We will review this situation at 12.30pm and if Spam Experts do not have a resolution we can on a case by case basis remove Spam Experts from email accounts to allow email to flow direct from our servers again. We're obviously wanting to avoid this and and we want to give Spam Experts Support some time to troubleshoot. As Spam Experts is a third party product we need to wait on them to respond and we are sorry for the break in email service this morning.
Please remember our email service is a courtesy service provided free of cost with hosting plans as per our terms of service and listed here: https://manage.bigwetfish.hosting/index.php/announcements/457/A-Note-about-our-Email-Service.html - we will of course do our best to restore service as soon as possible.
