A major Internet Service Provider suffered a technical outage on Sunday (30/08/2020). This had a ripple effect and created an internet outage, which impacted sites such as Amazon, Ebay, Twitter, Microsoft Xbox Live, Cloudfare, Reddit, EA, Blizzard, Steam, among many others.
What happened?
Yesterday, at around 10:00 UTC, a significant number of Border Gateway Protocol (BGP) updates (advertisements) were (mistakenly) made at the start of the incident.
While a post-mortem from the service provider in question has not yet come, CloudFlare has offered a detailed analysis. In short, Border Gateway Protocol (BGP) misconfiguration (more specifically a bad Flowspec rule) seems to be the cause. Other independent findings point to the same cause as well.
How was Mailfence impacted?
Due to the technical nature of the outage, involving BGP routing — which navigates the internet traffic from peer to peer, the error spread outward from source of the incident and impacted services on a global level. As a result of the domino-effect, Mailfence services were not available between 12:03 CET and 16:20 CET.
Answers to some questions you might have :
1. Was any of my data (emails, documents or personal account information) on the Mailfence service accessed from the outside, manipulated, corrupted or lost?
Absolutely not. It was a connectivity outage problem due to routing issues in global internet traffic.
2. Will certain emails sent to me during the period of the outage not be delivered?
Absolutely not. The SMTP protocol manages this kind of issues. Whenever sender servers fail to deliver an email right away, they will always retry to deliver it later.
3. Why is there no redundancy?
There are many redundancies built in the service but for such BGP routing issues, unfortunately not all were functional.
4. Is there any standard process to inform customers about major outages and status of the fixing in place? Are you planning to introduce something like this?
If a problem of this kind occurs, we keep you informed using Twitter and/or this blog. The best way to communicate directly with us in such cases is to use social media platforms. In the future, we’ll also provision a status page hosted outside of our current network.
Final thoughts regarding this major Internet outage:
We really apologize for the inconvenience caused. The technical issue was out of our control and impacted a large part of the Internet. Despite all our efforts to offer the most robust and continuous service, it shows that Internet services are not infallible and that we continue to be dependent of the Internet as a whole.
We thank you for your support and understanding for this major Internet outage. It motivates us to study and implement additional improvements to the robustness and redundancy of our service. Learn more about Mailfence on our press page.
Follow us on twitter/reddit and keep yourself posted at all times.
