Editor’s Note: Last week and again this week, Green Mountain Access users, as well as millions of others, were faced with two email outages. Here’s what happened)
By Kurt Gruendling
What caused this outage?
Our third-party email vendor proactively forced a failover of the firewall that controls access to the email cluster to prevent impact from a maintenance event by the infrastructure provider. The redundant firewall instance failed to resume operation. The infrastructure vendor and the firewall vendor assisted with troubleshooting. An errant command resulted in both the active and the failover firewalls becoming inaccessible during those efforts. The virtual firewalls had to be rebuilt at this time, causing an unusually long delay in resuming functionality.
Will emails sent during the outage be lost?
Messages that were sent to email addresses on the platform during the outage should have been deferred by the sending server for redelivery. The industry standard is to retry for up to four (4) days, but this may vary for each sending platform. If the issue persists beyond the retry period, senders should receive a delivery failure notice and will be able to resend their message.
Who are your email vendors?
National Rural Telecommunications Cooperative and Zimbra (Synacor). See below for more information on these companies. Please know, we select our partners very carefully and fully vet them on their technical and operational capabilities.
About National Rural Telecommunications Cooperative (www.nrtc.coop)
The National Rural Telecommunications Cooperative (NRTC) is a member owned cooperative which is made up of other companies like ours and allows us to increase the scope and scale of the services we provide (such as email). NRTC has the scale to be able to partner directly with Zimbra (Synacor) to leverage their hosted cloud email service.
About Synacor (www.synacor.com):
Synacor is known for managed portals, advertising solutions, email and collaboration platforms, and cloud-based identity management. Synacor hosts millions of email accounts. Our mission is to enable our customers to better engage with their consumers. We serve video, internet, and communications providers, device manufacturers, governments, and enterprises. They use our technology platforms and services to scale their businesses and extend their consumer relationships. They are headquartered in Buffalo, with offices in Boston, Dallas, London, Ottawa, Pune, Singapore, and Tokyo.
How will you prevent this issue from happening again?
We will work with NRTC and Synacor on a root cause analysis to ensure we fully understand everything that happened to create this situation. Once we have that information, we will work to resolve the issues that led to the disruption to prevent recurrence.
Why did you need a new email vendor prior to selecting Synacor?
We constantly evaluate our solutions to ensure we’re providing the best technology for our members and their subscribers.
On our previous system, which was managed directly by NRTC, we were susceptible to multiple points of failure, including Zimbra bugs, Cloud provider outages and GoSecure SPAM reliability issues and outages.
When outages did occur, delays were created because we first had to determine which of the three was the root cause before we could implement a fix, leading to extended outage periods.
Building a new email environment would have taken years and would likely not have improved on the multiple points of failure.
Alternative email systems such as Open-Xchange were considered, but came with a lengthy development cycle, a completely different user interface, a high customer learning curve, and significant anticipated tech support call volumes for all parties.
After a thorough vendor analysis, we chose Synacor as our hosting partner.
Why did you choose Synacor as your email vendor?
The data and information uncovered during our analysis indicated the platform would experience fewer outages and, if an outage did occur, recovery time would be greatly improved.
Same Zimbra email platform as previously provided by NRTC, so subscribers would not have to learn a new user interface
Modern architecture with full redundancy
Trusted by providers since 2001, Synacor email solutions have historically proven to be reliable and robust
Secure and compliant Cloud Hosting
99.99% SLA - full stack
Integrated SPAM UI utilizing Vade Secure with AI (SPAM vendor with 1+ billion users)
Unsubscribe button
Do you still have confidence in Synacor as your email provider?
We have confidence in the Synacor mail architecture and are confident that this event was the result of a string of unusual and unfortunate events that we don’t feel will reoccur. Synacor is scheduling a series of tests of the key equipment during low usage maintenance windows and have committed to make the necessary infrastructure enhancements and add safeguards to ensure a similar situation does not happen again. Their team has been very responsive and transparent during this situation.
Gruendling is vice president of marketing and business development for Waitsfield and Champlain Valley Telecom/Green Mountain Access.
