Thursday, Aug. 15, started pretty much like every other day for retired B.C. Catholic reader Eric Whalley. After breakfast he headed to his laptop for his daily 8 a.m. ritual of checking his Telus email to catch up on messages from family across Canada.
He immediately noticed something was amiss. Nothing had arrived in his inbox since 6 a.m. A prolific reader and subscriber to various news sources, Mr. Whalley was used to inbound content arriving in those intervening two hours.
Having been trained in computer server operations, he immediately suspected a problem on the Telus side. He checked in with the online response system, expecting to be told a security breach had occurred and that the mail servers had been taken offline as a precaution.
“No”, he was told, “it is a hardware problem.”
That, as it turned out, was just the beginning of a massive loss of service, and reputation, for the telecommunications giant, and the beginning of a multi-day loss of email service for Mr. Whalley and countless others across B.C. and Alberta, where Telus has the bulk of its business.
At the beginning of the outage, Telus was unwilling to make any public announcements about what had occurred or provide a timeline as to when service might be restored. It was the start of a public relations nightmare that perhaps could have been mitigated with some straightforward talk from the company.
Instead, Telus became the butt of a social media tirade from irate customers demanding to know what the company would be doing to offset business and personal losses. Discussion about compensation became a very hot topic on Twitter, and it was apparent there was no strategy to handle it.
While the outage was still in progress, offers were being made. One of the more common ones went something like this: a $10 credit, plus a choice of six movie rentals or 10 Telus points to be redeemed against services.
Whalley took the $10 and the points, although he was unclear just what he could do with them.
There have since been reports of $20 credits and a month of free Internet service. One individual tweeted he was given a $50 credit against a TV subscription for each of the next 24 months. It wasn’t clear if it was for an existing a TV subscription or for a new one. If the latter, it’s difficult to see that as compensation as it represents new money for Telus.
Bottom line: compensation is all over the map and appears to be tied to how many Telus services individual subscribers actually have. Telus did not reply to repeated requests from The B.C. Catholic for clarity on the compensation being offered to affected customers.
From the start, Telus was terribly slow to acknowledge there was a serious problem with its email service. When it did, three days in, it served up this paragraph, subsequently deleted, in which the company appeared to be apportioning at least some of the blame on a third party.
“Since Thursday, many of you have been unable to access, send or receive email via your TELUS.net account. This issue occurred during an overnight update to our servers in the early hours of Thursday, August 15, in partnership with our vendor Dell EMC, when a flawed repair procedure took the TELUS.net email system offline. We recognize that our customers rely on TELUS.net email for important business interactions, to communicate with family and friends and to manage their schedules and finances, and we apologize for this unacceptable service disruption.”
By Aug. 27, Telus was claiming service had been restored to the bulk of its email customers, along with historical emails, both sent and received.
Whalley noted Telus must have a different understanding of restoration than he did as he could only retrieve content back to Aug. 13, three days before the outage. Eventually all his sent emails were restored, but the incoming content remained unchanged around mid-month.
It is interesting that for such a high profile bad-news story, Telus did not have any major executive, least of all CEO Darren Entwistle, face the public with a mea culpa. That was left to a tired and somewhat dejected-looking chief customer officer Tony Geheran, three days into the outage.
A paragraph in a subsequent Telus update suggests the original email servers were rendered unusable:
“Work is underway between TELUS technicians and our vendor Dell EMC to restore emails from the original servers and we are working diligently through that process to smoothly integrate the old and new email addresses together. As data is recovered, old messages will be restored to the mailbox and full access will be available. This will take time, because the last thing we want to do is lose or compromise a customer’s precious information by being hasty. Whether a family picture, a sentimental note or a valuable document, we care about your data.”
Exactly what transpired to take down a major piece of telecommunications infrastructure in western Canada, and keep it down for two weeks, remains murky. Was it a vendor-supplied software patch that took down a storage cluster? Was the patch not properly tested before it was applied? If it was a patch, why was it referred to as a hardware problem?
Customers don’t really care. There are likely to be lawsuits filed over the matter and Telus has learned that email, as old as that technology may be, is still crucial for business and personal communication.
As this column goes to press there are still reports from customers with no access whatsoever to their email service. Others have access only through webmail on a desktop computer and not through a mobile device.
As for Whalley, he’s still patiently waiting for all his historical emails to be restored. And he’s wondering about the wisdom of settling for $10 and points.
Follow me on Facebook (facebook.com/PeterVogelCA), on Twitter (@PeterVogel), or on Instagram (@plvogel)