A cell tower outside the One Mount Pleasant Road offices of Rogers Communications in Toronto, March 15, 2021Melissa Tait/The Globe and Mail
Rogers Communications Inc. could not transfer millions of customers to its rivals’ networks – despite offers of assistance – during this month’s massive service outage, the company says in newly released documents.
The wireless giant’s chief technology officer reached out to his counterparts at BCE Inc. and Telus Corp. at 6 a.m. on July 8 to inform them of the service disruption and to warn them to look out for possible cyber attacks, according to a partially redacted submission from Rogers to the Canadian Radio-television and Telecommunications Commission, which was posted to the telecom regulator’s website late Friday.
The outage left millions without wireless, internet or home phone service and also affected 911 services and the Interac debit system. Four emergency alerts, including three tornado warnings and one dangerous person report, were not delivered to Rogers’ wireless customers, the company said in Friday’s disclosures.
Rogers later determined that the service disruption was caused by a coding error that occurred as part of a maintenance upgrade to its core network.
Although Telus and BCE’s Bell Canada offered assistance, Rogers quickly determined that its customers would not be able to use its rivals’ networks because certain elements of the Rogers network, such as its centralized user database, were inaccessible owing to the outage.
“Furthermore, given the national nature of this event, no competitor’s network would have been able to handle the extra and sudden volume of wireless users (over 10.2 million) and the related voice/data traffic surge,” Rogers said in response to a litany of questions from the CRTC. “If not done carefully, such an attempt could have impeded the operations of the other carriers’ networks.”
Industry Minister François-Philippe Champagne has instructed Canada’s telecoms to enter into a formal agreement to help each other during network outages and to provide customers with emergency roaming on their networks.
Rogers said it will explore various possibilities with its competitors as the telecoms work to formalize a mutual assistance pact to deliver to the minister in September.
The update that triggered the massive outage was the sixth phase of a seven-phase process that began on Feb. 8. The company said it went through a comprehensive planning process – including budget and project approvals, risk assessment and testing – before rolling out the upgrade. The first five phases had gone smoothly.
The sixth phase started on July 8 at 2:27 a.m. (Rogers said it typically does upgrades during times when network traffic is at its quietest.) The coding error that set off the cascade of events that caused the company’s core network to fail was introduced at 4:43 a.m., Rogers said.
“While every effort was made to prevent and limit the outage, the consequence of the coding change affected the network very quickly,” the Toronto-based telecom wrote.
Many of the telecom’s network employees were unable to connect to the company’s IT and network systems because of the outage. This slowed down the initial response, as staff had to travel to centralized locations were they could access the network. Complicating matters further was the fact that staff could not access the VPN (virtual private network) system linked to the company’s core network, which delayed efforts to identify the problem and restore services.
A number of Rogers employees were able to communicate with each other and coordinate recovery efforts using Bell or Telus SIM cards they had received in 2015 as part of an emergency contingency plan established between the wireless carriers.
A significant portion of the information that Rogers has disclosed to the CRTC has been redacted in the publicly available version of the submission. The wireless giant said it is being “as transparent as possible” in order to regain its customers’ trust, but noted that some of the information it has shared with the regulator must remain confidential because it is “commercially and operationally sensitive.”
“This is particularly true for systems designs and network operations that could be exploited by malicious actors who seek to disrupt our systems,” Rogers wrote.
The telecom said it has already taken steps to prevent future outages, and that it is reviewing its change, planning and implementation process. The company also said it plans to separate its wireless and wireline core networks to improve resiliency. However, most of the details of its plans – including the short, medium and long-term measures the company says it will implement in the coming days and weeks – are redacted.
Rogers said it took a number of steps after its wireless services went down for 16 hours in April 2021. But many of those measures were specific to the circumstances of the 2021 outage, which the company has said was caused by a software update by its supplier Ericsson. “Unfortunately, [these measures] did not prevent the particular circumstances that resulted in the July 8th outage, although they did contribute positively to its resolution,” the company wrote.
Rogers chief executive Tony Staffieri has vowed to make “every change and investment needed” to ensure that the company does not experience another similar outage.
Earlier this week, Rogers replaced its chief technology officer, former Vodafone executive Jorge Fernandes, with Ron McKenzie. Mr. McKenzie has three decades of telecom industry experience and was previously the president of Rogers for Business, the division that offers wireless and internet services to corporate clients.
The management change occurred just days ahead of a Monday hearing before the House of Commons committee on industry and technology, which is studying the outage.
The committee, which is made up of members of Parliament from all four major federal parties, is expected to grill Rogers executives about the five-day credit the company is providing to compensate its customers for the outage, as well as how the telecom will prevent the issue from recurring.
Our Morning Update and Evening Update newsletters are written by Globe editors, giving you a concise summary of the day’s most important headlines. Sign up today.