People gather around the Porter Airlines check-in counter at Pearson International Airport in Mississauga.Chris Young/The Canadian Press
1 of 19
It was supposed to be a basic content update, the kind of digital missive that software companies routinely send to customers to provide the latest features and fix bugs.
But a defect in an update sent by Austin, Tex.-based cybersecurity giant CrowdStrike Holdings Inc. CRWD-Q to customers that use Microsoft Windows resulted in a colossal string of information technology outages around the globe Friday.
The update caused computer screens at workstations, public display monitors and advertising billboards to be replaced by a malfunction window nicknamed the “Blue Screen of Death,” snarling operations for critical infrastructure providers including some of Canada’s largest institutions.
Hospitals, banks, insurers, airlines, grocers, governments and emergency service providers, media and countless others were thrown into chaos as many computers used to run their operations were rendered useless by CrowdStrike’s defect, which began affecting systems around 1:40 a.m. ET.
“This illustrates the fragility of our technology infrastructure on a global scale,” said Charles Finlay, executive director of Rogers Cybersecure Catalyst at Toronto Metropolitan University. “This is a wakeup call for all of us to recognize and understand that.”
CrowdStrike chief executive George Kurtz said on social media Friday the problem was “not a security or cyber incident” and that the company had quickly identified the problem and deployed a fix, including a basic four-step workaround for users. That message fell well short for some industry experts as bedlam reigned through the morning and early afternoon. CrowdStrike’s stock plunged 11 per cent on the day.
“This is a ‘Tylenol moment’ for CrowdStrike, and from what I read they aren’t stepping up,” said David Yach, BlackBerry Ltd.’s former chief technology officer for software, referring to a 1982 incident when seven people died from cyanide-laced pain-relief pills, prompting the manufacturer to recall all product and introduce tamperproof packaging in a widely lauded response. “Yes, CrowdStrike deployed a fix, and provided an involved manual process for recovery, but I get the sense they are now saying ‘So we are done’ rather than indicating they are working on something better.”
Calgary-based cybersecurity consultant Lisa Kearney said, “There seems to be somewhat of a breakdown in their change management processes in ensuring that any rollout of new updates and technologies are tested thoroughly, especially if they will impact critical infrastructure.” Ms. Kearney, CEO of the Women CyberSecurity Society, criticized Mr. Kurtz for saying the mistake wasn’t a security incident: “If you take the business of your clients, critical infrastructure, offline for several hours, to me that is a serious security incident” even if it isn’t a cyberattack. “He seems to be downplaying it and lacking accountability for what could have gone wrong.”
Long lines formed at airports on multiple continents as airlines lost access to check-in and booking services at the start of a busy summer travel weekend. In Canada, airlines cancelled 100 flights, according to aviation analytics company Cirium. Toronto’s Porter Airlines was the worst hit, cancelling all Friday flights until 3 p.m. and 56 of its scheduled 214 routes for the day as customers were unable to access its reservations, flight changes or inquiry systems.
Jennifer McHenry was supposed to be headed to New York with her fiancé to see a Luke Combs concert this weekend ahead of a work event on Monday. Instead, an hour-long plane ride from Toronto’s Billy Bishop Airport was turning into a possible day-long journey as she searched for train or bus tickets to get out of the city. “It’s a once-in-15-years kind of thing, but what can you do,” she said. The airport disruptions came less than a month after a mechanics’ strike at WestJet Airlines caused hundreds of flight cancellations. Porter said passengers who needed to cancel would get a refund.
Canada’s big banks were affected, as many of their online banking platforms including CIBC’s Investors Edge, TD Easy Trade and RBC Direct Investing warned customers some services and features were temporarily unavailable because of the global issue, including access to stock quotes and research data and the ability to make some transactions.
Intact Financial, Canada’s largest property and casualty insurer, reported delays in the abilities of subsidiaries Belairdirect and Johnson Insurance to serve customers, later saying all systems were restored by 4:25 p.m. Sun Life Financial saw a disruption to some of its global systems, though its Canadian clients were still able to access accounts and submit claims, spokesperson Alessandra Nigro said.
Insurance Bureau of Canada spokesperson Andrew Bartucci said it was too early to predict the impact of the IT outages on insurers and their policy holders, but said they could potentially trigger claims from a variety of policies.
Canada’s railway and telecommunication operators said they experienced limited impacts, while Canada Border Services Agency said it had experienced a partial systems outage of its telephone reporting system used by small aircraft passengers and boaters, which was resolved by early afternoon.
Federal Health Minister Mark Holland said in a statement the global outage created challenges for doctors and hospitals across Canada. “We are actively monitoring the situation and working with partners across Canada to support our health infrastructure.”
Several hospitals were affected Friday morning but had regained service by early afternoon. In Toronto, North York General Hospital and Sunnybrook Hospital said that patients were experiencing some delays, and Sunnybrook said work was under way to fix computer systems. University Health Network, Hospital for Sick Children and Unity Health said all services were operating normally, and the Ottawa Hospital and Alberta Health Services also said there were no disruptions to patient care.
Some CBC radio and TV programs didn’t air as scheduled Friday morning as the situation changed “minute to minute,” CBC spokesperson Leon Mar said. Starbucks said its mobile order ahead and pay features were temporarily unavailable. Meanwhile, Mirvish Productions lost the ability to sell theatre tickets online, by phone or even in person, using a system run by New York’s Tessitura Network.
While many companies said they fixed their issues Friday, the computer cleanup could last much longer for others. “IT departments are going to be very, very busy for the next week repairing this,” said veteran technologist Ken Nickerson, a partner focusing on innovation with Maverix Private Equity in Toronto.
Digital outages and problems with updates sent over the internet are relatively common and go back decades, while cyberattacks affecting millions are a regular occurrence. In November, 1988, a “computer worm” distributed to unwitting early internet users affected thousands of computers and resulted in a computer fraud and abuse conviction for its creator, a Cornell University graduate student.
BlackBerry sustained widely publicized network outages when it was a smartphone heavyweight in the 2000s and early 2010s. Rogers Communications in July, 2022, experienced a massive outage caused by a coding error introduced through an upgrade that left millions without cellphone, internet or home phone service and even affected the Interac debit system. Microsoft MSFT-Q users have sustained issues from updates in the past.
But bad software updates are becoming “exceptionally abnormal” as testing protocols for mission critical programs have become increasingly automated and are typically subject to an extensive quality assurance process before deployment, said John Sicard, CEO of Kinaxis Inc., an Ottawa-based supplier of supply chain management technology to some of the world’s largest corporations. But Mr. Sicard, whose company was not affected by the outage, added: “It’s not impossible. Software is written by human hands.”
Matt Holland, CEO of Ottawa-based cybersecurity company Field Effect – a competitor to CrowdStrike – called the event “a worst case scenario for a cybersecurity vendor from a responsibility perspective” as cybersoftware is typically installed deep in a computer’s operating system.
Mr. Holland said his own company subjected software updates to “a very rigorous quality assurance process that can last weeks or months” that includes the company installing updates on its own internal systems to ensure there are no problems before they are shipped. “It’s a way to ensure we don’t get into situations like this. I feel for CrowdStrike because that is a really challenging thing to recover from. This seems like a complete miss. I imagine there will be internal changes there to make sure this doesn’t happen again.”
Veteran cybersecurity investor and entrepreneur Michael Hyatt, executive chairman of Toronto’s DataStealth Inc., said Friday’s incident “shows the soft underbelly of how we’ve become so dependent on technology and how fallible we are” and suggested technology giants including Microsoft “are going to have to rethink” relationships with third-party vendors and focus on acquiring and integrating some of them to prevent similar incidents in the future.
“The good news was this was not a cyberattack,” he added. “If it was, it would have been a lot worse.”
With reports by Reuters, The Canadian Press and the Associated Press