Blame game means we haven’t heard the end of CrowdStrike story

Recently, as I was watching the umpteenth re-run of a classic TV comedy series on MeTV (longtime Vancouverites will know this is one of many digital channels comprising the old KVOS analogue Channel 12), I was suddenly faced with darkness. No other nearby channel appeared to be affected so I wrote it off as a temporary glitch.

Only a couple of hours later did I realize this had been one of the first impacts of what became the massive worldwide CrowdStrike Microsoft systems failure.

For the next 24 hours or so this technology collapse spread far and wide, affecting all sorts of facets of modern life. Not being able to view the end of a Hogan’s Heroes re-run was of course hardly the end of the world. Many of the other impacts were consequential and in some cases had effects that lasted a week or even slightly longer.

By about midnight local time here in Vancouver the Twittersphere (X-sphere?) was coming alive with speculation about a mysterious glitch striking Windows-based computers. However, while the glitch was widespread it clearly wasn’t affecting all machines running the Microsoft operating system. In fact, evidence was mounting that personal computers weren’t affected at all.

In addition to my local observation about a TV channel there were reports about TV networks in the United States, Australia, and the United Kingdom being unable to operate. Then reports emerged of airlines being affected and specifically of two American airlines, United and Delta, having FAA-mandated ground stop orders. In essence these airlines could no longer fly until the technical reason behind the halting of computer systems was determined and fixed.

To get a better handle on how people in my area were affected I made a general post to the local community Facebook group. Several respondents reported family members stuck in United States airports. Another reported a relative stuck in Amsterdam on a flight to Prague when the ground stop order was implemented.

Analytical tests at local Lifelabs facilities were slowed significantly, depending on how many computers were out of action. Group posters noted that staff did their best with pencil and paper solutions. In some cases tests couldn’t go ahead as staff couldn’t access doctor requisitions sent by email. One writer reported being at a local hospital and told she might receive a bill as there was no way to confirm a valid MSP number.

On a more mundane level people reported their McDonald’s online point system couldn’t be accessed, and others noted they couldn’t pre-order Starbucks drinks from their mobile devices. Of more consequence was access to banking services as some of the big five had problems with teller and ATM systems. In some cases, anticipated payroll deposits weren’t made or weren’t accessible.

By morning it was apparent that this technology meltdown wasn’t a cyberattack. Linking all the outages was a common thread: corporate computers, especially from many Fortune 500 companies, running a security product from a business called CrowdStrike. Something had gone amiss with an update that had gone out the night before. As people tried to activate their work machines they were faced with the so-called “blue screen of death.” The usual reboot didn’t help. Affected machines remained unusable.

CrowdStrike offers security software to corporate and institutional customers with a focus on preventing ransomware encrypting machines. As such it interfaces closely with the operating system. It was a CrowdStrike update with some incorrect coding, presumably not tested properly before being widely, and automatically deployed, that caused the Windows operating system to go into a shutdown mode, and then displaying that blue screen.

Naturally a blame game ensued, with some coming to rest on the European Union and a claim that the EU required Microsoft to open sensitive parts of its operating system to third party vendors such as CrowdStrike, whose CEO was quick to note the issue was “not a security incident or cyberattack.”

We haven’t heard the last on this story. What made it particularly impactful is that each computer running the CrowdStrike package had to be “touched” by a technician in order to have the offending update file removed and thereby return the machine to a usable state. That entailed time and expense, with Microsoft estimating 8.5 million computers across the world were taken down, about one percent of all Windows devices, according to the company.

Follow me on Facebook (facebook.com/PeterVogelCA), or on Twitter (@PeterVogel).

[email protected]

Your voice matters! Join the conversation by submitting a Letter to the Editor here.