CrowdStrike: Blue screens, delayed flights in an interconnected world

In July 2024, CrowdStrike, a global cybersecurity company, faced a significant outage when an update they sent out caused system crashes on about 8.5 million Windows devices worldwide. CrowdStrike released a routine security update on July 19, 2024. Unfortunately, this update contained an error that caused blue screen errors (BSOD) on affected computers. The CrowdStrike team fixed the error in approximately 78 minutes, but the short-lived issue caused widespread economic and operational disruptions.

For example, the aviation sector faced major disruptions with over 5,000 flights canceled globally on July 19, 2024, including 1,250 by Delta Airlines. This incident highlighted weaknesses in many organizations' business continuity plans, showing the need for stronger and more flexible recovery strategies. It has led many to rethink and enhance their cybersecurity and operational resilience approaches.

Impact Analysis

The systems affected were primarily Windows devices running a specific version of the Falcon sensor, CrowdStrike's advanced cybersecurity software. Their users spanned multiple mission-critical industries, including finance, healthcare, government, and transportation, highlighting the pervasive reliance on CrowdStrike’s technology for security operations and threat mitigation.

Global Business Operations Impact

The immediate consequence of the system crashes was operational paralysis.The significance of the CrowdStrike incident extends far beyond a mere technical glitch, impacting several critical areas globally:

Economic Disruption: Short interruptions in service can trigger significant financial losses, especially for businesses that operate in real-time markets such as stock exchanges and airlines. The CrowdStrike incident  led to a 21% drop in its shares, equating to a $16 billion loss in valuation, and a 0.71% decrease in Microsoft's share price, which caused a $23 billion loss in market value. Economic damages from this event are estimated to reach tens of billions of dollars.

Security Vulnerability: The downtime also left systems exposed to additional cyber threats. Normally, CrowdStrike’s Falcon sensor actively monitors and defends these systems, but during the outage, this protection was compromised, increasing the risk of security breaches. This incident affected 8.5 million computers globally, representing less than one percent of Windows machines worldwide, but had broad economic and societal impacts due to the use of CrowdStrike by enterprises running critical services.

Reputational Damage: For CrowdStrike, a leader in the cybersecurity industry, the incident raised serious concerns about their reliability and the safety of their updates — both crucial to their reputation and business model. The situation posed a significant question about their ability to safeguard not just their own systems but those of their clients. The outage also impacted major companies such as Amazon, Delta Airlines, Bank of America, Visa, and many others.

Operational Resilience: The incident exposed vulnerabilities in many organizations’ business continuity plans. The challenge of rapidly addressing and navigating the disruption revealed a clear need for more robust and flexible recovery strategies. This has pushed many to reconsider and strengthen their approaches to cybersecurity and operational resilience. For instance, the aviation sector experienced severe disruptions with over 5,000 flights cancelled globally on July 19, including 1,250 flights by Delta Airlines alone.

CrowdStrike's Response

CrowdStrike was quick to clarify that the incident was not a result of a security breach or cyberattack but a defect in a content update specifically for Windows hosts. Their CEO, George Kurtz, issued statements apologizing for the disruption and inconvenience, emphasizing that the safety and security of their clients remained a top priority. Kurtz reassured clients that despite the setback, the protective capabilities of their Falcon platform were not compromised outside of the downtime.

“Nothing is more important to me than the trust and confidence that our customers and partners have put into CrowdStrike. As we resolve this incident, you have my commitment to provide full transparency on how this occurred and steps we’re taking to prevent anything like this from happening again.” - George Kurtz, CEO of CrowdStrike

The first 48 hours following the incident were the most challenging in over 12 years of experience, according to Shawn Henry, Chief Security Officer at CrowdStrike. The assurance built over the years was lost in hours, letting down their customers and partners. Thousands of team members worked 24/7 to restore systems. Henry highlights that they are the true heroes and he thanks every customer and partner working around the clock. Despite this setback, the mission endures. They commit to emerging more substantial and better for every customer and partner.

Immediate Actions to Mitigate the Issue

Upon identifying the problem, CrowdStrike quickly isolated and fixed the error. Here are the steps they took:

1. Isolation of the Defect: CrowdStrike identified the specific update and the logic error within the Falcon sensor's configuration file that triggered the blue screens. They traced it back to a particular channel file designed to enhance named pipe execution monitoring, which had inadvertently caused the system crashes.
2. Deployment of a Fix: A corrected version of the channel file was rapidly developed and rolled out to replace the problematic update. This fix was deployed within an hour and eighteen minutes of the first report of the issue.
3. Communication with Clients: CrowdStrike provided continuous updates through their support portal, official blog, and direct communications. They advised all impacted users on immediate steps to mitigate the issue, such as rebooting systems to apply the fix.
4. Enhanced Support: Additional technical support resources were mobilized to assist customers in implementing the fix and ensuring that systems were restored to full functionality.
5. Monitoring and Testing: After deploying the fix, CrowdStrike conducted extensive monitoring to ensure the stability of the update and prevent any further issues. They also initiated a review of their update procedures to strengthen their deployment processes.

Our Assessment of the Outage and Recovery

Existing Contingency Plans

Crowdstrike’s  rapid identification and resolution of the software defect suggests that effective emergency protocols were in place. However, the widespread impact also revealed potential weaknesses in their pre-release testing and update deployment procedures. Enhancing these areas with more rigorous scenario-based testing can better prepare the company for similar future disruptions.

Customer Communication Effectiveness

CrowdStrike's communication during the incident was proactive and transparent, utilizing a variety of channels to keep customers well-informed. This strategy not only helped in managing the immediate fallout by maintaining customer trust but also ensured that users had access to real-time updates and recovery guidance. Their approach exemplifies effective crisis communication, crucial in sustaining customer relationships during unexpected challenges.

Restoration Actions

CrowdStrike acted swiftly to deploy a fix, effectively minimizing downtime. The corrective update was issued promptly, accompanied by detailed instructions for implementation, demonstrating an effective immediate response strategy. They also provided direct support to assist customers through the recovery process, emphasizing the company’s commitment to service continuity.

Future Risk Mitigation

In response to the incident, CrowdStrike is likely enhancing their software testing and deployment protocols. Strengthening these processes with advanced automated testing and manual reviews before release can prevent similar issues. Furthermore, consulting with external cybersecurity experts for audits and recommendations can reinforce their defenses, ensuring robust protection against internal errors and external threats.

Lessons learned from the CrowdStrike outage

The CrowdStrike incident provided several key insights into the vulnerability of cybersecurity frameworks, even within companies specializing in digital protection.

Firstly, it highlighted the necessity of thorough internal controls over software updates, underscoring how even minor errors can escalate into significant disruptions. Secondly, the incident illustrated the critical importance of swift, transparent communication during a crisis. Keeping stakeholders informed not only helps mitigate immediate concerns but also preserves trust and confidence in the long term.

Improvements Proposed Post-Incident

Following the incident, CrowdStrike has initiated several strategic improvements to enhance their operational resilience and prevent future disruptions. These include:

1. Enhanced Testing Procedures: Implementing more stringent testing protocols for software updates to catch potential issues before they reach the deployment stage. This might involve expanded testing phases and the inclusion of scenario-based simulations that mimic a wide range of operational environments.
2. Robust Incident Response Framework: Strengthening their incident response framework to ensure faster and more efficient management of unexpected issues. This includes refining escalation paths and decision-making processes to improve response times.
3. Increased Stakeholder Engagement: Developing a more structured approach to stakeholder engagement during crises, ensuring that communication is not only rapid and transparent but also tailored to meet the needs of different user groups.
4. Continuous Education and Training: Investing in continuous education and training for both technical teams and customer service representatives to handle crises effectively. This ensures that all team members are prepared to respond promptly and appropriately, equipped with the latest knowledge and skills.
5. Post-Mortem Analysis and Feedback Loops: Instituting regular post-mortem analyses following any incident, allowing for a feedback loop that continually refines processes. These reviews help identify root causes and prevent recurrence, ensuring continuous improvement in security practices and response strategies.

By addressing these areas, CrowdStrike aims to fortify its systems against future challenges, enhancing overall security and operational stability for themselves and their clients. These lessons and improvements are essential for maintaining the cutting edge in cybersecurity practices and for reinforcing the trust that clients place in their protective measures.

For more news and updates from our Security Analysts and Vendor Assessment teams on security and GRC, subscribe to our newsletter below.