Download PDF

Open in Causelink

RCA TRAINING

Root Cause Analysis training by Sologic provides the tools, skills, and knowledge necessary to solve complex problems in any sector, within any discipline, and of any scale.  Learn More
 

SOFTWARE

Sologic’s Causelink has the right root cause analysis software product for you and your organization. Single users may choose to install the software locally or utilize the cloud.  Our flagship Enterprise-scale software is delivered On Premise or as SaaS in the cloud.  Learn More

 
The purpose of this example RCA (Root Cause Analysis) is to provide a learning aid to Sologic students and others interested in root cause analysis. Information and cause-and-effect charts contained in Sologic published examples are from publicly available sources, such as newspaper and magazine articles, published governmental reports, academic papers, or occasionally from first-hand knowledge of the event. Sologic has not investigated any event published at sologic.com professionally. If you have questions, comments, or concerns please contact us.

Parameter Count Mismatch Present
The IPC Template Type introduced in sensor version 7.11 defined 21 input parameter fields, but the integration code that invoked the Content Interpreter with Channel File 291’s Template Instances incorrectly supplied only 20 input values. This mismatch between the expected 21 inputs and the provided 20 caused the system crashes when the software tried to access the missing 21st input. It is not clear from the RCA report why only 20 input values were provided.

AdobeStock_144033326_delayed-flights-SMALL.jpeg

Updates Released Before July Did Not Identify Error Issue with Windows
Previous updates and initial deployments of the IPC Template Instances in Channel File 291 did not identify the parameter mismatch issue. The error remained undetected due to the use of wildcard criteria in tests and the lack of specific tests for non-wildcard matching criteria in the 21st input parameter field. It was only after the July 19, 2024, update, which introduced non-wildcard criteria, that the mismatch caused system crashes. The Content Validator and other testing mechanisms failed to detect this issue because they were based on the assumption that all 21 inputs would be provided.

QA/QC Did Not Detect Problematic Mismatched Input Values
The quality assurance and quality control (QA/QC) processes did not catch the mismatched input values due to several oversights. First, there was no compile-time validation to ensure that the number of inputs matched the defined template. Additionally, the Content Validator had a logic error that allowed the faulty Template Instances to be deployed. The testing processes, which included stress testing and deployment validations, also failed to cover the scenario of non-wildcard matching criteria for the 21st field. As a result, the problematic content was released without being identified during QA/QC.

Solutions
The following solutions add layers of protection that address the root causes of the Channel File 291 incident by improving validation, testing, deployment, and control mechanisms, thereby enhancing the overall resilience of CrowdStrike's Falcon sensor system.
  • Introduce a rigorous compile-time validation mechanism that checks for consistency between the number of input parameters defined in the template and the number provided to the Content Interpreter.
  • Expand the testing framework to include a wider variety of matching criteria, particularly non-wildcard criteria for all input fields.
  • Modify the Content Validator to include logic that verifies the input-output consistency, ensuring that the number of inputs provided matches the number expected by the Template Type.
  • Add Runtime Input Array Bounds Checks to the Content Interpreter.
  • Create additional checks in the content validator.
  • Provide customer control over rapid response content updates.

RCA TRAINING

Root Cause Analysis training by Sologic provides the tools, skills, and knowledge necessary to solve complex problems in any sector, within any discipline, and of any scale.  Learn More
 

SOFTWARE

Sologic’s Causelink has the right root cause analysis software product for you and your organization. Single users may choose to install the software locally or utilize the cloud.  Our flagship Enterprise-scale software is delivered On Premise or as SaaS in the cloud.  Learn More