Practicing scientists routinely make causal generalizations in their research, and they almost never use formal probability [random] sampling when they do…. We present a theory of causal generalization that is grounded in the actual practice of science…. Although this theory was originally developed from ideas that were grounded in the construct and external validity literatures[,]… we have since found that these ideas are common in a diverse literature about scientific generalizations….169
Although random sampling is arguably “a better fit between the sampling particulars of a study and more general inferences to constructs or over variations in persons, settings, treatments, and outcomes,”170 non-random sampling is generally more practical than random sampling in many instances.171 For example, relevant population characteristics may not be fully described as in the case of outdated census information. Another might be the inability to access the entire population due to logistical, cost, or other constraints. Fortunately, these and other issues with random sampling may be dealt with in various ways. One could, for example, simply modify the approach to random sampling through the use of stratified and multi-stage sampling techniques. Or, to take advantage of its practicality, one could use non-random sampling that is grounded in the theory of generalized causal inference.
Generalized causal inference requires the identification of construct labels—controls and related characteristics in our case—and exploring the extent to which a causal relationship generalizes over variations in these construct labels, and causal generalizations can generally be made based on five closely related principles:
- Surface similarity, e.g., categorizing phenomena based on similar characteristics;
- Ruling out irrelevancies, e.g., ignoring characteristics of phenomena not relevant to their categorization;
- Making discriminations, e.g., discarding phenomena that do not fit a categorization;
- Interpolation / extrapolation, e.g., generalizing between / beyond sampled values; and
- Causal explanation, e.g., attributing causation to phenomena based on theorized structural similarity (as opposed to surface similarity).172
For our purposes, the two most applicable principles are surface similarity and causal explanation.
The general construct we use in our assessments is essentially information risk, which we measure through the assessment of information security (cybersecurity), privacy, and compliance control implementation maturity. Purposive (non-random) samples of a risk-based specification of controls—derived from the comprehensive library of controls provided in the HITRUST CSF—are then selected to address specific questions around the state of information protection provided by an organization. The primary rationale for using this approach is that an appropriate purposive sample will provide reasonable assurances to relying parties at a reasonable cost to the organization.
HITRUST’s initial approach to certification was to use a ‘purposive sample of a typical instance’ of HITRUST CSF controls based on specific assurance-related criteria determined by industry to be relevant to relying parties, which included such things as controls that were foundational to good security (e.g., workforce training), controls that were highly correlated with known data breaches (e.g., encryption), and controls broadly relevant to an organization’s entire security program (e.g., risk management). Note the latter are also used to help improve generalization of the results of a HITRUST CSF assessment to an organization’s overall security posture.
However, as all risk factors are applied to each of the controls sampled using this approach, certification was essentially based on an assessment of ‘everything about something’ in the HITRUST CSF. While this did not impact generalizability of the sample, it resulted in some external relying parties requesting additional information about controls that were not addressed in an assessment. (This was especially true for organizations that were seeking assurances regarding compliance with an external standard such as NIST SP 800-53.) HITRUST organizations also saw additional variability in the control requirements specified from one HITRUST CSF release to the next since all relevant requirements in a control are assessed. This made it difficult at times for some organizations to implement new control requirements and obtain recertification before their certification under a prior release expired.
To address these concerns, HITRUST moved to a new purposive sampling strategy that reflects the overall diversity of the HITRUST CSF. Called a ‘purposive sample of heterogenous instances’ (PSI-Het), this approach involves the selection of control requirements that are heterogenous on characteristics that potentially make a difference to our intended inference around an organization’s overall security posture relative to the requested assurances. Focused on control requirements vice controls and, more specifically, on the core good security hygiene and industry best practices-level of requirements available with the release of HITRUST CSF v11, HITRUST CSF certification is now based on an assessment of ‘something about everything.’
This ensures that all types of control requirements will be assessed (e.g., administrative, technical, and physical; preventive, detective, and responsive), and these requirements will be reflective of intended outcomes for all HITRUST CSF controls and control objectives (i.e., the characteristics that potentially impact generalizability). HITRUST organizations should also see fewer changes in their control specification for HITRUST CSF certification as the information protection hygiene practices included in core are less subject to perturbation by changes in supporting authoritative sources.173
For those organizations that only need to demonstrate the implementation of good security hygiene (e.g., based on the amount of risk posed to a relying party by their business relationship), an assessment can be performed against a purposive sample of those control requirements across the breadth of the HITRUST CSF. The same can be done for a higher level of assurance around industry best practices. And, for those organizations that are subject to specific regulations, HITRUST certification may be based on a purposive sample of these core good hygiene and best practice requirements plus any additional control requirements needed to comply with those regulations.
169 Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin Company, p. 24.
170 Ibid., p. 22.
171 Ibid., p. 348.
172 Ibid., pp. 356 – 358.
173 HITRUST views the recent relaxation in NIST SP 800-53 Revision 5 around password controls as ‘the exception that proves the rule.’