Confabulation

Description:

• The production of confidently stated but incorrect content by which users or developers may be misled or deceived. Colloquially as AI “hallucinations” or “fabrications”.
  

Impact:

• Inaccurate output (an integrity issue), the impact of which varies greatly depending on the context. The issue is exacerbated through overreliance on the AI system.
  

Applies to which types of AI models? Generative AI specifically

Which AI security requirements function against this threat? [?]

• Control function: Corrective
  • Updating incident response for AI specifics
  • Humans can intervene if needed
    
    
• Control function: Decision support
  • Identifying security threats to the AI system
  • Threat modeling
  • Security evaluations such as AI red teaming
  • ID and evaluate any constraints on data used for AI
  • ID and evaluate compliance & legal obligations for AI system development and deployment
  • Inventory deployed AI systems
  • Model card publication
  • Linkage between dataset, model, and pipeline config
  • Review the model cards of models used by the AI system
    
    
• Control function: Detective
  • Security evaluations such as AI red teaming
  • Log AI system inputs and outputs
  • Monitor AI system inputs and outputs
    
    
• Control function: Directive
  • Augment written policies to address AI specificities
  • Documentation of AI specifics during system design and development
    
    
• Control function: Variance reduction
  • Assign Roles and Resp. for AI
  • Provide AI security training to AI builders and deployers
  • Version control of AI assets
  • Change control over AI models
    

Discussed in which authoritative sources? [?]

• CSA Large Language Model (LLM) Threats Taxonomy
  2024, © Cloud Security Alliance
  • Where:
    • 4. LLM Service Threat Categories > 4.5. Model Failure/Malfunctioning
      
      
• Engaging with Artificial Intelligence
  Jan. 2024, Australian Signals Directorate’s Australian Cyber Security Centre (ASD’s ACSC)
  • Where:
    • Challenges when engaging with AI > 3. Generative AI hallucinations
      
      
• Mitigating Artificial Intelligence (AI) Risk: Safety and Security Guidelines for Critical Infrastructure Owners and Operators
  April 2024, © Department of Homeland Security (DHS)
  • Where:
    • Cross-sector AI risks and mitigation strategies > Risk category: AI design and implementation failures > Brittleness
    • Cross-sector AI risks and mitigation strategies > Risk category: AI design and implementation failures > Inadvertent Systemic and Design Flaws
      
      
• NIST AI 600-1:Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
  2024, © National Institute of Standards and Technology (NIST)
  • Where:
    • 2. Overview of Risks Unique to or Exacerbated by GAI > 2.2. Confabulation
      
      
• OWASP 2023 Top 10 for LLM Applications
  Oct. 2023, © The OWASP Foundation
  • Where:
    • LLM09: Overreliance
      
      
• OWASP 2025 Top 10 for LLM Applications
  2025, © The OWASP Foundation
  • Where:
    • LLM09: Misinformation
      
      
• Securing Machine Learning Algorithms
  2021, © European Union Agency for Cybersecurity (ENISA)
  • Where:
    • 3. ML Threats and Vulnerabilities > 3.1. Identification of Threats > Failure or malfunction of ML application
      
      

Discussed in which commercial sources? [?]

• AI Risk Atlas
  2024, © IBM Corporation
  • Where:
    • Hallucination risk for AI
      
      
• Databricks AI Security Framework
  Sept. 2024, © Databricks
  • Where:
    • Risks in AI System Components > Model serving – Inference requests 9.8: LLM hallucinations
      
      
• StackAware AI Security Reference
  2024, © StackAware
  • Where:
    • AI Risks > Unwitting inaccurate output
      

Additional information

• Included here is the threat hallucinated package squatting.

• HITRUST is intentionally focusing on the threat of LLM confabulation (which is almost always undesired) instead of hallucination (which is often a feature—not a bug—of stochastic systems).
  • See this document further discussing the difference between these related by distinct concepts in the context of generative AI.
  • This distinction is also addressed in NIST AI 600-1 which states, “Some commenters have noted that the terms hallucination and fabrication anthropomorphize GAI, which itself is a risk related to GAI systems as it can inappropriately attribute human characteristics to non-human entities.”