Sensitive information disclosed in output

Description:

• Without proper guardrails, generative AI outputs can contain confidential and/or sensitive information included in the model’s training dataset, RAG data sources, or data residing in data sources that the AI system is connected to (e.g., through language model tools such as agents or plugins). Examples of such information include that which is covered under data protection laws and regulations (e.g., personally identifiable information, protected health information, cardholder data) and corporate secrets.
  

Impact:

• Can lead to a confidentiality breach of sensitive/confidential data included in the model’s training dataset, RAG data sources, or data residing in data sources that the AI system is connected to (e.g., through language model tools such as agents or plugins).
• Can also lead to a confidentiality breach of the system’s metaprompt, which can be an extremely valuable piece intellectual property. Discovering the metaprompt can inform adversaries about the internal workings of the AI application and overall AI system.
  

Applies to which types of AI models? Generative AI specifically

Which AI security requirements function against this threat? [?]

• Control function: Corrective
  • Updating incident response for AI specifics
    
    
• Control function: Decision support
  • Identifying security threats to the AI system
  • Threat modeling
  • Security evaluations such as AI red teaming
  • ID and evaluate any constraints on data used for AI
  • ID and evaluate compliance & legal obligations for AI system development and deployment
  • Inventory deployed AI systems
  • Model card publication
  • AI data and data supply inventory
  • Linkage between dataset, model, and pipeline config
  • Review the model cards of models used by the AI system
    
    
• Control function: Detective
  • Security evaluations such as AI red teaming
  • Log AI system inputs and outputs
  • Monitor AI system inputs and outputs
    
    
• Control function: Directive
  • Augment written policies to address AI specificities
    
    
• Control function: Preventative
  • Data minimization or anonymization
  • GenAI model least privilege
  • Output filtering
    
    
• Control function: Variance reduction
  • Assign Roles and Resp. for AI
  • AI security requirements communicated to AI providers
  • Due diligence review of AI providers
  • Provide AI security training to AI builders and deployers
  • Version control of AI assets
  • Change control over AI models
  • Maintain a catalog of trusted data sources for AI
    

Discussed in which authoritative sources? [?]

• CSA Large Language Model (LLM) Threats Taxonomy
  2024, © Cloud Security Alliance
  • Where:
    • 4. LLM Service Threat Categories > 4.3. Sensitive Data Disclosure
      
      
• Mitigating Artificial Intelligence (AI) Risk: Safety and Security Guidelines for Critical Infrastructure Owners and Operators
  April 2024, © Department of Homeland Security (DHS)
  • Where:
    • Appendix A: Cross-sector AI risks and mitigation strategies > Attacks on AI > Loss of data
      
      
• MITRE ATLAS
  2024, © The MITRE Corporation
  • Where:
    • AML.T0056: LLM Meta Prompt Extraction
    • AML.T0057: LLM Data Leakage
      
      
• Multilayer Framework for Good Cybersecurity Practices for AI
  2023, © European Union Agency for Cybersecurity (ENISA)
  • Where:
    • 2. Framework for good cybersecurity practices for AI > 2.2. Layer II – AI fundamentals and cybersecurity > Model or data disclosure
      
      
• OWASP 2023 Top 10 for LLM Applications
  Oct. 2023, © The OWASP Foundation
  • Where:
    • LLM06: Sensitive Information Disclosure
      
      
• OWASP 2025 Top 10 for LLM Applications
  2025, © The OWASP Foundation
  • Where:
    • LLM02: Sensitive Information Disclosure
    • LLM07: System Prompt Leakage
      
      
• OWASP AI Exchange
  2024, © The OWASP Foundation
  • Where:
    • 2.3.1: Sensitive data output from model
    • 4.5: Leak sensitive input data
      
      
• Securing Machine Learning Algorithms
  2021, © European Union Agency for Cybersecurity (ENISA)
  • Where:
    • 3. ML Threats and Vulnerabilities > 3.1. Identification of Threats > Model or data disclosure
      

Discussed in which commercial sources? [?]

• AI Risk Atlas
  2024, © IBM Corporation
  • Where:
    • Revealing personal information risk for AI
    • Revealing confidential information risk for AI
    • Personal information in data risk for AI
    • Confidential information in data risk for AI
      
      
• The anecdotes AI GRC Toolkit
  2024, © Anecdotes A.I Ltd.
  • Where:
    • The GenAI Risk Register > ANEC-AI-1: Input of unsanitized PII/PHI
    • The GenAI Risk Register > ANEC-AI-4: Input of sensitive business information
      
      
• Databricks AI Security Framework
  Sept. 2024, © Databricks
  • Where:
    • Risks in AI System Components > Model serving – Inference requests 9.10: Accidental exposure of unauthorized data to models
      
      
• Snowflake AI Security Framework
  2024, © Snowflake Inc.
  • Where:
    • Training data leakage
      
      
• StackAware AI Security Reference
  2024, © StackAware
  • Where:
    • AI Risks > Sensitive data generation (external)