Data poisoning

Description: Poisoning attacks in which a part of the training data is under the control of the adversary. Source: NIST AI 100-2 Glossary

Impact: Affects the integrity of model outputs, decisions, or behaviors.

Applies to which types of AI models? Data-driven models (e.g., predictive ML models, generative AI models)

Which AI security requirements function against this threat? [?]

• Control function: Corrective
  • Updating incident response for AI specifics
  • Backing up AI system assets
  • Dataset sanitization
    
    
• Control function: Decision support
  • Identifying security threats to the AI system
  • Threat modeling
  • Security evaluations such as AI red teaming
  • ID and evaluate any constraints on data used for AI
  • ID and evaluate compliance & legal obligations for AI system development and deployment
  • Inventory deployed AI systems
  • AI data and data supply inventory
  • Linkage between dataset, model, and pipeline config
    
    
• Control function: Detective
  • Security evaluations such as AI red teaming
  • Monitoring for data, models, and configs for suspicious changes
  • Log AI system inputs and outputs
  • Monitor AI system inputs and outputs
    
    
• Control function: Directive
  • Augment written policies to address AI specificities
  • Documentation of AI specifics during system design and development
    
    
• Control function: Preventative
  • Verification of origin and integrity of AI assets
  • Restrict access to data used for AI
  • Encrypt AI assets at rest
  • Restrict access to the AI engineering environment and AI code
    
    
• Control function: Resistive
  • Limit the release of technical info about the AI system
  • Additional Training Data Measures
    
    
• Control function: Variance reduction
  • Assign Roles and Resp. for AI
  • Provide AI security training to AI builders and deployers
  • Version control of AI assets
  • Maintain a catalog of trusted data sources for AI
    

Discussed in which authoritative sources? [?]

• AI Risk Atlas
  2024, © IBM Corporation
  • Where:
    • Data poisoning risk for AI
      
      
• Attacking Artificial Intelligence: AI’s Security Vulnerability and What Policymakers Can Do About It
  August 2019, Belfer Center for Science and International Affairs, Harvard Kennedy School
  • Where:
    • Part I. Technical Problem > Poisoning Attacks > Dataset Poisoning
      
      
• CSA Large Language Model (LLM) Threats Taxonomy
  2024, © Cloud Security Alliance
  • Where:
    • 4. LLM Service Threat Categories > 4.2. Data Poisoning
      
      
• Cybersecurity of AI and Standardization
  March 2023, © European Union Agency for Cybersecurity (ENISA)
  • Where:
    • 4. Analysis of coverage > 4.1. Standardization in support of cybersecurity of AI – Narrow sense
      
      
• Engaging with Artificial Intelligence
  Jan. 2024, Australian Signals Directorate’s Australian Cyber Security Centre (ASD’s ACSC)
  • Where:
    • Challenges when engaging with AI > 1. Data poisoning of an AI model
      
      
• ISO/IEC TR 24028:2020: Information technology — Artificial intelligence — Overview of trustworthiness in artificial intelligence
  2020, © International Standards Organization (ISO)/International Electrotechnical Commission (IEC)
  • Where:
    • 8. Vulnerabilities, Risks, and Challenges > 8.2. AI-specific security threats > 8.2.2. Data poisoning
      
      
• Multilayer Framework for Good Cybersecurity Practices for AI
  2023, © European Union Agency for Cybersecurity (ENISA)
  • Where:
    • 2. Framework for good cybersecurity practices for AI > 2.2. Layer II – AI fundamentals and cybersecurity > Poisoning
      
      
• Mitigating Artificial Intelligence (AI) Risk: Safety and Security Guidelines for Critical Infrastructure Owners and Operators
  April 2024, © Department of Homeland Security (DHS)
  • Where:
    • Appendix A: Cross-sector AI risks and mitigation strategies > Risk category: Attacks on AI > Adversarial manipulation of AI algorithms or data
      
      
• MITRE ATLAS
  2024, © The MITRE Corporation
  • Where:
    • AML.T0020: Poison Training Data
    • AML.T0043.004: Insert Backdoor Trigger
      
      
• NIST AI 100-2 E2023: Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
  Jan. 2024, National Institute of Standards and Technology (NIST)
  • Where:
    • 2. Predictive AI Taxonomy > 2.3. Poisoning Attacks and Mitigations > 2.3.1. Availability Poisoning
    • 2. Predictive AI Taxonomy > 2.3. Poisoning Attacks and Mitigations > 2.3.2. Targeted Poisoning
    • 2. Predictive AI Taxonomy > 2.3. Poisoning Attacks and Mitigations > 2.3.3. Backdoor Poisoning
    • 3. Generative AI Taxonomy > 3.2. AI Supply Chain Attacks and Mitigations > 3.2.2. Poisoning Attacks
      
      
• OWASP 2023 Top 10 for LLM Applications
  Oct. 2023, © The OWASP Foundation
  • Where:
    • LLM03: Training Data Poisoning
      
      
• OWASP 2025 Top 10 for LLM Applications
  2025, © The OWASP Foundation
  • Where:
    • LLM04: Data and Model Poisoning
    • LLM08: Vector and Embedding Weaknesses
      
      
• OWASP Machine Learning Security Top 10
  2023, © The OWASP Foundation
  • Where:
    • ML02: Data Poisoning Attack
    • ML08: Model Skewing
      
      
• OWASP AI Exchange
  2024, © The OWASP Foundation
  • Where:
    • 3.1: Broad model poisoning development-time
    • 3.1.1 Data poisoning
      
      
• Securing Artificial Intelligence (SAI); AI Threat Ontology
  2022, © European Telecommunications Standards Institute (ETSI)
  • Where:
    • 6. Threat landscape > 6.4. Threat modeling > 6.4.2 > Data acquistion and curation
      
      
• Securing Machine Learning Algorithms
  2021, © European Union Agency for Cybersecurity (ENISA)
  • Where:
    • 3. ML Threats and Vulnerabilities > 3.1. Identification of Threats > Poisoning
    • 3. ML Threats and Vulnerabilities > 3.1. Identification of Threats > Poisoning > Label Modification
      

Discussed in which commercial sources? [?]

• Databricks AI Security Framework
  Sept. 2024, © Databricks
  • Where:
    • Risks in AI System Components > 2.2 Data Prep
    • Risks in AI System Components > Data Prep 2.3: Raw data criteria
    • Risks in AI System Components > Datasets 3.1: Data poisoning
    • Risks in AI System Components > Evaluation 6.1: Evaluation data poisoning
    • Risks in AI System Components > Model serving – Inference requests 9.9: Input resource control
      
      
• Failure Modes in Machine Learning
  Nov. 2022, © Microsoft
  • Where:
    • Intentionally-Motivated Failures > Poisoning attacks
      
      
• HiddenLayer’s 2024 AI Threat Landscape Report
  2024, © HiddenLayer
  • Where:
    • Part 2: Risks faced by AI-based systems > Data poisoning
      
      
• Snowflake AI Security Framework
  2024, © Snowflake Inc.
  • Where:
    • Training data poisoning
      
      
• StackAware AI Security Reference
  2024, © StackAware
  • Where:
    • AI Risks > Training data poisoning