Model poisoning

Description: Model poisoning attacks attempt to directly modify the trained AI model to inject malicious functionality into the model. Once trained, a model is often just a file residing on a server. Attackers can alter the model file or replace it entirely with a poisoned model file. In this respect, even if a model has been correctly trained with a dataset that has been thoroughly vetted, this model can still be replaced with a poisoned model at various points in the AI SDLC or in the runtime environment.

Impact: Affects the integrity of model outputs, decisions, or behaviors.

Applies to which types of AI models? Any

Which AI security requirements function against this threat? [?]

• Control function: Corrective
  • Updating incident response for AI specifics
  • Backing up AI system assets
    
    
• Control function: Decision support
  • Identifying security threats to the AI system
  • Threat modeling
  • Security evaluations such as AI red teaming
  • ID and evaluate any constraints on data used for AI
  • ID and evaluate compliance & legal obligations for AI system development and deployment
  • Inventory deployed AI systems
  • Model card publication
  • Linkage between dataset, model, and pipeline config
  • Review the model cards of models used by the AI system
    
    
• Control function: Detective
  • Security evaluations such as AI red teaming
  • Monitoring for data, models, and configs for suspicious changes
  • Log AI system inputs and outputs
  • Monitor AI system inputs and outputs
    
    
• Control function: Directive
  • Augment written policies to address AI specificities
  • Documentation of AI specifics during system design and development
    
    
• Control function: Preventative
  • Verification of origin and integrity of AI assets
  • Restrict access to data used for AI
  • Restrict access to AI models
  • Encrypt AI assets at rest
  • Restrict access to the AI engineering environment and AI code
  • Input filtering
  • Change control over AI models
    
    
• Control function: Resistive
  • Limit the release of technical info about the AI system
    
    
• Control function: Variance reduction
  • Assign Roles and Resp. for AI
  • AI security requirements communicated to AI providers
  • Due diligence review of AI providers
  • Provide AI security training to AI builders and deployers
  • Version control of AI assets
  • Maintain a catalog of trusted data sources for AI
    

Discussed in which authoritative sources? [?]

• Attacking Artificial Intelligence: AI’s Security Vulnerability and What Policymakers Can Do About It
  August 2019, Belfer Center for Science and International Affairs, Harvard Kennedy School
  • Where:
    • Part I. Technical Problem > Poisoning Attacks > Model Poisoning
      
      
• Cybersecurity of AI and Standardization
  March 2023, © European Union Agency for Cybersecurity (ENISA)
  • Where:
    • 4. Analysis of coverage > 4.1. Standardization in support of cybersecurity of AI – Narrow sense
      
      
• MITRE ATLAS
  2024, © The MITRE Corporation
  • Where:
    • AML.T0018: Backdoor ML Model
    • AML.T0018.000: Backdoor ML Model: Poison ML Model
    • AML.T0018.001: Backdoor ML Model: Inject payload
    • AML.T0043.004: Insert Backdoor Trigger
      
      
• Multilayer Framework for Good Cybersecurity Practices for AI
  2023, © European Union Agency for Cybersecurity (ENISA)
  • Where:
    • 2. Framework for good cybersecurity practices for AI > 2.2. Layer II – AI fundamentals and cybersecurity > Poisoning
      
      
• NIST AI 100-2 E2023: Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
  Jan. 2024, National Institute of Standards and Technology (NIST)
  • Where:
    • 2. Predictive AI Taxonomy > 2.3. Poisoning Attacks and Mitigations > 2.3.4. Model Poisoning
      
      
• OWASP Machine Learning Security Top 10
  2023, © The OWASP Foundation
  • Where:
    • ML07: Transfer Learning Attack
    • ML10: Model Poisoning
      
      
• OWASP AI Exchange
  2024, © The OWASP Foundation
  • Where:
    • 3.1. Broad model poisoning development time
    • 3.1.2. Development-environment model poisoning
    • 4.2. Runtime model poisoning
      
      
• OWASP 2025 Top 10 for LLM Applications
  2025, © The OWASP Foundation
  • Where:
    • LLM04: Data and Model Poisoning
      
      
• Securing Artificial Intelligence (SAI); AI Threat Ontology
  2022, © European Telecommunications Standards Institute (ETSI)
  • Where:
    • 6. Threat landscape > 6.4. Threat modeling > 6.4.2.3 > Implementation
      
      
• Securing Machine Learning Algorithms
  2021, © European Union Agency for Cybersecurity (ENISA)
  • Where:
    • 3. ML Threats and Vulnerabilities > 3.1. Identification of Threats > Poisoning
      

Discussed in which commercial sources? [?]

• Databricks AI Security Framework
  Sept. 2024, © Databricks
  • Where:
    • Risks in AI System Components > Algorithms 5.3: Hypermeters stealing
    • Risks in AI System Components > Algorithms 7.1: Backdoor machine learning / Trojaned model
      
      
• HiddenLayer’s 2024 AI Threat Landscape Report
  2024, © HiddenLayer
  • Where:
    • Part 2: Risks faced by AI-based systems > Malicious models
    • Part 2: Risks faced by AI-based systems > Model backdoors
      
      
• Snowflake AI Security Framework
  2024, © Snowflake Inc.
  • Where:
    • Backdooring models (insider attacks)
    • Model poisoning
      
      
• StackAware AI Security Reference
  2024, © StackAware
  • Where:
    • AI Risks > Corrupted model seeding (external)