Compromised 3rd-party models or code

Description:

• Attacks that take advantage of compromised or vulnerable ML software packages and third-party pre-trained models used for fine tuning, plugins or extensions, including outdated or deprecated models or components.
  

Impact:

• Use of models or AI software packages poisoned upstream in the AI supply chain can lead to integrity issues such as biased outcomes, confidentially issues such as redirecting AI system outputs or loss of API keys, or even availability issues like outage of the AI system. The impact depends heavily on the context of the overall AI system.
  

Applies to which types of AI models? Any

Which AI security requirements function against this threat? [?]

• Control function: Corrective
  • Updating incident response for AI specifics
  • Backing up AI system assets
    
    
• Control function: Decision support
  • Identifying security threats to the AI system
  • Threat modeling
  • Security evaluations such as AI red teaming
  • ID and evaluate any constraints on data used for AI
  • ID and evaluate compliance & legal obligations for AI system development and deployment
  • Inventory deployed AI systems
  • Model card publication
  • Linkage between dataset, model, and pipeline config
  • Review the model cards of models used by the AI system
    
    
• Control function: Detective
  • Security evaluations such as AI red teaming
  • Log AI system inputs and outputs
  • Monitor AI system inputs and outputs
    
    
• Control function: Directive
  • Augment written policies to address AI specificities
  • Documentation of AI specifics during system design and development
    
    
• Control function: Preventative
  • Inspection of AI software assets
  • Verification of origin and integrity of AI assets
  • AI security requirements communicated to AI providers
  • Due diligence review of AI providers
    
    
• Control function: Resistive
  • Limit the release of technical info about the AI system
    
    
• Control function: Variance reduction
  • Assign Roles and Resp. for AI
  • Provide AI security training to AI builders and deployers
  • Version control of AI assets
    

Discussed in which authoritative sources? [?]

• CSA Large Language Model (LLM) Threats Taxonomy
  2024, © Cloud Security Alliance
  • Where:
    • 4. LLM Service Threat Categories > 4.6. Insecure Supply Chain
    • 4. LLM Service Threat Categories > 4.7. Insecure Apps/Plugins
      
      
• Mitigating Artificial Intelligence (AI) Risk: Safety and Security Guidelines for Critical Infrastructure Owners and Operators
  April 2024, © Department of Homeland Security (DHS)
  • Where:
    • Appendix A: Cross-sector AI risks and mitigation strategies > Risk category: AI design and implementation failures > Supply chain vulnerabilities
    • Appendix A: Cross-sector AI risks and mitigation strategies > Risk category: Attacks on AI > Adversarial manipulation of AI algorithms or data
      
      
• MITRE ATLAS
  2024, © The MITRE Corporation
  • Where:
    • AML.T0010: ML Supply Chain Compromise
    • AML.T0010.001: ML Supply Chain Compromise: ML Software
    • AML.T0010.002: ML Supply Chain Compromise: Data
    • AML.T0010.003: ML Supply Chain Compromise: Model
      
      
• Multilayer Framework for Good Cybersecurity Practices for AI
  2023, © European Union Agency for Cybersecurity (ENISA)
  • Where:
    • 2. Framework for good cybersecurity practices for AI > 2.2. Layer II – AI fundamentals and cybersecurity > Compromise of ML application components
      
      
• OWASP 2023 Top 10 for LLM Applications
  Oct. 2023, © The OWASP Foundation
  • Where:
    • LLM05: Supply Chain Vulnerabilities
      
      
• OWASP 2025 Top 10 for LLM Applications
  Oct. 2025, © The OWASP Foundation
  • Where:
    • LLM03: Supply Chain
    • LLM04: Data and Model Poisoning
      
      
• OWASP Machine Learning Security Top 10
  2023, © The OWASP Foundation
  • Where:
    • ML06: AI Supply Chain Attacks
      
      
• OWASP AI Exchange
  2024, © The OWASP Foundation
  • Where:
    • #SUPPLYCHAINMANAGE
      
      
• Securing Artificial Intelligence (SAI); AI Threat Ontology
  2022, © European Telecommunications Standards Institute (ETSI)
  • Where:
    • 6. Threat landscape > 6.4. Threat modeling > 6.4.2.3 > Implementation
      
      
• Securing Machine Learning Algorithms
  2021, © European Union Agency for Cybersecurity (ENISA)
  • Where:
    • 3. ML Threats and Vulnerabilities > 3.1. Identification of Threats > Compromise of ML Application and Components
      
      

Discussed in which commercial sources? [?]

• AI Risk Atlas
  2024, © IBM Corporation
  • Where:
    • AML.T0010: ML Supply Chain Compromise
    • AML.T0010.001: ML Supply Chain Compromise: ML Software
    • AML.T0010.002: ML Supply Chain Compromise: Data
    • AML.T0010.003: ML Supply Chain Compromise: Model
      
      
• Databricks AI Security Framework
  Sept. 2024, © Databricks
  • Where:
    • Risks in AI System Components > Algorithms 5.4: Malicious libraries
    • Risks in AI System Components > Model 7.3: ML supply chain vulnerabilities
    • Risks in AI System Components > Model 7.4: Source code control attack
      
      
• Failure Modes in Machine Learning
  Nov. 2022, © Microsoft
  • Where:
    • Intentionally-Motivated Failures > Attacking the ML supply chain
    • Intentionally-Motivated Failures > Backdoor Machine Learning
      
      
• HiddenLayer’s 2024 AI Threat Landscape Report
  2024, © HiddenLayer
  • Where:
    • Part 2: Risks faced by AI-based systems > Supply chain attacks
    • Part 2: Risks faced by AI-based systems > Security of public model repositories
      
      
• Snowflake AI Security Framework
  2024, © Snowflake Inc.
  • Where:
    • Self-hosted OSS LLMs Security