Model extraction and theft

Description:

• Model extraction aims to extract model architecture and parameters. (Source: NIST AI 100-2 Glossary)
• Adversaries may extract a functional copy of a private model. (Source: MITRE ATLAS )
  

Impact:

• Seeks to breach the confidentiality of the model itself.
• Model extraction can lead to model stealing, which corresponds to extracting a sufficient amount of data from the model to enable the complete reconstruction of the model. (Source: Wikipedia)
• Adversaries may exfiltrate model artifacts and parameters to steal intellectual property and cause economic harm to the victim organization. (Source: MITRE ATLAS )
  

Applies to which types of AI models? Predictive (non-generative) machine learning models as well as rule-based / heuristic AI models.

Which AI security requirements function against this threat? [?]

• Control function: Corrective
  • Updating incident response for AI specifics
    
    
• Control function: Decision support
  • Identifying security threats to the AI system
  • Threat modeling
  • Security evaluations such as AI red teaming
  • ID and evaluate any constraints on data used for AI
  • ID and evaluate compliance & legal obligations for AI system development and deployment
  • Inventory deployed AI systems
  • Linkage between dataset, model, and pipeline config
    
    
• Control function: Detective
  • Log AI system inputs and outputs
  • Monitor AI system inputs and outputs
    
    
• Control function: Directive
  • Augment written policies to address AI specificities
  • Documentation of AI specifics during system design and development
    
    
• Control function: Preventative
  • Restrict access to AI models
  • Encrypt AI assets at rest
  • Restrict access to the AI engineering environment and AI code
  • Input filtering
  • Restrict access to interact with the AI model
    
    
• Control function: Resistive
  • Limit the release of technical info about the AI system
  • Limit output specificity and precision
  • Model Rate Limiting / Throttling
    
    
• Control function: Variance reduction
  • Assign Roles and Resp. for AI
  • AI security requirements communicated to AI providers
  • Due diligence review of AI providers
  • Provide AI security training to AI builders and deployers
  • Version control of AI assets
  • Change control over AI models
    

Discussed in which authoritative sources? [?]

• CSA Large Language Model (LLM) Threats Taxonomy
  2024, © Cloud Security Alliance
  • Where: 4. LLM Service Threat Categories > 4.4. Model Theft
    
    
• Cybersecurity of AI and Standardization
  March 2023, © European Union Agency for Cybersecurity (ENISA)
  • Where: 4. Analysis of coverage > 4.1. Standardization in support of cybersecurity of AI – Narrow sense
    
    
• Engaging with Artificial Intelligence
  Jan. 2024, Australian Signals Directorate’s Australian Cyber Security Centre (ASD’s ACSC)
  • Where: Challenges when engaging with AI > 5. Model stealing attack
    
    
• ISO/IEC TR 24028:2020: Information technology — Artificial intelligence — Overview of trustworthiness in artificial intelligence
  2020, © International Standards Organization (ISO)/International Electrotechnical Commission (IEC)
  • Where: 8. Vulnerabilities, Risks, and Challenges > 8.2. AI-specific security threats > 8.2.4. Model stealing
    
    
• MITRE ATLAS
  2024, © The MITRE Corporation
  • Where:
    • AML.T0024.002: Exfiltration via ML Inference API: Extract ML Model
    • AML.T0048.004: External Harms: ML Intellectual Property Theft
      
      
• Multilayer Framework for Good Cybersecurity Practices for AI
  2023, © European Union Agency for Cybersecurity (ENISA)
  • Where: 2. Framework for good cybersecurity practices for AI > 2.2. Layer II – AI fundamentals and cybersecurity > Model or data disclosure
    
    
• NIST AI 100-2 E2023: Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
  Jan. 2024, National Institute of Standards and Technology (NIST)
  • Where: 2. Predictive AI Taxonomy > 2.4. Privacy Attacks > 2.4.3. Model Extraction
    
    
• OWASP Top 10 for LLM Applications
  Oct. 2023, © The OWASP Foundation
  • Where: LLM10: Model theft
    
    
• OWASP Machine Learning Security Top 10
  2023, © The OWASP Foundation
  • Where:
    • ML03: Model Inversion Attack
    • ML05: Model theft
      
      
• OWASP AI Exchange
  2024, © The OWASP Foundation
  • Where:
    • 2.4. Model theft through use
    • 3.2.2. Model theft through development-time model parameter leak
    • 4.3. Direct runtime model theft
      
      
• Securing Artificial Intelligence (SAI); AI Threat Ontology
  2022, © European Telecommunications Standards Institute (ETSI)
  • Where: 6. Threat landscape > 6.4. Threat modeling > 6.4.1 > Attacker objectives
    
    
• Securing Machine Learning Algorithms
  2021, © European Union Agency for Cybersecurity (ENISA)
  • Where:
    • 3. ML Threats and Vulnerabilities > 3.1. Identification of Threats > Oracle
    • 3. ML Threats and Vulnerabilities > 3.1. Identification of Threats > Model or Data Disclosure > Model Disclosure
      
      
• Wikipedia.org
  2024, © Wikimedia Foundation
  • Where: Adversarial machine learning > Model extraction
    
    

Discussed in which commercial sources? [?]

• AI Risk Atlas
  2024, © IBM Corporation
  • Where: Extraction attack risk for AI
    
    
• Databricks AI Security Framework
  Sept. 2024, © Databricks
  • Where:
    • Risks in AI System Components > Model 7.2: Model assets leak
    • Risks in AI System Components > Model management 8.2: Model theft
    • Risks in AI System Components > Model serving – Inference requests 9.6: Discover ML model ontology
    • Risks in AI System Components > Model serving – Inference response 10.3: Discover ML model ontology
    • Risks in AI System Components > Model serving – Inference response 10.3: Discover ML model family
      
      
• Failure Modes in Machine Learning
  Nov. 2022, © Microsoft
  • Where: Intentionally-Motivated Failures > Model stealing
    
    
• HiddenLayer’s 2024 AI Threat Landscape Report
  2024, © HiddenLayer
  • Where:
    • Part 2: Risks faced by AI-based systems > Model evasion > Inference attacks
    • Part 2: Risks faced by AI-based systems > Model theft
      
      
• Snowflake AI Security Framework
  2024, © Snowflake Inc.
  • Where: Model stealing