Evasion (including adversarial examples)

Description:

• Evasion attacks consist of exploiting the imperfection of a trained model. For instance, spammers and hackers often attempt to evade detection by obfuscating the content of spam emails and malware. Samples are modified to evade detection; that is, to be classified as legitimate. This does not involve influence over the training data. A clear example of evasion is image-based spam in which the spam content is embedded within an attached image to evade textual analysis by anti-spam filters. (Source: Wikipedia )
• Evasion attacks attempt fool the AI model through inputs designed to mislead it into performing its task incorrectly.
  

Impact: Affects the integrity of model outputs, decisions, or behaviors.

Applies to which types of AI models? Predictive (non-generative) machine learning models as well as rule-based / heuristic AI models.

Which AI security requirements function against this threat? [?]

• Control function: Corrective
  • Updating incident response for AI specifics
    
    
• Control function: Decision support
  • Identifying security threats to the AI system
  • Threat modeling
  • Security evaluations such as AI red teaming
  • ID and evaluate any constraints on data used for AI
  • ID and evaluate compliance & legal obligations for AI system development and deployment
  • Inventory deployed AI systems
  • Model card publication
  • Linkage between dataset, model, and pipeline config
  • Review the model cards of models used by the AI system
    
    
• Control function: Detective
  • Log AI system inputs and outputs
  • Monitor AI system inputs and outputs
    
    
• Control function: Directive
  • Augment written policies to address AI specificities
  • Documentation of AI specifics during system design and development
    
    
• Control function: Preventative
  • Input filtering
  • Restrict access to interact with the AI model
  • Additional Training Data Measures
    
    
• Control function: Resistive
  • Limit the release of technical info about the AI system
  • Limit output specificity and precision
  • Model Rate Limiting / Throttling
    
    
• Control function: Variance reduction
  • Assign Roles and Resp. for AI
  • AI security requirements communicated to AI providers
  • Due diligence review of AI providers
  • Provide AI security training to AI builders and deployers
  • Version control of AI assets
  • Change control over AI models
    

Discussed in which authoritative sources? [?]

• Cybersecurity of AI and Standardization
  March 2023, © European Union Agency for Cybersecurity (ENISA)
  • Where: 4. Analysis of coverage > 4.1. Standardization in support of cybersecurity of AI – Narrow sense
    
    
• Engaging with Artificial Intelligence
  Jan. 2024, Australian Signals Directorate’s Australian Cyber Security Centre (ASD’s ACSC)
  • Where: Challenges when engaging with AI > 2. Input manipulation attacks – Prompt injection and adversarial examples
    
    
• ISO/IEC TR 24028:2020: Information technology — Artificial intelligence — Overview of trustworthiness in artificial intelligence
  2020, © International Standards Organization (ISO)/International Electrotechnical Commission (IEC)
  • Where: 8. Vulnerabilities, Risks, and Challenges > 8.2. AI-specific security threats > 8.2.3. Adversarial attacks
    
    
• Mitigating Artificial Intelligence (AI) Risk: Safety and Security Guidelines for Critical Infrastructure Owners and Operators
  April 2024, © Department of Homeland Security (DHS)
  • Where: Cross-sector AI risks and mitigation strategies > Risk category: Attacks on AI > Evasion attacks
    
    
• MITRE ATLAS
  2024, © The MITRE Corporation
  • Where:
    • AML.T0015: Evade ML model
    • AML.T0043: Craft Adversarial Data
      
      
• Multilayer Framework for Good Cybersecurity Practices for AI
  2023, © European Union Agency for Cybersecurity (ENISA)
  • Where: 2. Framework for good cybersecurity practices for AI > 2.2. Layer II – AI fundamentals and cybersecurity > AI threat assessment
    
    
• NIST AI 100-2 E2023: Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
  Jan. 2024, National Institute of Standards and Technology (NIST)
  • Where: 2. Predictive AI Taxonomy > 2.1. Evasion attacks
    
    
• OWASP AI Exchange
  2024, © The OWASP Foundation
  • Where: 2.1: Evasion
    
    
• Securing Artificial Intelligence (SAI); AI Threat Ontology
  2022, © European Telecommunications Standards Institute (ETSI)
  • Where: 6. Threat landscape > 6.4. Threat modeling > 6.4.1 > Attacker objectives
    
    
• Securing Machine Learning Algorithms
  2021, © European Union Agency for Cybersecurity (ENISA)
  • Where: 3. ML Threats and Vulnerabilities > 3.1. Identification of Threats > Evasion
    
    
• Wikipedia.org
  2024, © Wikimedia Foundation
  • Where:
    • Adversarial machine learning > Evasion
    • Adversarial machine learning > Adversarial examples
      

Discussed in which commercial sources? [?]

• AI Risk Atlas
  2024, © IBM Corporation
  • Where: Evasion attack risk for AI
    
    
• Databricks AI Security Framework
  Sept. 2024, © Databricks
  • Where: Risks in AI System Components > Model serving – Inference requests 9.3: Model breakout
    
    
• Failure Modes in Machine Learning
  Nov. 2022, © Microsoft
  • Where:
    • Intentionally-Motivated Failures > Perturbation attack
    • Intentionally-Motivated Failures > Adversarial example in the physical domain
      
      
• HiddenLayer’s 2024 AI Threat Landscape Report
  2024, © HiddenLayer
  • Where: Part 2: Risks faced by AI-based systems > Model evasion > Evasion attacks
    
    
• Snowflake AI Security Framework
  2024, © Snowflake Inc.
  • Where:
    • Adversarial samples
    • Fuzzing