Harmful code generation

Description:

• Without proper guardrails, generative AI models might generate code that causes harm or unintentionally affects other systems (e.g., via SQLi) or the end user (e.g., via XSS).
  

Impact:

• Varied based on the nature of the harmful code generated.
  

Applies to which types of AI models? Generative AI specifically

Which AI security requirements function against this threat? [?]

• Control function: Corrective
  • Updating incident response for AI specifics
    
    
• Control function: Detective
  • Security evaluations such as AI red teaming
  • Log AI system inputs and outputs
  • Monitor AI system inputs and outputs
    
    
• Control function: Directive
  • Augment written policies to address AI specificities
    
    
• Control function: Preventative
  • Output encoding
  • Input filtering
    
    
• Control function: Variance reduction
  • Assign Roles and Resp. for AI
  • Provide AI security training to AI builders and deployers
    

Discussed in which authoritative sources? [?]

• OWASP AI Exchange
  2024, © The OWASP Foundation
  • Where:
    • 4.4: Insecure output handling
      
      

• OWASP Top 10 for LLM Applications
  Oct. 2023, © The OWASP Foundation
  • Where:
    • LLM02: Insecure output handling
      
      

Discussed in which commercial sources? [?]

• AI Risk Atlas
  2024, © IBM Corporation
  • Where:
    • Harmful code generation risk for AI