Excessive agency

Description:

• Generative AI systems may undertake actions outside of the developer intent, organizational policy, and/or legislative, regulatory, and contractual requirements, leading to unintended consequences. This issue is facilitated by excessive permissions, excessive functionality, excessive autonomy, poorly defined operational parameters or granting the AI system the ability to make decisions or act without human intervention or oversight.
  

Impact:

• Heavily dependent on which systems the overall AI system is connected to and can interact with (e.g., messaging systems, file servers, command prompts). Can lead to confidentiality, availability, or integrity issues.
  

Applies to which types of AI models? Generative AI specifically

Which AI security requirements function against this threat? [?]

• Control function: Corrective
  • Updating incident response for AI specifics
    
    
• Control function: Decision support
  • Identifying security threats to the AI system
  • Threat modeling
  • Security evaluations such as AI red teaming
  • ID and evaluate any constraints on data used for AI
  • ID and evaluate compliance & legal obligations for AI system development and deployment
  • Inventory deployed AI systems
  • Model card publication
  • Linkage between dataset, model, and pipeline config
  • Review the model cards of models used by the AI system
    
    
• Control function: Detective
  • Log AI system inputs and outputs
  • Monitor AI system inputs and outputs
    
    
• Control function: Directive
  • Augment written policies to address AI specificities
  • Documentation of AI specifics during system design and development
    
    
• Control function: Preventative
  • Humans can intervene if needed
  • Change control over language model tools
  • GenAI model least privilege
    
    
• Control function: Resistive
  • Model Rate Limiting / Throttling
    
    
• Control function: Variance reduction
  • Assign Roles and Resp. for AI
  • Provide AI security training to AI builders and deployers
  • Version control of AI assets
  • Change control over AI models
    

Discussed in which authoritative sources? [?]

• Mitigating Artificial Intelligence (AI) Risk: Safety and Security Guidelines for Critical Infrastructure Owners and Operators
  April 2024, © Department of Homeland Security (DHS)
  • Where:
    • Cross-sector AI risks and mitigation strategies > Risk category: AI design and implementation failures > Autonomy
      
      
• OWASP 2023 Top 10 for LLM Applications
  Oct. 2023, © The OWASP Foundation
  • Where:
    • LLM08: Excessive Agency
      
      
• OWASP 2025 Top 10 for LLM Applications
  2025, © The OWASP Foundation
  • Where:
    • LLM06: Excessive Agency
      
      
• OWASP AI Exchange
  2024, © The OWASP Foundation
  • Where:
    • 1.3. Controls to limit the effects of unwanted behavior
      

Discussed in which commercial sources? [?]

• AI Risk Atlas
  2024, © IBM Corporation
  • Where:
    • Hallucination risk for AI
      
      

Additional information

• See this post for an overview of the difference between autonomy and agency, paraphrased as follows:
  • Autonomy, in the context of technology, generally refers to the ability to perform tasks without human intervention. The expansion of autonomy could indeed set the stage for the emergence of agency. When a system gains the capability to perform a network of tasks (constituting a decision situation) autonomously, it could be seen as a foundation upon which agency might build.
  • Agency implies a higher order of function—not just carrying out tasks, but also making choices about which tasks to undertake and when.