Building Security Using Generative AI

OWASP Top 10 Risks for LLM Applications

• LLM01 Prompt Injection: Attackers can manipulate LLMs through carefully crafted prompts or inputs, causing the LLM to execute unintended actions. This could lead to unauthorized access, data breaches, or the generation of misleading or harmful content.
  For example, An attacker might craft a prompt that tricks the LLM into revealing sensitive information or performing actions that violate security policies.
• LLM02 Insecure Output Handling: Treating LLM outputs as completely reliable can be dangerous. If outputs are not carefully validated and sanitized, they could be used for downstream attacks like code injection or social engineering scams.
  For example, An LLM used to generate marketing copy might create content containing malicious scripts if the output isn't properly reviewed and filtered.
• LLM03 Training Data Poisoning: The quality and security of an LLM heavily depend on the data it is trained on. Attackers might try to inject biased or poisoned data into the training data, influencing the LLM's behavior and potentially causing it to generate harmful content or perpetuate biases.
  For example, Training an LLM on a dataset containing a lot of racist or offensive language could lead to the model generating similar outputs, perpetuating hate speech.
• LLM04 Model Denial-of-Service (DoS): Attackers might attempt to overload an LLM with excessive requests, causing it to become unavailable to legitimate users. This could disrupt critical services or applications that rely on the LLM.
  For example, A coordinated attack flooding an LLM with millions of useless prompts could exhaust its resources and prevent legitimate users from accessing its functionality.
• LLM05 Supply Chain Vulnerabilities: LLMs often rely on external libraries, frameworks, and other components. Vulnerabilities in these dependencies can be exploited to compromise the LLM itself or the applications that use it.
  For example, An LLM that uses a text processing library with a critical security flaw might be susceptible to attacks that exploit that vulnerability.
• LLM06 Sensitive Information Disclosure: LLMs can inadvertently disclose sensitive information if they are not properly configured or if they are trained on data containing confidential details.
  For example, An LLM used for customer service chatbots might accidentally reveal a customer's credit card number if it wasn't trained to handle such information securely.
• LLM07 Insecure Plugin Design: Many platforms allow extending LLM functionality with plugins. Poorly designed or insecure plugins can introduce vulnerabilities that attackers can exploit to compromise the entire system.
  For example, A malicious plugin for an LLM used for content generation might inject malware into the generated content without detection.
• LLM08 Excessive Agency: Assigning too much autonomy or decision-making power to an LLM can be risky. LLMs can make mistakes or be biased based on their training data. Human oversight and control mechanisms are crucial.
  For example, An LLM used for automated financial trading decisions might make risky investments due to limitations in its understanding of the financial market.
• LLM09 Overreliance: Solely relying on LLMs for critical tasks without proper validation and human involvement can be dangerous. Overreliance can lead to missed errors, biases, and security vulnerabilities.
  For example, Using an LLM to write legal documents without human review could result in inaccurate or legally problematic contracts.
• LLM10 Model Theft: LLMs represent valuable intellectual property. Measures need to be in place to prevent unauthorized access, copying, or theft of the LLM model itself.
  For example, An attacker might steal a trained LLM model and use it for malicious purposes, such as generating fake news or creating deepfakes.