LLM Security Checklist

A checklist for LLM security inspired by OWASP Top 10 for LLMs (2025)

Test for Direct Prompt Injection where crafted inputs alter behavior unexpectedly.

⬇️ Sample Attack Scenarios:
- An attacker injects a prompt in a chatbot to bypass guidelines, query private data stores, and escalate privileges.
- Payload splitting: malicious prompts are fragmented to evade detection but manipulate the LLM when combined.

Validate against Indirect Prompt Injection by testing inputs from external sources.
- ⬇️ Sample Attack Scenarios:
  Summarizing a webpage with hidden instructions, causing the LLM to exfiltrate private conversation details.
  Using Retrieval-Augmented Generation (RAG) to inject modified content in a repository, leading to misleading outputs.
Ensure defenses against Jailbreaking attempts to bypass safety protocols.
Conduct adversarial tests for Multimodal Prompt Injection (hidden instructions in images, audio, etc.).
- ⬇️ Sample Attack Scenario:
  A malicious prompt embedded in an image alters the model’s behavior when processed with text.
Evaluate risks of Adversarial Suffix Attacks and multilingual/obfuscated input strategies.

Test for Training Data Leakage using specific queries.
Validate system prevention of PII or Confidential Data Extraction.
- ⬇️ Sample Attack Scenario:
  An attacker queries the model repeatedly to infer sensitive training data patterns.
Verify output sanitization to avoid unintended System Prompt Disclosure.

Audit dependencies for vulnerabilities in the MLOps Pipeline.
Test integrity and authenticity of third-party components in the pipeline.
- ⬇️ Sample Attack Scenario:
  A compromised pre-trained model dependency introduces malicious behaviors in production.
Ensure proper version control and immutability for LLM components.

Test for resistance to Adversarial Training Data Insertion.
- ⬇️ Sample Attack Scenario:
  Poisoned training data subtly biases an LLM to produce harmful or incorrect outputs under specific prompts.
Monitor for unauthorized modifications of training data.
Validate input data integrity during model fine-tuning.

Validate output to ensure compliance with safety and relevance constraints.
- ⬇️ Sample Attack Scenario:
  An LLM produces responses that violate content policies when queried with edge-case inputs.
Test that sensitive or harmful content cannot bypass output filters.

Test for improper escalation of autonomous agent permissions.
- ⬇️ Sample Attack Scenario:
  An LLM autonomously escalates privileges to execute unauthorized API calls.
Validate agent actions to prevent risky or unintended decisions.

Verify that system prompts remain inaccessible through direct or indirect queries.
- ⬇️ Sample Attack Scenario:
  An attacker uses adversarial prompts to infer and extract system-level prompt templates.
Monitor for leakage through metadata, logs, or embedded queries.

Test vector database query security against unauthorized access.
- ⬇️ Sample Attack Scenario:
  An attacker exploits embedding similarity searches to infer sensitive stored vectors.
Validate embedding sanitization to prevent injection or retrieval flaws.

Test for generation of factually incorrect or biased outputs.
- ⬇️ Sample Attack Scenario:
  An attacker manipulates LLM responses to spread false narratives by exploiting content sourcing flaws.
Validate retrieval-augmented generation (RAG) for accurate and grounded sourcing.

Test for resource exhaustion vulnerabilities, including memory and API limits.
- ⬇️ Sample Attack Scenario:
  Malicious inputs cause an LLM to perform excessive computations, leading to denial-of-service or unexpected costs.
Monitor for abusive usage patterns.
Test rate-limiting of Models, APIs, etc.

Perform extensive input validation for injection attacks (e.g., SQL, XSS, command).
Ensure outputs are sanitized and properly encoded.
Prevent sensitive data from being accidentally returned in outputs.

Enforce access control policies (RBAC, ABAC) to restrict orchestrator-level permissions.
Test for identity manipulation and unauthorized API calls.
Test multi-factor authentication for orchestrator interfaces.

Enable comprehensive logging of interactions for audit and forensic purposes.
Regularly conduct tabletop exercises to test incident response to LLM-related threats.
Create clear post-incident analysis methodologies

References

Last updated 11 hours ago