> For the complete documentation index, see [llms.txt](https://playbook.sidthoviti.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://playbook.sidthoviti.com/ai-security/llm-security-checklist.md).

# LLM Security Checklist

### **1. OWASP Top 10 for LLM Applications (2025)**

***

#### **1.1 Prompt Injection**

* [ ] Test for **Direct Prompt Injection** where crafted inputs alter behavior unexpectedly.

- ⬇️ **Sample Attack Scenarios**:
  * An attacker injects a prompt in a chatbot to bypass guidelines, query private data stores, and escalate privileges.
  * **Payload splitting:** malicious prompts are fragmented to evade detection but manipulate the LLM when combined.

* [ ] Validate against **Indirect Prompt Injection** by testing inputs from external sources.
  * ⬇️ **Sample Attack Scenarios**:
    * Summarizing a webpage with hidden instructions, causing the LLM to exfiltrate private conversation details.
    * Using Retrieval-Augmented Generation (RAG) to inject modified content in a repository, leading to misleading outputs.
* [ ] Ensure defenses against **Jailbreaking** attempts to bypass safety protocols.
* [ ] Conduct adversarial tests for **Multimodal Prompt Injection** (hidden instructions in images, audio, etc.).
  * ⬇️ **Sample Attack Scenario**:
    * A malicious prompt embedded in an image alters the model’s behavior when processed with text.
* [ ] Evaluate risks of **Adversarial Suffix Attacks** and multilingual/obfuscated input strategies.

***

#### **1.2 Sensitive Information Disclosure**

* [ ] Test for **Training Data Leakage** using specific queries.
* [ ] Validate system prevention of **PII or Confidential Data Extraction**.
  * ⬇️ **Sample Attack Scenario**:
    * An attacker queries the model repeatedly to infer sensitive training data patterns.
* [ ] Verify output sanitization to avoid unintended **System Prompt Disclosure**.

***

#### **1.3 Supply Chain Vulnerabilities**

* [ ] Audit dependencies for vulnerabilities in the **MLOps Pipeline**.
* [ ] Test integrity and authenticity of third-party components in the pipeline.
  * ⬇️ **Sample Attack Scenario**:
    * A compromised pre-trained model dependency introduces malicious behaviors in production.
* [ ] Ensure proper version control and immutability for LLM components.

***

#### **1.4 Data and Model Poisoning**

* [ ] Test for resistance to **Adversarial Training Data Insertion**.
  * ⬇️ **Sample Attack Scenario**:
    * Poisoned training data subtly biases an LLM to produce harmful or incorrect outputs under specific prompts.
* [ ] Monitor for unauthorized modifications of training data.
* [ ] Validate input data integrity during model fine-tuning.

***

#### **1.5 Improper Output Handling**

* [ ] Validate output to ensure compliance with safety and relevance constraints.
  * ⬇️ **Sample Attack Scenario**:
    * An LLM produces responses that violate content policies when queried with edge-case inputs.
* [ ] Test that sensitive or harmful content cannot bypass output filters.

***

#### **1.6 Excessive Agency**

* [ ] Test for improper escalation of **autonomous agent permissions**.
  * ⬇️ **Sample Attack Scenario**:
    * An LLM autonomously escalates privileges to execute unauthorized API calls.
* [ ] Validate agent actions to prevent risky or unintended decisions.

***

#### **1.7 System Prompt Leakage**

* [ ] Verify that system prompts remain inaccessible through direct or indirect queries.
  * ⬇️ **Sample Attack Scenario**:
    * An attacker uses adversarial prompts to infer and extract system-level prompt templates.
* [ ] Monitor for leakage through metadata, logs, or embedded queries.

***

#### **1.8 Vector and Embedding Weaknesses**

* [ ] Test **vector database query security** against unauthorized access.
  * ⬇️ **Sample Attack Scenario**:
    * An attacker exploits embedding similarity searches to infer sensitive stored vectors.
* [ ] Validate embedding sanitization to prevent injection or retrieval flaws.

***

#### **1.9 Misinformation Risks**

* [ ] Test for generation of **factually incorrect or biased outputs**.
  * ⬇️ **Sample Attack Scenario**:
    * An attacker manipulates LLM responses to spread false narratives by exploiting content sourcing flaws.
* [ ] Validate retrieval-augmented generation (RAG) for accurate and grounded sourcing.

***

#### **1.10 Unbounded Consumption**

* [ ] Test for **resource exhaustion vulnerabilities**, including memory and API limits.
  * ⬇️ **Sample Attack Scenario**:
    * Malicious inputs cause an LLM to perform excessive computations, leading to denial-of-service or unexpected costs.
* [ ] Monitor for abusive usage patterns.
* [ ] Test rate-limiting of Models, APIs, etc.

***

### **2. Additional Categories**

#### **2.1 Input and Output Security**

* [ ] Perform extensive **input validation** for injection attacks (e.g., SQL, XSS, command).
* [ ] Ensure outputs are sanitized and properly encoded.
* [ ] Prevent sensitive data from being accidentally returned in outputs.

***

#### **2.2 Orchestrator Security**

* [ ] Enforce **access control policies** (RBAC, ABAC) to restrict orchestrator-level permissions.
* [ ] Test for identity manipulation and unauthorized API calls.
* [ ] Test multi-factor authentication for orchestrator interfaces.

***

#### **2.3 Incident Response and Monitoring**

* [ ] Enable comprehensive logging of interactions for **audit and forensic purposes**.
* [ ] Regularly conduct tabletop exercises to test incident response to LLM-related threats.
* [ ] Create clear post-incident analysis methodologies

## References

* [OWASP Top 10 for LLM Applications 2025 PDF](https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-v2025.pdf)
* [MITRE ATLAS](https://atlas.mitre.org/matrices/ATLAS)
* [Syncubes](https://www.syncubes.com/llm-pentesting-checklist)
* [PortSwigger's recommendations](https://portswigger.net/web-security/llm-attacks).