> For the complete documentation index, see [llms.txt](https://playbook.sidthoviti.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://playbook.sidthoviti.com/ai-security/old-drafts/llm-security2.md).

# LLM Security2

Checklist for LLM Security

### Threat Modeling for LLMs

* [ ]  **Hyper-Personalized Attacks:** Assess how attackers might use Generative AI for more targeted spear phishing or social engineering.
* [ ]  **Customer Impersonation Risks:** Evaluate risks of GenAI-generated content being used for attacks targeting customers or clients.
* [ ]  **Malicious Input Detection:** Develop mechanisms to detect and neutralize harmful or malicious inputs to the LLM.
* [ ]  **Secure Integration:** Ensure secure connections between LLMs and other systems, with proper safeguards at all trust boundaries.
* [ ]  **Insider Threat Management:** Implement strategies to prevent misuse by authorized users.
* [ ]  **Intellectual Property Protection:** Prevent unauthorized access to proprietary models or data.
* [ ]  **Automated Content Filtering:** Implement automated methods to prevent generation of harmful or inappropriate content.
* [ ]  **Metrics for AI Evaluation:** Define metrics to measure AI performance, productivity, and resilience against other cybersecurity methods.

### Secure Implementation of LLM Solutions

* [ ]  **Threat Modeling of LLM Components:** Map out trust boundaries in the architecture and identify potential risks.
* [ ]  **Data Security:** Verify data classification and protection measures, including how sensitive data is managed.
* [ ]  **Access Control:** Use least privilege principles and implement defense-in-depth strategies.
* [ ]  **Training Pipeline Security:** Control training data governance and ensure secure pipelines, models, and algorithms.
* [ ]  **Input & Output Security:** Validate inputs and sanitize outputs to prevent harmful data from being processed or generated.
* [ ]  **Monitoring & Response:** Ensure automation, logging, and auditing capabilities, with secure storage of audit records.
* [ ]  **Testing & Review:** Include application testing, source code review, vulnerability assessments, and red teaming before release.
* [ ]  **Supply Chain Security:** Perform third-party audits and code reviews for external providers and dependencies.
* [ ]  **Infrastructure Security:** Assess vendor resilience testing frequency, availability, scalability, and performance SLAs.
* [ ]  **Incident Response Drills:** Include LLM-specific incidents in tabletop exercises and update playbooks accordingly.

### LLM-Specific Vulnerabilities (OWASP-inspired)

#### Prompt Injection Attacks

* [ ]  **Direct Prompt Injection:** Test for direct prompt injection by attempting to bypass system prompts.
* [ ]  **Indirect Prompt Injection:** Attempt indirect prompt injection by manipulating data sources that feed into the LLM.
* [ ]  **System Instruction Manipulation:** Try to override or modify system instructions within user inputs.
* [ ]  **Jailbreaking Attempts:** Test for attempts to bypass ethical guidelines or content restrictions.

#### Authorization Bypass

* [ ]  **Access Control Testing:** Attempt to access data or perform actions beyond the user's authorized scope.
* [ ]  **Unauthorized API Calls:** Test if the LLM can be tricked into making unauthorized API calls.
* [ ]  **Sensitive Data Extraction:** Check if sensitive information can be extracted through carefully crafted prompts.
* [ ]  **Role-Based Access:** Verify if the LLM respects user roles and permissions in its responses.

#### Data Leakage

* [ ]  **Training Data Exposure:** Probe for potential exposure of training data through specific queries.
* [ ]  **Sensitive Information Disclosure:** Test if personal or sensitive information can be extracted from the model.
* [ ]  **System Architecture Disclosure:** Check for unintended disclosure of system architecture or backend details.
* [ ]  **Cache Security:** Attempt to retrieve information from LLM caches that should be access-controlled.

#### Input Validation and Sanitization

* [ ]  **SQL Injection Testing:** Test for SQL injection in LLM-generated database queries.
* [ ]  **XSS Vulnerability Testing:** Attempt XSS attacks through LLM-generated outputs.
* [ ]  **Command Injection Testing:** Check for command injection possibilities in LLM-processed inputs.
* [ ]  **Special Character Handling:** Verify proper handling and escaping of special characters.

### Vector Database Security

* [ ]  **Access Control Verification:** Test access controls on vector database queries.
* [ ]  **Document-Level Security:** Attempt to bypass document-level security in vector stores.
* [ ]  **Data Leakage in Similarity Searches:** Check for potential data leakage through similarity searches.
* [ ]  **ACL Synchronization:** Verify proper synchronization of ACLs between source systems and vector databases.

### API and External Service Interactions

* [ ]  **Unauthorized API Requests:** Test for unauthorized API calls through LLM-generated requests.
* [ ]  **API Parameter Manipulation:** Attempt to manipulate API parameters to gain elevated privileges.
* [ ]  **Confused Deputy Attacks:** Check for potential confused deputy attacks in multi-system interactions.
* [ ]  **Identity Propagation:** Verify proper identity propagation in API calls made by the orchestrator.

### LLM-Generated Code Execution

* [ ]  **Sandbox Escape Testing:** Test sandbox escape attempts in environments running LLM-generated code.
* [ ]  **Malicious Code Injection:** Attempt to inject malicious code through crafted prompts.
* [ ]  **Unauthorized Imports:** Check for unauthorized library imports or function calls in generated code.
* [ ]  **Resource Limits:** Verify resource usage limits and execution timeouts.

### Memory and Context Manipulation

* [ ]  **Memory Poisoning:** Attempt to poison the LLM's short-term or long-term memory.
* [ ]  **Context Leakage:** Test for context leakage between different user sessions.
* [ ]  **Context Window Manipulation:** Try to manipulate the context window to gain unauthorized information.
* [ ]  **Sensitive Data Clearing:** Check for proper clearing of sensitive data from the LLM's working memory.

### Autonomous Agent Vulnerabilities

* [ ]  **Unauthorized Actions:** Test for unauthorized actions in multi-agent systems.
* [ ]  **Decision-Making Manipulation:** Attempt to manipulate agent decision-making processes.
* [ ]  **Inter-Agent Data Leakage:** Check for potential data leakage between collaborating agents.
* [ ]  **Agent Communication Controls:** Verify proper access controls in agent-to-agent communications.

### MLOps Pipeline Security

* [ ]  **Training Data Poisoning:** Attempt to poison training data used for model fine-tuning.
* [ ]  **Model Versioning Security:** Test for unauthorized access to model versioning and deployment systems.
* [ ]  **Supply Chain Vulnerabilities:** Check for potential supply chain vulnerabilities in the ML pipeline.
* [ ]  **Training Artifacts Access Control:** Verify proper access controls on training logs and model artifacts.

### Orchestrator Security

* [ ]  **Authorization Bypass:** Test for potential bypass of orchestrator-level authorization checks.
* [ ]  **Identity Manipulation:** Attempt to manipulate identity information passed by the orchestrator.
* [ ]  **Error Handling:** Check for proper handling of errors and edge cases in the orchestration layer.
* [ ]  **Cache Security:** Verify secure implementation of any caching mechanisms in the orchestrator.

### Output Validation and Filtering

* [ ]  **Harmful Content Filtering:** Test if malicious or sensitive content can bypass output filters.
* [ ]  **Manipulation of Outputs:** Attempt to trick the system into generating harmful or inappropriate responses.
* [ ]  **Sensitive Data Filtering:** Check for potential data leakage through carefully crafted output requests.
* [ ]  **Handling PII:** Verify proper handling of PII and other sensitive information in LLM outputs.