# LLM Security2 ### Threat Modeling for LLMs * [ ] **Hyper-Personalized Attacks:** Assess how attackers might use Generative AI for more targeted spear phishing or social engineering. * [ ] **Customer Impersonation Risks:** Evaluate risks of GenAI-generated content being used for attacks targeting customers or clients. * [ ] **Malicious Input Detection:** Develop mechanisms to detect and neutralize harmful or malicious inputs to the LLM. * [ ] **Secure Integration:** Ensure secure connections between LLMs and other systems, with proper safeguards at all trust boundaries. * [ ] **Insider Threat Management:** Implement strategies to prevent misuse by authorized users. * [ ] **Intellectual Property Protection:** Prevent unauthorized access to proprietary models or data. * [ ] **Automated Content Filtering:** Implement automated methods to prevent generation of harmful or inappropriate content. * [ ] **Metrics for AI Evaluation:** Define metrics to measure AI performance, productivity, and resilience against other cybersecurity methods. ### Secure Implementation of LLM Solutions * [ ] **Threat Modeling of LLM Components:** Map out trust boundaries in the architecture and identify potential risks. * [ ] **Data Security:** Verify data classification and protection measures, including how sensitive data is managed. * [ ] **Access Control:** Use least privilege principles and implement defense-in-depth strategies. * [ ] **Training Pipeline Security:** Control training data governance and ensure secure pipelines, models, and algorithms. * [ ] **Input & Output Security:** Validate inputs and sanitize outputs to prevent harmful data from being processed or generated. * [ ] **Monitoring & Response:** Ensure automation, logging, and auditing capabilities, with secure storage of audit records. * [ ] **Testing & Review:** Include application testing, source code review, vulnerability assessments, and red teaming before release. * [ ] **Supply Chain Security:** Perform third-party audits and code reviews for external providers and dependencies. * [ ] **Infrastructure Security:** Assess vendor resilience testing frequency, availability, scalability, and performance SLAs. * [ ] **Incident Response Drills:** Include LLM-specific incidents in tabletop exercises and update playbooks accordingly. ### LLM-Specific Vulnerabilities (OWASP-inspired) #### Prompt Injection Attacks * [ ] **Direct Prompt Injection:** Test for direct prompt injection by attempting to bypass system prompts. * [ ] **Indirect Prompt Injection:** Attempt indirect prompt injection by manipulating data sources that feed into the LLM. * [ ] **System Instruction Manipulation:** Try to override or modify system instructions within user inputs. * [ ] **Jailbreaking Attempts:** Test for attempts to bypass ethical guidelines or content restrictions. #### Authorization Bypass * [ ] **Access Control Testing:** Attempt to access data or perform actions beyond the user's authorized scope. * [ ] **Unauthorized API Calls:** Test if the LLM can be tricked into making unauthorized API calls. * [ ] **Sensitive Data Extraction:** Check if sensitive information can be extracted through carefully crafted prompts. * [ ] **Role-Based Access:** Verify if the LLM respects user roles and permissions in its responses. #### Data Leakage * [ ] **Training Data Exposure:** Probe for potential exposure of training data through specific queries. * [ ] **Sensitive Information Disclosure:** Test if personal or sensitive information can be extracted from the model. * [ ] **System Architecture Disclosure:** Check for unintended disclosure of system architecture or backend details. * [ ] **Cache Security:** Attempt to retrieve information from LLM caches that should be access-controlled. #### Input Validation and Sanitization * [ ] **SQL Injection Testing:** Test for SQL injection in LLM-generated database queries. * [ ] **XSS Vulnerability Testing:** Attempt XSS attacks through LLM-generated outputs. * [ ] **Command Injection Testing:** Check for command injection possibilities in LLM-processed inputs. * [ ] **Special Character Handling:** Verify proper handling and escaping of special characters. ### Vector Database Security * [ ] **Access Control Verification:** Test access controls on vector database queries. * [ ] **Document-Level Security:** Attempt to bypass document-level security in vector stores. * [ ] **Data Leakage in Similarity Searches:** Check for potential data leakage through similarity searches. * [ ] **ACL Synchronization:** Verify proper synchronization of ACLs between source systems and vector databases. ### API and External Service Interactions * [ ] **Unauthorized API Requests:** Test for unauthorized API calls through LLM-generated requests. * [ ] **API Parameter Manipulation:** Attempt to manipulate API parameters to gain elevated privileges. * [ ] **Confused Deputy Attacks:** Check for potential confused deputy attacks in multi-system interactions. * [ ] **Identity Propagation:** Verify proper identity propagation in API calls made by the orchestrator. ### LLM-Generated Code Execution * [ ] **Sandbox Escape Testing:** Test sandbox escape attempts in environments running LLM-generated code. * [ ] **Malicious Code Injection:** Attempt to inject malicious code through crafted prompts. * [ ] **Unauthorized Imports:** Check for unauthorized library imports or function calls in generated code. * [ ] **Resource Limits:** Verify resource usage limits and execution timeouts. ### Memory and Context Manipulation * [ ] **Memory Poisoning:** Attempt to poison the LLM's short-term or long-term memory. * [ ] **Context Leakage:** Test for context leakage between different user sessions. * [ ] **Context Window Manipulation:** Try to manipulate the context window to gain unauthorized information. * [ ] **Sensitive Data Clearing:** Check for proper clearing of sensitive data from the LLM's working memory. ### Autonomous Agent Vulnerabilities * [ ] **Unauthorized Actions:** Test for unauthorized actions in multi-agent systems. * [ ] **Decision-Making Manipulation:** Attempt to manipulate agent decision-making processes. * [ ] **Inter-Agent Data Leakage:** Check for potential data leakage between collaborating agents. * [ ] **Agent Communication Controls:** Verify proper access controls in agent-to-agent communications. ### MLOps Pipeline Security * [ ] **Training Data Poisoning:** Attempt to poison training data used for model fine-tuning. * [ ] **Model Versioning Security:** Test for unauthorized access to model versioning and deployment systems. * [ ] **Supply Chain Vulnerabilities:** Check for potential supply chain vulnerabilities in the ML pipeline. * [ ] **Training Artifacts Access Control:** Verify proper access controls on training logs and model artifacts. ### Orchestrator Security * [ ] **Authorization Bypass:** Test for potential bypass of orchestrator-level authorization checks. * [ ] **Identity Manipulation:** Attempt to manipulate identity information passed by the orchestrator. * [ ] **Error Handling:** Check for proper handling of errors and edge cases in the orchestration layer. * [ ] **Cache Security:** Verify secure implementation of any caching mechanisms in the orchestrator. ### Output Validation and Filtering * [ ] **Harmful Content Filtering:** Test if malicious or sensitive content can bypass output filters. * [ ] **Manipulation of Outputs:** Attempt to trick the system into generating harmful or inappropriate responses. * [ ] **Sensitive Data Filtering:** Check for potential data leakage through carefully crafted output requests. * [ ] **Handling PII:** Verify proper handling of PII and other sensitive information in LLM outputs. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://playbook.sidthoviti.com/ai-security/old-drafts/llm-security2.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.