LLM Security2

Checklist for LLM Security

Threat Modeling for LLMs

• Hyper-Personalized Attacks: Assess how attackers might use Generative AI for more targeted spear phishing or social engineering.

• Customer Impersonation Risks: Evaluate risks of GenAI-generated content being used for attacks targeting customers or clients.

• Malicious Input Detection: Develop mechanisms to detect and neutralize harmful or malicious inputs to the LLM.

• Secure Integration: Ensure secure connections between LLMs and other systems, with proper safeguards at all trust boundaries.

• Insider Threat Management: Implement strategies to prevent misuse by authorized users.

• Intellectual Property Protection: Prevent unauthorized access to proprietary models or data.

• Automated Content Filtering: Implement automated methods to prevent generation of harmful or inappropriate content.

• Metrics for AI Evaluation: Define metrics to measure AI performance, productivity, and resilience against other cybersecurity methods.

Secure Implementation of LLM Solutions

• Threat Modeling of LLM Components: Map out trust boundaries in the architecture and identify potential risks.

• Data Security: Verify data classification and protection measures, including how sensitive data is managed.

• Access Control: Use least privilege principles and implement defense-in-depth strategies.

• Training Pipeline Security: Control training data governance and ensure secure pipelines, models, and algorithms.

• Input & Output Security: Validate inputs and sanitize outputs to prevent harmful data from being processed or generated.

• Monitoring & Response: Ensure automation, logging, and auditing capabilities, with secure storage of audit records.

• Testing & Review: Include application testing, source code review, vulnerability assessments, and red teaming before release.

• Supply Chain Security: Perform third-party audits and code reviews for external providers and dependencies.

• Infrastructure Security: Assess vendor resilience testing frequency, availability, scalability, and performance SLAs.

• Incident Response Drills: Include LLM-specific incidents in tabletop exercises and update playbooks accordingly.

LLM-Specific Vulnerabilities (OWASP-inspired)

Prompt Injection Attacks

• Direct Prompt Injection: Test for direct prompt injection by attempting to bypass system prompts.

• Indirect Prompt Injection: Attempt indirect prompt injection by manipulating data sources that feed into the LLM.

• System Instruction Manipulation: Try to override or modify system instructions within user inputs.

• Jailbreaking Attempts: Test for attempts to bypass ethical guidelines or content restrictions.

Authorization Bypass

• Access Control Testing: Attempt to access data or perform actions beyond the user's authorized scope.

• Unauthorized API Calls: Test if the LLM can be tricked into making unauthorized API calls.

• Sensitive Data Extraction: Check if sensitive information can be extracted through carefully crafted prompts.

• Role-Based Access: Verify if the LLM respects user roles and permissions in its responses.

Data Leakage

• Training Data Exposure: Probe for potential exposure of training data through specific queries.

• Sensitive Information Disclosure: Test if personal or sensitive information can be extracted from the model.

• System Architecture Disclosure: Check for unintended disclosure of system architecture or backend details.

• Cache Security: Attempt to retrieve information from LLM caches that should be access-controlled.

Input Validation and Sanitization

• SQL Injection Testing: Test for SQL injection in LLM-generated database queries.

• XSS Vulnerability Testing: Attempt XSS attacks through LLM-generated outputs.

• Command Injection Testing: Check for command injection possibilities in LLM-processed inputs.

• Special Character Handling: Verify proper handling and escaping of special characters.

Vector Database Security

• Access Control Verification: Test access controls on vector database queries.

• Document-Level Security: Attempt to bypass document-level security in vector stores.

• Data Leakage in Similarity Searches: Check for potential data leakage through similarity searches.

• ACL Synchronization: Verify proper synchronization of ACLs between source systems and vector databases.

API and External Service Interactions

• Unauthorized API Requests: Test for unauthorized API calls through LLM-generated requests.

• API Parameter Manipulation: Attempt to manipulate API parameters to gain elevated privileges.

• Confused Deputy Attacks: Check for potential confused deputy attacks in multi-system interactions.

• Identity Propagation: Verify proper identity propagation in API calls made by the orchestrator.

LLM-Generated Code Execution

• Sandbox Escape Testing: Test sandbox escape attempts in environments running LLM-generated code.

• Malicious Code Injection: Attempt to inject malicious code through crafted prompts.

• Unauthorized Imports: Check for unauthorized library imports or function calls in generated code.

• Resource Limits: Verify resource usage limits and execution timeouts.

Memory and Context Manipulation

• Memory Poisoning: Attempt to poison the LLM's short-term or long-term memory.

• Context Leakage: Test for context leakage between different user sessions.

• Context Window Manipulation: Try to manipulate the context window to gain unauthorized information.

• Sensitive Data Clearing: Check for proper clearing of sensitive data from the LLM's working memory.

Autonomous Agent Vulnerabilities

• Unauthorized Actions: Test for unauthorized actions in multi-agent systems.

• Decision-Making Manipulation: Attempt to manipulate agent decision-making processes.

• Inter-Agent Data Leakage: Check for potential data leakage between collaborating agents.

• Agent Communication Controls: Verify proper access controls in agent-to-agent communications.

MLOps Pipeline Security

• Training Data Poisoning: Attempt to poison training data used for model fine-tuning.

• Model Versioning Security: Test for unauthorized access to model versioning and deployment systems.

• Supply Chain Vulnerabilities: Check for potential supply chain vulnerabilities in the ML pipeline.

• Training Artifacts Access Control: Verify proper access controls on training logs and model artifacts.

Orchestrator Security

• Authorization Bypass: Test for potential bypass of orchestrator-level authorization checks.

• Identity Manipulation: Attempt to manipulate identity information passed by the orchestrator.

• Error Handling: Check for proper handling of errors and edge cases in the orchestration layer.

• Cache Security: Verify secure implementation of any caching mechanisms in the orchestrator.

Output Validation and Filtering

• Harmful Content Filtering: Test if malicious or sensitive content can bypass output filters.

• Manipulation of Outputs: Attempt to trick the system into generating harmful or inappropriate responses.

• Sensitive Data Filtering: Check for potential data leakage through carefully crafted output requests.

• Handling PII: Verify proper handling of PII and other sensitive information in LLM outputs.