Secure GenAI Adoption: Understanding LLM Threats
Large Language Models are the engine behind the generative AI revolution. They power chatbots, code assistants, document analysers, and decision support systems across every industry. But with this capability comes a new class of security threats that traditional application security doesn't fully address.
As a member of the OWASP Top 10 for LLM Applications working group, I've seen firsthand how these threats are evolving. Here's what organisations need to understand to adopt GenAI securely.
The OWASP Top 10 for LLM Applications
The OWASP Top 10 for LLM Applications provides the most widely referenced framework for understanding LLM-specific security risks. Key threats include:
Prompt Injection
The most discussed LLM vulnerability. Attackers craft inputs that override the model's system instructions, causing it to perform unintended actions. This can be direct (user input) or indirect (malicious content in documents the model processes).
Defence: Input sanitisation, output filtering, privilege separation between the model and backend systems, and treating all model outputs as untrusted.
Sensitive Information Disclosure
LLMs can inadvertently expose sensitive data — from training data memorisation to leaking information through crafted prompts that extract system instructions or confidential context.
Defence: Data classification before ingestion, access controls on retrieval-augmented generation (RAG) data sources, and output monitoring for sensitive patterns.
Supply Chain Vulnerabilities
The LLM supply chain includes pre-trained models, fine-tuning datasets, embedding models, vector databases, and orchestration frameworks. Each component is a potential attack vector.
Defence: Vendor security assessments, model provenance verification, dependency scanning, and regular updates of all components.
Insecure Output Handling
When LLM outputs are passed to other systems without validation — executing generated code, rendering HTML, or making API calls — the model becomes a vector for injection attacks against downstream systems.
Defence: Treat all model outputs as untrusted input. Apply the same validation and sanitisation you would to any user input before processing or rendering.
Excessive Agency
LLMs connected to tools and APIs can take actions with real-world consequences. If the model can send emails, query databases, or modify files, an attacker who compromises the model's behaviour gains those capabilities.
Defence: Apply least-privilege principles to all tool integrations. Require human approval for high-impact actions. Implement rate limiting and anomaly detection on model-initiated actions.
Building a Secure LLM Architecture
Beyond addressing individual threats, organisations need an architectural approach to LLM security:
Defence in Depth
No single control will prevent all LLM attacks. Layer your defences:
- Input filtering and prompt guards
- Model-level safety training and system prompts
- Output validation and monitoring
- Backend access controls and sandboxing
- Audit logging and anomaly detection
Separation of Concerns
Keep the LLM separate from sensitive systems. Use intermediary services that validate and authorise model actions before they reach production systems.
Monitoring and Observability
Log all interactions with the LLM — inputs, outputs, tool calls, and errors. Monitor for patterns that indicate prompt injection attempts, data exfiltration, or anomalous usage.
Regular Assessment
LLM security is a moving target. Conduct regular assessments that include:
- Automated prompt injection testing
- Manual red team exercises
- Review of RAG data access controls
- Supply chain dependency audits
- Compliance checks against applicable regulations
The Regulatory Dimension
The EU AI Act classifies AI systems by risk level and imposes requirements accordingly. High-risk AI systems — including those used in critical infrastructure, healthcare, and financial services — face the most stringent requirements for transparency, security, and human oversight.
Organisations deploying LLMs in these sectors need to align their security controls with both the OWASP framework and regulatory requirements from the start.
Conclusion
Generative AI offers transformative potential, but secure adoption requires deliberate effort. The threats are real, novel, and evolving. Organisations that invest in understanding LLM-specific threats — and build security into their AI systems from the ground up — will be the ones that realise GenAI's benefits without becoming its casualties.
The OWASP Top 10 for LLM Applications is the best place to start. It's practical, community-driven, and continuously updated to reflect the real-world threat landscape.