LLM Security: Prompt Injection, Data Leakage and Safe Agents

LLM security is the practice of stopping a language model application from being manipulated, leaking data or taking harmful actions. The main risks are prompt injection, sensitive data exposure, over-permissioned agents and weak retrieval controls. You cannot make the model itself trustworthy, so you secure everything around it: inputs, permissions, tools, outputs and monitoring.

That framing matters. Most teams start by trying to write a better system prompt that tells the model to behave. A prompt is a request, not a control. The rest of this guide covers the controls that actually hold up when a real user, or a real attacker, starts poking at your application.

Why LLM security is different from application security

Traditional applications separate code from data. A SQL query has a structure, and user input fills in parameters. Language models have no such boundary. Instructions and data arrive as the same stream of text, and the model decides what to treat as an instruction.

This means classic defences such as input validation only go so far. You are dealing with a component that is probabilistic, can be persuaded and will sometimes do something you did not expect, even with no attacker present. The security model has to assume the model will occasionally be wrong or compromised, and limit what damage that can cause.

Prompt injection: direct and indirect

Prompt injection is the defining risk for LLM applications. It comes in two forms.

Direct prompt injection

A user types instructions designed to override your system prompt: "ignore previous instructions", role-play tricks, encoded text or long, confusing conversations that wear down the model's adherence. The goal is usually to extract the system prompt, bypass content rules or get the model to do something outside its intended scope.

Direct injection is easier to reason about, because the attacker is the user. If the model has no special privileges, the damage is limited to what that user could do anyway.

Indirect prompt injection

Indirect injection is more dangerous. The malicious instructions sit inside content the model reads on someone else's behalf: a web page, an email, a PDF, a support ticket or a document in your knowledge base. When an employee asks an assistant to summarise their inbox, an email containing hidden instructions can tell the model to forward data elsewhere or misrepresent what it found.

The user did nothing wrong, and the attacker never touched your application directly. Any system that reads untrusted content and also has access to private data or tools is exposed. That combination is where most serious incidents come from.

Data leakage

LLM applications can leak data in several ways:

• Through the prompt: system prompts often contain internal rules, product details or even credentials. Assume the system prompt will eventually be extracted, and keep secrets out of it.
• Through retrieval: a RAG system that indexes everything and filters nothing will happily surface a salary sheet to anyone who asks the right question.
• Through outputs: models can render links or images that send data to an external server when displayed, a known exfiltration route.
• Through logs and vendors: prompts and responses stored in logs, analytics tools or third-party model providers become another copy of sensitive data to govern.

If your application handles personal data, the usual data protection obligations still apply, such as India's DPDP Act or GDPR. This is general information, not legal advice; check specifics with your counsel.

Insecure tool use by agents

Agents raise the stakes because they act, not just answer. An agent that can send emails, update records, run code or call internal APIs turns a successful prompt injection from an embarrassing reply into a real-world action.

The core controls are familiar from access management:

1. Least privilege: give each agent only the tools and scopes its task needs. A support agent that reads orders does not need a refund API.
2. User-scoped credentials: the agent should act with the permissions of the person it serves, not a powerful service account.
3. Human approval for consequential actions: payments, deletions, external messages and permission changes should need a person to confirm.
4. Validated tool inputs: treat model-generated arguments like untrusted user input. Validate types, ranges and allowed values before execution.
5. Rate and spend limits: cap how many actions an agent can take per session, so a loop or attack cannot run unchecked.

We cover agent architecture in more depth in our guide to enterprise AI agents in production.

RAG access control

Retrieval augmented generation is often where sensitive data leaks first, because teams index a shared drive and forget that the drive had permissions. The fix is to carry access control from the source into the retrieval layer.

Store document permissions as metadata alongside each chunk, and filter retrieval by the requesting user's identity before anything reaches the model. Never rely on the model to decide what a user may see. Keep the index in sync when permissions change at the source, and treat a stale permission as a security bug. If you are still deciding on an architecture, our comparison of RAG vs fine-tuning explains why fine-tuning on sensitive data creates a different, harder problem: you cannot easily remove what a model has learned.

Model and data supply chain

Your LLM application depends on components you did not build: hosted model APIs, open-weight models, embedding models, vector databases, orchestration libraries, plugins and datasets. Each one is a supply chain risk.

Practical steps include pinning model versions so behaviour does not change underneath you, downloading open models only from trusted sources and verifying checksums, scanning libraries like any other dependency, and reviewing vendor terms on data retention and training. Poisoned training or fine-tuning data is a real concern for teams that train on scraped or user-contributed content.

OWASP Top 10 for LLM applications

The OWASP Top 10 for LLM applications is the most widely used community list of risks, and it is a good shared vocabulary for engineering and security teams. At a high level it covers prompt injection, sensitive information disclosure, supply chain weaknesses, data and model poisoning, improper output handling, excessive agency, system prompt leakage, weaknesses in vectors and embeddings, misinformation and unbounded consumption of resources. The list is revised periodically, so check the current edition rather than relying on a copy someone saved last year.

For governance, the NIST AI Risk Management Framework gives a structure for identifying and managing AI risk across an organisation, and ISO/IEC 42001 sets out requirements for an AI management system that can be certified. Neither tells you how to stop a specific injection attack, but both help you show that AI risk is managed deliberately.

Red teaming LLM applications

Red teaming means attacking your own application before someone else does. For LLM apps, that includes:

• Injection attempts: direct jailbreaks and indirect payloads planted in documents, emails or web pages the system reads.
• Data extraction: trying to retrieve system prompts, other users' data or documents outside the tester's permissions.
• Tool abuse: persuading agents to call tools with harmful arguments or chain actions in unintended ways.
• Resource abuse: long or recursive inputs that drive up cost or latency.

Combine manual testing by people who understand both security and how models behave with automated suites that replay known attacks on every release. Add each new finding to the regression suite, the same way you would add a test for a fixed bug. Red teaming is not a one-off exercise; as models, prompts and tools change, so does your exposure.

Guardrails and monitoring

Guardrails are checks that run before and after the model. Input guardrails screen for known injection patterns, personal data and off-topic requests. Output guardrails check for leaked secrets, unsafe content, unexpected links and responses that do not match the expected format.

Guardrails reduce risk but do not remove it, because classifiers miss novel attacks. Treat them as one layer alongside least privilege and approval steps.

Monitoring closes the loop. Log prompts, retrieved documents, tool calls and outputs, with sensible retention and redaction. Alert on unusual patterns: spikes in refusals, repeated attempts to extract instructions, tool calls outside normal ranges or sudden cost increases. Feed incidents back into your evaluation and red team suites.

An LLM security checklist

Use this checklist before any LLM application reaches production, and review it at each major change.

1. Map every data source the model can read and mark which ones contain untrusted content.
2. Remove secrets and sensitive internal rules from system prompts.
3. Enforce document level permissions in retrieval, based on the user's identity.
4. Give agents least privilege tools and user-scoped credentials.
5. Require human approval for payments, deletions, external messages and permission changes.
6. Validate every tool argument the model generates before execution.
7. Block or sanitise external links and images in model output.
8. Pin model versions and review vendor data retention terms.
9. Run input and output guardrails, and test them against known attacks.
10. Red team before launch and keep an automated attack suite in CI.
11. Log prompts, retrievals and tool calls with redaction and agreed retention.
12. Set alerts and spend limits, and include the application in your incident response plan.

Frequently asked questions

What is prompt injection in LLM security?

Prompt injection is an attack where text fed to a language model contains instructions that override the developer's intended behaviour. Direct injection comes from the user; indirect injection hides inside content the model reads, such as web pages or documents. It is the most important risk to design for in LLM applications.

Can prompt injection be fully prevented?

Not with current models. Because instructions and data share the same channel, no filter or prompt reliably stops every attack. The practical approach is to limit impact: least privilege, permission-aware retrieval, human approval for risky actions and close monitoring.

Is it safe to use a hosted LLM API with company data?

It can be, with the right configuration. Review the provider's data retention and training terms, use enterprise agreements where available, minimise what you send and keep logs governed. For highly sensitive data, consider private deployment options or open models hosted in your own environment.

How often should we red team an LLM application?

Before launch, after any significant change to models, prompts, tools or data sources, and on a regular schedule in between. Keep an automated suite of known attacks running in CI so regressions are caught on every release, and reserve deeper manual exercises for major changes.

Does the OWASP Top 10 for LLM applications replace normal security testing?

No. It adds LLM-specific risks on top of the usual concerns such as authentication, authorisation, injection into downstream systems and infrastructure security. An LLM feature is still part of a web application, and it needs the full set of standard controls as well.

How Sunday Labs can help

Sunday Labs builds and secures LLM applications and agents for enterprises, from threat modelling and red teaming to guardrails, permission-aware retrieval and monitoring. Senior engineers do the work, and our founders review it every week before anything goes to production. See our AI security services for how we approach it, or start a conversation and we will look at your application's risks with you.