An LLM Security Checklist for Shipping AI Features

Shipping an LLM feature safely isn't about one silver-bullet control — it's about covering each layer where things go wrong. Here's a checklist we use, grouped by concern. Treat it as a starting point and adapt it to your risk.

1. System prompt & instructions

• Keep secrets, keys, and internal URLs out of the system prompt.
• Explicitly separate instructions from user/retrieved data.
• State a refusal policy and least-privilege tool rules.
• Don't rely on the prompt alone for security — assume it can leak.

Grade your prompt against these with the System Prompt Auditor before you ship.

2. Untrusted input

• Screen user input and any retrieved/ingested text for injection primitives.
• Strip zero-width and control characters during ingestion.
• Label retrieved content as untrusted in the prompt.
• Enforce access control before retrieval, not after generation.

The Prompt Injection Scanner and Prompt Firewall cover the screening step; for RAG specifically, see Securing RAG Pipelines.

3. Output handling

• Scan model output for leaked secrets and PII before displaying or logging.
• Sanitize markup; disable auto-loading of remote images in responses.
• Validate links/citations against an allow-list of domains.
• Never eval or directly execute model-generated code or SQL.

Catch accidental leakage with the PII & Secrets Scanner.

4. Tools & actions

• Give the model the minimum set of tools it needs.
• Require confirmation for irreversible or high-impact actions.
• Validate and constrain every tool argument server-side.
• Scope tool credentials narrowly; rotate them.

5. Data & privacy

• Store only what you need; avoid logging raw prompts and PII.
• Encrypt data at rest and in transit.
• Give users a way to delete their data.
• Document what leaves your infrastructure (which provider, which region).

6. Supply chain

• Verify model and dataset checksums against the publisher's hashes.
• Pin dependency versions; prefer versions published a while ago.
• Track provenance for models, adapters, and datasets.

Verify checksums with the Model Integrity Checker.

7. Testing & red-teaming

• Run adversarial prompts against your deployed guardrails.
• Test both jailbreaks and indirect prompt injection — they're different attacks.
• Re-run the suite in CI so regressions are caught before release.

Generate a test battery with the Red-Team Toolkit, or wire the scanner into CI via the @opensecureai/scanner GitHub Action.

8. Monitoring & response

• Log injection attempts, blocks, and anomalies (without raw content).
• Alert on spikes in high-severity events.
• Have a rollback plan for prompt/model changes.
• Review compliance obligations (EU AI Act, NIST AI RMF, ISO 42001).

Ship monitoring with LLM Observability, and check your posture against frameworks with the Compliance Self-Assessment.

Using this checklist

You won't do all of this on day one — and you don't have to. Start with the layers that map to your biggest risk (for most teams that's untrusted input and output handling), get those green, then work outward. Security for LLM features is iterative; the goal is steady coverage, not perfection on the first release.