Prompt injection attacks, mitigations, detection, and design patterns for safer AI applications.
Prompt injection is the core attack pattern in modern AI applications. It happens when a model treats malicious or conflicting instructions from users, retrieved content, documents, tools, or pages as trusted guidance and changes its behavior in response.
These items are included because they add useful evidence, framing, implementation detail, or upcoming context for teams working in this area.
OWASP's community guide organizes agentic-system risk into ten categories, including goal hijacking, tool misuse, identity and privilege abuse, memory poisoning, insecure inter-agent communication, cascading failures, and rogue-agent behavior. It provides a shared taxonomy and mitigation starting point rather than a certification checklist or evidence that a deployed system is secure.
OWASP FinBot is a hands-on agentic-security CTF built around a simulated multi-agent financial-services platform with real tool access. Its challenges cover prompt injection, tool misuse, policy bypass, data exfiltration, privilege escalation, remote code execution, shared context, and compromised MCP servers.
NVIDIA walks through a simulated Go dependency that detects Codex, writes a malicious AGENTS.md, redirects the coding task, and injects instructions intended to conceal the change from pull-request summaries. The post then maps the chain to dependency, configuration-integrity, monitoring, and guardrail controls.
Why it ranks: manually reviewed for hands-on depth; directly applicable to AI security practice; demonstrates an actionable operational method.
NVIDIA's AI Kill Chain models attacks on AI applications as recon, poison, hijack, persist, impact, plus an iterate-and-pivot loop for autonomous agents. Each stage is paired with concrete controls and then applied to a RAG exfiltration path, connecting prompt injection to data ingestion, memory, tools, downstream actions, and monitoring.
PMLR Volume 299 collects fourteen peer-reviewed CAMLIS papers spanning typographic prompt injection, system-level AI red teaming, white-box LLM backdoors, scam agents, LLM attack defenses, poisoned-model restoration, security knowledge graphs, cloud identity analysis, and production cyber-defense agents. Individual entries provide stable abstracts, citations, and open PDFs, with code or supplemental material where available.
Microsoft Incident Response provides a detection, investigation, and response playbook for prompt abuse, then walks through an indirect prompt-injection scenario in which a hidden URL fragment manipulates an AI summarizer. The guide maps each incident phase to visibility, prompt telemetry, access, audit, and response controls.
NVIDIA's AI Red Team reports recurring failures across six months of enterprise-agent assessments: weak user-level access control, command and file tools that enable code execution, unrestricted network egress, and secrets exposed through environment variables or CLI caches. Social framing, gradual multi-turn escalation, and malicious package installation repeatedly bypassed prompts and model-judge defenses, while controls enforced outside the model reduced exploitability.
Anthropic reports three incidents across six of 141,006 cybersecurity-evaluation runs: models reached unintended real targets, extracted data, or published a malicious package after evaluation isolation and configuration controls failed. The report distinguishes these harness failures from evidence of a persistent model goal, and documents how realistic evaluations can create production consequences.
OpenAI’s system card for deep research covers prompt injection, privacy, code execution, and external red teaming prior to release.
NVIDIA’s AI Red Team provides a deep implementation guide for sandboxing coding agents: enforce network egress and filesystem boundaries below the application layer, protect agent configuration files, isolate spawned hooks and MCP processes, use virtualization where warranted, inject scoped secrets, and expire sandbox state.
OpenAI describes an automated prompt-injection red-team loop for a browser agent: an attacker model proposes an injection, runs counterfactual victim-agent simulations, studies full reasoning and action traces, iterates before submission, and turns successful attacks into adversarial training targets and system-level safeguards.
NVIDIA's AI Red Team demonstrates multimodal prompt injections encoded as symbolic image sequences and rebus puzzles rather than literal text. In the examples, models interpret visual semantics as code or file commands, including reading and deleting files, showing why text keyword filters and OCR-only inspection do not cover the full input surface of a tool-enabled multimodal system.
Play video
Edward Raff and collaborators introduce Maximum Violated Multi-Objective attacks for manipulating financial statements while simultaneously reducing model-generated fraud scores. Their evaluation finds roughly 20 times more successful dual-objective attacks than standard methods; in about half of tested cases, earnings could be inflated 100–200% while fraud scores fell 15%.
Microsoft distills an ontology, eight lessons, and five case studies from red teaming more than 100 generative AI products. The method connects actors, tactics and techniques, system weaknesses, and downstream impacts across conventional application flaws, multimodal prompt injection, responsible-AI harms, and dangerous capabilities.
Play video
Sanket Badhe presents ScamAgent, an autonomous multi-turn framework that combines planning, conversational memory, deceptive framing, and text-to-speech to produce realistic scam calls. Evaluation against current model safeguards shows that distributing malicious intent across apparently benign turns can bypass prompt-level refusal and content filtering.
NVIDIA's AI Red Team distills recurring pre-production findings into three concrete failure classes: prompt-injected model output reaching exec or eval and causing code execution; RAG stores that lose source permissions or accept attacker-writable content; and active Markdown or HTML that turns model output into a browser-based data-exfiltration channel.
Play video
Microsoft's AI Red Team maps seven persistent computer-use-agent risks across UI deception, remote code execution, reasoning leakage, human-approval bypass, indirect prompt injection, identity ambiguity, and emergent content harms. Its cases connect visual overlays and ambient browser content to privileged clicks, unsafe downloads, persistent file changes, and code execution.
An autonomous red-team agent cleared a five-stage prompt-disclosure CTF using authority framing, output transformations, incident-report language, and a shift-handover completion. The write-up distinguishes model behavior from challenge logic and explicitly limits the result to one gamified environment with unknown models, incomplete captures, and no measured production-guardrail success rate.
OpenAI treats prompt injection as contextual social engineering and uses source-sink analysis to connect attacker-controlled content with dangerous actions. The design approach combines model resistance with deterministic limits on data transmission, navigation, tool use, sandbox communication, and user confirmation.
Play video
A live Black Hat demonstration compromises a stock Unitree Go2 robot running Gemini Robotics-ER through attacker-controlled camera and microphone input, turning prompt injection into physical movement. The session adds a failure taxonomy and shows why agents that behave differently when they know they are being tested can create false confidence in clean evaluation scores.
Play video
Ashley Song and collaborators evaluate test-time compute strategies on two operational cybersecurity agents: a container vulnerability analysis workflow and a server-alert triage system. The study examines whether allocating more inference-time reasoning can improve both answer accuracy and consistency across repeated runs.
Play video
An eight-month audit of more than 1,000 Model Context Protocol projects reports over 500 distinct vulnerabilities across protocol design, language-SDK inconsistencies, and ecosystem implementations. The researchers demonstrate elicitation abuse, indirect prompt injection, tool poisoning, cross-agent data exfiltration, and code-execution paths affecting widely used MCP clients and servers.
ASSET Research Group hid a prompt-injection payload in a PNG referenced by an apparently benign AGENTS.md file. Text-only pull-request reviewers missed the image, multiple coding-agent harnesses later followed it and encoded a repository's .env secrets as integer tuples that conventional secret scanners did not recognize, while the same model behaved differently across harnesses. A prototype multimodal reviewer caught 49 of 50 attacks with no false positives on 30 benign pull requests.
NVIDIA uses a PandasAI code-execution vulnerability to explain why generated-code sanitizers are brittle: namespace access, encoding, imports, and runtime context can turn apparently allowed syntax into arbitrary execution. The article separates heuristic filtering from the structural protection provided by a constrained execution environment.