Adversarial machine learning attacks, taxonomies, and mitigations across the ML lifecycle.
Adversarial ML gives useful language for thinking about evasion, poisoning, privacy attacks, and model misuse. It becomes especially valuable when teams need to distinguish classic model risk from the broader application-layer risks found in modern LLM systems.
These items are included because they add useful evidence, framing, implementation detail, or upcoming context for teams working in this area.
NIST finalizes AI 100-2e2025, providing a terminology and taxonomy for adversarial machine learning across predictive and generative AI systems.
PMLR Volume 299 collects fourteen peer-reviewed CAMLIS papers spanning typographic prompt injection, system-level AI red teaming, white-box LLM backdoors, scam agents, LLM attack defenses, poisoned-model restoration, security knowledge graphs, cloud identity analysis, and production cyber-defense agents. Individual entries provide stable abstracts, citations, and open PDFs, with code or supplemental material where available.
Play video
Edward Raff and collaborators introduce Maximum Violated Multi-Objective attacks for manipulating financial statements while simultaneously reducing model-generated fraud scores. Their evaluation finds roughly 20 times more successful dual-objective attacks than standard methods; in about half of tested cases, earnings could be inflated 100–200% while fraud scores fell 15%.
Play video
The AgentFuzz researchers present directed greybox fuzzing for finding paths from attacker-controlled natural-language input to security-sensitive agent operations. Their evaluation across 20 open-source agents combines generated seed prompts, semantic and distance feedback, and argument-aware mutation, reporting 34 high-risk zero-days and 23 assigned CVEs.
Play video
Sanket Badhe presents ScamAgent, an autonomous multi-turn framework that combines planning, conversational memory, deceptive framing, and text-to-speech to produce realistic scam calls. Evaluation against current model safeguards shows that distributing malicious intent across apparently benign turns can bypass prompt-level refusal and content filtering.
NVIDIA argues that most proposed model CVEs actually describe vulnerable serving applications, unsafe serialization and supply-chain formats, access-control failures, or statistical behaviors shared by a model class. A narrow exception may exist for deliberately poisoned training that creates a reproducible backdoor in a specific weight artifact.
garak v0.16.0 begins context-aware scanning with technique and intent annotations and IntentProbe, adds native Anthropic and adaptive-attack plugins, and introduces a unified selection grammar plus revised report output.
Play video
Ashley Song and collaborators evaluate test-time compute strategies on two operational cybersecurity agents: a container vulnerability analysis workflow and a server-alert triage system. The study examines whether allocating more inference-time reasoning can improve both answer accuracy and consistency across repeated runs.
Play video
BlackIce packages fourteen open-source responsible-AI, LLM-security, and adversarial-ML tools into a reproducible, version-pinned container with a unified command-line interface. The CAMLIS presentation explains tool selection, coverage, dependency isolation, image architecture, and a working assessment demonstration rather than presenting the bundle as a substitute for test design.
garak 0.14.0 adds JSON configuration, redesigned HTML reports, detector calibration and benchmarking guidance, new generator integrations, and broader report records. It also changes the eval and digest JSONL formats and aggregate scoring, making the release operationally important for teams that parse reports or compare historical baselines.
TrustFall shows how project-defined MCP configuration can turn a generic “trust this folder” decision into unsandboxed command execution in several coding agents, with zero-click variants in unattended CI. The vendor-authored research traces the issue to conflating permission to read or edit a workspace with permission to start repository-supplied executables.
Trail of Bits bypassed multiple agent-skill scanners with compiled Python hidden beside benign source and with prompt-like prose that persuaded an LLM classifier to accept a malicious configuration. The experiments show recurring blind spots around unreferenced files, binaries, assets, and ambiguous installer behavior, and also explain why legitimate skills can contain patterns that look malicious.
Play video
Text2VLM is a reproducible pipeline that extracts harmful concepts from text-only safety datasets and renders them as typographic images for multimodal evaluation. Human validation supports the transformation pipeline, and tests of open-source visual language models find greater prompt-injection susceptibility when the same concepts arrive through images instead of plain text.
GuardFall tests 11 open-source coding and computer-use agents against shell-command transformations that evade string and regex deny lists, including quote removal, $IFS expansion, command substitution, and encoded payloads. The study finds configuration- and model-dependent failures and shows that local or auto-approve modes can turn untrusted repository content into host command execution; it is vendor-authored research, not an independent benchmark.
Google introduced Gemini Omni Flash, a multimodal model that combines text, image, audio, and video references to generate and iteratively edit video through natural-language conversation. Generated videos include a SynthID watermark.
Play video
Fuzzinglabs researchers explain how threat modeling, file-format fuzzing, and plugin analysis exposed an authentication bypass and memory-corruption issues in Ollama plus command injection in NVIDIA Triton Inference Server's model-configuration pipeline. The Pwn2Own case study also examines RedisAI, ChromaDB, and container-runtime attack surfaces.
Adversa's IICL study evaluates a few-shot jailbreak that interleaves benign and harmful demonstrations and uses short output-field labels to shift model behavior. Across more than 3,500 probes, ten models, and seven ablations, results vary materially with example order and field names; the work is vendor-authored and its model-specific attack rates should be independently reproduced.
Trail of Bits describes supervising GPT-5.5-Cyber as it built ASan and UBSan variants, derived seed corpora, and wrote fuzz harnesses for roughly a dozen zlib entry points in one day. The useful result is the workflow and its emphasis on reachability and reportability; vulnerability details remain under coordinated disclosure and the speed comparison is the authors' estimate.
SymJack demonstrates that a user-approved, apparently harmless copy command can write through a repository-controlled symlink into executable agent configuration, producing code execution when the tool restarts. The vendor-authored study reports variants across six coding agents and highlights a gap between approval text, shell semantics, and the resolved filesystem target.
Release notes for garak, an LLM vulnerability scanning and evaluation toolkit. Relevant to tracking new probes, detectors, and repeatable red-team workflows.
garak release adding probes and detector improvements for LLM security testing. Relevant to maintaining practical red-team coverage across evolving attack techniques.
OWASP’s GenAI security project remains a practical baseline for teams building or assessing LLM applications and agentic systems.
Google Cloud outlines a defense-in-depth view of AI security spanning application controls, data protections, and infrastructure isolation.
Play video
This AI Explained video reviews a major AI development through the lens of agentic workflows and tool-use risk. It is useful context for AI engineering, evaluation, governance, and operational risk.