An AI chatbot may answer a normal question well. But a change in wording may make it expose private data. An image model may classify one picture correctly but fail when small changes are made. An AI agent may follow a harmful instruction hidden in a document it was meant to read.

These are the kinds of problems AI security testing aims to find. The goal is to see how an AI system acts under attack. Testers look at the possible harm. They also check whether its security controls work. This means testing more than the model. You also need to test its data, APIs, tools, storage, and the app around it.

This guide covers AI security risks, red teaming, and prompt injection. It also covers adversarial machine learning and retrieval-augmented generation (RAG). You will learn how to test agents, choose tools, and fix flaws.

What Is AI Security Testing?

AI security testing is the process of finding weak points in AI models and the systems that use them. Testers try inputs and actions that could make a system leak data or break its rules. They also check for unsafe results and tasks done without permission. Then they review the harm, fix weak points, and test again.

AI security testing covers both machine learning (ML) and generative AI. It can test predictive models that classify images or detect fraud. It also covers large language models (LLMs) that create text. AI agents that call tools or take actions need testing too.

Four parts of an AI system need close attention:

Testing areaWhat it checksExamples of risks
AI application testingHow the full app reacts to real and hostile inputsPrompt injection, unsafe answers, leaked context, tool misuse
AI model testingHow the model acts when inputs or its behavior are changedEvasion, model extraction, poisoning, membership inference
AI infrastructure testingThe systems that host, connect, and run the modelWeak APIs, supply-chain flaws, excessive access, resource abuse
AI data testingThe data used for training, retrieval, and inferencePoisoned data, poor data quality, harmful content, private data exposure

AI security testing is not the same as using AI tools to scan ordinary software. Here, the AI system itself is the thing being tested.

Identify Risks. Fix Vulnerabilities. Strengthen Security

Why Traditional Security Testing Is Not Enough for AI

An AI application still has normal software risks. Attackers may target its database, network, cloud setup, API, or build pipeline. Standard penetration testing is needed for those areas.

But an AI model brings new ways to attack a system. A person can place harmful instructions in a chat message, web page, document, or tool result. A model may act on that text even when the text came from an untrusted source. Small changes to an image or a line of text may also change a model’s decision.

AI outputs are often probabilistic. The same prompt can give different results on different runs. A failed attack on the first try may work later. That is why a single test is not enough.

AI systems also change over time. A new model version or system prompt can change the answers. So can new training data or a new tool. Security tests must be run again after such changes.

There is one more difference. AI red teams test deliberate attacks and harmful results during normal use. A user may mean no harm. They may still receive a biased, unsafe, or false answer. That can create safety and trust problems even when there is no attacker.

The Three Main Strategies Behind AI Security Testing

A sound testing program uses three related methods.

1. Conventional security testing

This checks normal attack surfaces such as servers, networks, databases, APIs, source code, and development tools. In AI systems, these store valuable assets. They include model weights, prompts, training data, and agent memory.

Testers look for weak access controls, exposed keys, and flaws in data handling. They also check insecure code, bad settings, and other attack paths. Static application security testing (SAST) and software composition analysis (SCA) can help. So can dynamic testing, API checks, and penetration testing.

2. Model performance validation

This checks whether a model still works as intended. The team uses test data with expected outputs and clear acceptance rules. The tests can show changes in accuracy, model drift, or other performance issues.

For security, this is also a way to spot changes caused by data poisoning or model poisoning. If a model starts making new mistakes, the team should check what changed. Its data or training process may be the cause.

3. AI-specific adversarial testing

This tests attacks that use AI inputs and behavior. Examples include prompt injection, evasion, model extraction, data leakage, and misuse of AI tools. The attack may come from a user prompt. It may also come from content that the model reads later.

Some attacks cross all three areas. An attacker might tamper with a training database. A later trigger could then change how the model acts. Another may use a normal software flaw to place hostile text in an agent’s memory. This is why traditional security teams and AI red teams need to work together.

AI Security Testing vs. AI Red Teaming

AI red teaming is a part of AI security testing. It uses planned, authorized attacks to find risks that routine checks may miss. The team thinks like an attacker, tries different paths, and records which controls fail.

Red teaming goes beyond one test script. It can include prompt attacks, model-level attacks, and API abuse. It also checks harm caused by normal users. It is often done before release and then repeated during the life of the system.

Three types of AI red teaming

TypeMain focusSkills commonly needed
Full-stack red teamingApp, infrastructure, APIs, data pipelines, CI/CD, hosting, and supply chainPenetration testing and software security
Adversarial machine learning (AML)Attacks on the model, such as evasion, poisoning, and extractionML, data science, and security
Prompt injection testingHarmful instructions sent directly or through external contentLLM behavior, app logic, and security

A full assessment can include all three. The right balance depends on the model, the app, and the harm that could follow a failure.

Full-stack work covers the code and deployment path. This includes CI/CD pipelines, model hosts, network links, APIs, vector databases, and access to stored data. It also checks how third-party models and tools enter the supply chain.

Adversarial ML work can use black-box tests, where only inputs and outputs are visible. It can also use white-box tests, where the tester can inspect model parts such as parameters or gradients. Prompt-focused work covers direct injections, indirect injections, jailbreaks, and attacks that build up over many turns.

These are authorized tests with a defined scope. They are not the same as a real attacker trying to gain access without permission.

Test the base model and the complete app

A base LLM can be tested through an API. This helps reveal its own limits and weaknesses. The full application should also be tested through the interface people use. That second test checks more than the model. It includes the app’s prompts, filters, tools, and search controls.

The two results can differ. A base model may give an unsafe answer that the application blocks. An application may also create a new flaw by giving the model access to private content or powerful tools.

My tip: If you test an AI chatbot, list what it can know and do. Then test those limits through both the model API and the live app. A safe model response does not prove the whole app is safe.

AI Security Threats You Need to Test

Different AI systems face different attacks. Threat modeling helps a team set priorities. It shows which assets, inputs, and actions matter most. An AI attack technique catalog can help organize known attack paths.

Prompt injection

A prompt injection attack places instructions in content the model receives. The goal is to make it ignore its intended task or follow the attacker’s directions. A direct injection comes from a user prompt. An indirect injection comes from an outside source. That may be a page, file, search result, or tool response.

The outcome may be a policy bypass, exposed system prompt, private data leak, or unwanted tool action. Indirect attacks matter for RAG apps and AI agents. These systems often read content they do not control.

Jailbreaking and unsafe outputs

A jailbreak tries to bypass safety rules. Testers may change roles, wording, format, or language, or build up a request over many turns. Tests check whether the model produces harmful, abusive, biased, or otherwise blocked content.

Not every unsafe result requires a clever attack. Normal users can also get harmful or misleading answers. Both kinds of outcomes belong in the test plan.

Adversarial examples and evasion

In an evasion attack, an attacker changes an input so the model gives the wrong result. The change may be small or hard for a person to notice. Evasion tests are common for image, audio, text, and classification systems.

Ask whether the model can be fooled. Then ask if the changed input could appear in real use. A lab result has less value if the attack cannot happen in real use.

Data poisoning and model poisoning

Poisoning changes data or the training process. It may also alter fine-tuning data or the model itself. The goal is unwanted behavior. A poisoned dataset may contain false labels, hidden triggers, or harmful samples. A backdoor may remain hidden until a later input activates it.

Tests should check data sources, labels, lineage, access rights, and signs of unusual model behavior after changes.

Model extraction and theft

Attackers may query an AI model many times to copy its behavior or learn about its design. They may also try to steal model files or parameters from development or runtime systems. A copied model can expose intellectual property and help attackers build more targeted attacks.

Testing may include API probing, access checks, rate limits, and reviews of model storage and deployment controls.

Membership inference and model inversion

Membership inference tries to determine whether a particular record was part of a training dataset. Model inversion tries to reconstruct information about the data used to train a model. Both can create privacy risks.

These tests check what a model’s outputs can reveal. Training records may leak clues even when the user cannot access those records.

Sensitive data leakage

An AI system may reveal private user or company data. It may also expose keys or content from training and retrieval. The data might appear directly in an answer or pass through a tool call.

Tests should cover prompts, context, memory, and retrieved content. Check logs, API responses, and paths that send data outside the system too.

Insecure output handling

Generated text is not always safe to pass to another system. If an app treats model output as trusted code, it can create a software flaw. The same applies to HTML, SQL, or commands. Check the system that reads or shows the output. Do not inspect only the model’s words.

API, infrastructure, and supply-chain attacks

AI apps use normal software and cloud services. Attack surfaces include model endpoints, libraries, notebooks, and vector stores. Plugins, containers, build systems, and deployment settings add more paths. Tests should cover authentication, authorization, misconfiguration, tampered dependencies, exposed interfaces, and resource exhaustion.

Agent goal hijacking and tool misuse

AI agents may plan steps, call tools, keep memory, and work with other agents. An attacker may try to change the agent’s goal. They may trigger a tool without permission or poison its memory. They may also pass harmful instructions between agents.

These failures can have a larger impact than a bad chat answer because an agent may take an action. Testers need to check what happened in the tool, database, or external service.

Bias, hallucinations, and misuse

Security and safety tests also look for unfair results and false claims. Other risks include harmful content, poor explanations, and over-reliance on AI. Weak design or oversight can also lead to misuse. An AI system can fail even when no one breaks into it.

Judge these issues against the system’s purpose and users. Ask what harm an answer or action could cause.

Poor design can also make harmful use too easy. An AI system might support unfair automated choices, harmful content, or surveillance that was not meant to be allowed. Test the intended use, likely misuse, and the controls around both. Overfitting, weak handling of edge cases, or undue weight given to a few model features may also make attacks easier. Testing should look for these paths as well as deliberate break-ins.

Four Practical Approaches to AI Security Testing

AI penetration testing

AI penetration testing adapts traditional attack methods to AI models and apps. Testers probe the whole system. They check the interface, model endpoint, data flow, APIs, tools, and hosting. They may look for stolen data, bypassed rules, weak permissions, or flaws in connected services.

LLM and agent red teaming

This uses planned attacks, repeated prompts, and creative testing to find flaws in LLMs and agents. Both manual testing and automated scans are useful. Multi-turn attacks and tool-based scenarios need extra attention because they may not fail on the first turn.

Adversarial input testing

This uses crafted changes to images, text, audio, or structured data to test how a model responds. It checks model robustness and the limits of its decision process.

AI API fuzzing

API fuzzing sends malformed, unusual, or semi-random requests to a service. The aim is to find crashes, errors, weak input checks, bad authorization, parsing flaws, or information leaks. Both public and internal AI endpoints can be tested this way.

AI Security Testing Guide: What to Test at Each Layer

The AI Testing Guide organizes checks across application, model, infrastructure, and data layers. The following test groups give teams a clear starting point. Not every check fits every system; choose tests based on scope and risk.

AI application testing checklist

Test IDTest focusWhat to check
AITG-APP-01Direct prompt injectionCan user text change or override the model’s instructions?
AITG-APP-02Indirect prompt injectionCan an external page, file, or result carry a harmful instruction?
AITG-APP-03Sensitive data leakCan the model reveal private or confidential data?
AITG-APP-04Input leakageDoes the app repeat or retain data it should not expose?
AITG-APP-05Unsafe outputsCan the app generate content that breaks its safety rules?
AITG-APP-06Agentic behavior limitsCan an agent take harmful or unauthorized actions?
AITG-APP-07Prompt disclosureCan a user reveal hidden system instructions?
AITG-APP-08Embedding manipulationCan crafted input distort semantic search or embedding behavior?
AITG-APP-09Model extractionCan repeated queries help copy the model’s behavior?
AITG-APP-10Content biasDo results change unfairly across sensitive groups?
AITG-APP-11HallucinationsDoes the model invent facts instead of using reliable information?
AITG-APP-12Toxic outputCan the model produce abusive or hateful content?
AITG-APP-13Over-reliance on AIDoes the app handle risky or uncertain requests responsibly?
AITG-APP-14ExplainabilityCan the app give useful and accurate reasons for results?

AI model testing checklist

Test IDTest focusWhat to check
AITG-MOD-01Evasion attacksCan small changes to input cause harmful wrong results?
AITG-MOD-02Runtime model poisoningCan new data at runtime alter behavior over time?
AITG-MOD-03Poisoned training setsAre there hidden triggers, bad labels, or malicious samples?
AITG-MOD-04Membership inferenceCan outputs show whether a record was used in training?
AITG-MOD-05Model inversionCan outputs be used to recover training information?
AITG-MOD-06Robustness to new dataDoes the model cope with noise, edge cases, and unfamiliar inputs?
AITG-MOD-07Goal alignmentDoes the model follow its intended goals when instructions conflict?

AI infrastructure testing checklist

Test IDTest focusWhat to check
AITG-INF-01Supply-chain tamperingAre models, packages, and build inputs authentic and intact?
AITG-INF-02Resource exhaustionCan high load or malformed input disrupt the service?
AITG-INF-03Plugin boundary violationsCan a tool cross its security or sandbox limits?
AITG-INF-04Capability misuseCan file access, code execution, or other powers be abused?
AITG-INF-05Fine-tuning poisoningDoes fine-tuning introduce harmful behavior or backdoors?
AITG-INF-06Development-time model theftAre model files or keys exposed during development?

AI data testing checklist

Test IDTest focusWhat to check
AITG-DAT-01Training data exposureCan a model reveal sensitive training examples?
AITG-DAT-02Runtime exfiltrationCan an attacker make the system send out hidden data?
AITG-DAT-03Dataset diversity and coverageDoes the dataset cover users, conditions, and edge cases?
AITG-DAT-04Harmful content in dataDoes the dataset contain toxic or harmful material?
AITG-DAT-05Data minimization and consentIs collected data needed, allowed, and handled under privacy rules?

These categories also help teams plan reports. A test should state its input, expected result, actual result, and the risk if the control fails.

How to Conduct AI Security Testing Step by Step

Step 1: Set the goals and scope

Start with the AI system’s purpose and the harm you want to prevent. Work with security, engineering, legal, policy, and risk teams to set clear limits.

List the models, apps, APIs, data stores, and tools in scope. Identify the users and likely attackers. Decide which risks must be tested and what level of harm is unacceptable.

Build a ranked list of harms with concrete examples. It should cover data loss, unsafe actions, harmful content, and risks to users. Ask legal and policy staff which issues have a hard limit. Leave room for new harms that testers find during open-ended work. Add those findings to the list for the next test round.

Step 2: Map the AI system and its trust boundaries

Record how input reaches the model and where the output goes. Include training data, retrieval sources, caches, model hosting, agent memory, and outside services. Mark where untrusted content can enter or where one user or service could gain access to another’s data.

Threat modeling helps show which attacks are possible. It also helps stop teams from spending time on threats that do not apply to their system.

Step 3: Build an attack and test plan

Choose test cases based on the system and its risks. Include both known attack patterns and open-ended tests that let red teamers find new problems.

Test the base model and the full app. Test direct inputs and indirect sources. Set the model version, system prompt, tools, permissions, and settings you will use. When possible, use a test environment that closely matches production.

For each test, set an entry point, attacker access level, safe result, and clear sign of failure. Test the base model with its own safeguards through an API. Then test the app through its user interface with app-level controls. Repeat both tests before and after fixes. Use live interfaces only when the exercise is approved and can be run safely.

Step 4: Build the right red team

Use people with different skills. Security testers may find flaws in APIs and code. ML experts may find adversarial examples. People who know the service’s field may spot harmful results that a security engineer would miss.

Give each tester a clear goal, feature area, time window, and way to record results. A healthcare chatbot may need people who know patient care. It also needs testers who understand its technical setup.

Include people who approach the app as normal users, not just attackers. They may find harms that scripted attacks miss. Assign each person a class of harms or a product feature. For work done in several rounds, rotate roles to bring a fresh view. Agree on a test ID, date, input record, and saved output or screenshot before work starts.

Step 5: Run manual and automated tests

Manual red teaming finds unusual paths and helps explain why a failure happens. Automated tools can try many prompts, input changes, and model settings. Use both.

Run the same test more than once when the system can give different results. Include normal inputs too, so you can spot controls that block valid users or damage the service.

Start with the base model to learn its weak points. Then test the full workflow, including any tools and extra controls. Compare runs with and without a proposed fix where safe. For automated tests, use a nonproduction setup with settings and resources close to production. This keeps harmful agent actions and test data under control.

Step 6: Measure harm and set priorities

Do not judge a test by one unsafe sentence alone. Ask what data was exposed and what action took place. Who could be harmed? How easy would the attack be in real use?

Record repeatable steps and the attack success rate (ASR) when tests can be measured that way. A report should distinguish a technical success from a harmful outcome. A stopped attack may cause little harm. An attack that silently sends data out is a different risk.

Step 7: Fix, retest, and monitor

Fix the weak control, then run the failed tests again. Check that the fix has not harmed normal use. Add important cases to your regression suite.

Repeat testing before release and after changes to models, prompts, tools, data, or infrastructure. Keep a record of risks you accepted and areas you did not test.

Set a new test round after each major fix. Review the full known harm list, not only the last failed prompt. Some fixes reduce one risk but leave another in place. Record residual risk: the risk that remains after controls are added.

My tip: I would not mark a test as passed just because the attack failed once. Record how often it worked across repeated runs. For a high-impact leak or action, even one success can deserve urgent review.

How to Test AI Systems Against Prompt Injection

Prompt injection testing needs more than a list of hostile phrases. The test must match the real app, including its safety filters, tools, and data sources.

1. Gather relevant attack cases

Start with known patterns: instruction override, role confusion, and jailbreaks. Include encoded text, hidden instructions, and malicious links. Also test attempts to reveal private data. Keep only the cases that match the service’s risk model. Save other cases for later review if the system changes.

The case set can come from known attack patterns or a testing tool. Check that it covers the main ways the attacker can reach the model. Do not erase lower-priority cases. Archive them so the team can use them if the app’s purpose or risk limits change.

2. Adapt attacks to the app

Make the tests fit the system’s work. A support bot may have ticket data and a search tool. An agent may have email or file access. The key question is what a harmful answer or action would mean for that system.

If the app can act on a response, check the action as well. Looking at the final text alone may miss a tool call that already made a change.

For a service bot, examples may include exposing a test phone number or private token, or giving a price quote the business does not allow it to give. For a tool-using agent, an attack may aim to send test data to an outside address. Use fake secrets and approved test endpoints. The goal is to prove the control failed without risking real data.

3. Pair attacks with ways to detect success

For each case, define a clear sign of failure. It may be a private value in an answer or a blocked API call that still runs. It may also be a changed record or data sent through a tool. Some results need human review. A text pattern cannot judge every harmful response.

Connect each attack input to its own check. A test for a leaked marker might scan output for that marker. A test for data theft should inspect the outbound tool log. One for privilege gain should check the actual account rights. A human reviewer or an AI-based judge may help with unclear text, but the final risk decision should use the app’s rules and the evidence.

4. Test the production path

Send attacks through the app’s normal entry points where possible. The test should include the filters and checks real users face. Testing only the bare model can miss the controls or the new flaws inside the full app.

5. Test indirect injection

If the model reads external content, place approved test data through that same route. Examples include a document upload, search result, ticket, or retrieved page. Confirm the app actually reads the content, then check whether it follows instructions from that lower-trust source.

For example, a support ticket may contain words hidden by white-on-white formatting. The visible ticket can look harmless while the hidden text tells the agent to search for a secret. The test succeeds only if the intended boundary is crossed. Check whether the agent reads the content, calls a protected tool, or sends the test secret out. Do not count the mere presence of hostile words as a successful attack.

6. Add input variations

Change wording, format, encoding, or other parts of the test. This checks whether simple changes bypass filters. Multi-turn tests can also show whether a system becomes less strict after a long exchange.

7. Run more than once

Use repeated runs with the same model version and settings. Keep the prompt, response, actions, and test context so results can be compared.

8. Judge the risk, not just the output

Review the data exposed, the actions triggered, and the likely harm. A minor wrong answer may have little impact. Revealing a secret or changing a record without permission can cause far more harm.

Risk also depends on how the attacker could reach the input. A direct user prompt may be easy to submit. An indirect attack may require the attacker to edit a source document first. A rare but severe result still needs attention.

If an alert detects and stops the action, record that too. Detection and response time can limit harm. A filter that blocks a simple wording change is useful. Yet it does not prove safety against a person willing to try many variants. Judge both the effort needed and the impact of a successful attempt.

9. Retest and include normal use

Recheck fixes and update the attack set as new methods appear. Include benign prompts that the app should answer. This helps find false positives and stops security filters from making the product unusable.

How to Test Against Evasion Attacks

Evasion tests try to find inputs that make a model give a harmful wrong result. The work has to reflect what an attacker can actually do.

Define feasible input changes

Decide how much an input may change before the attack becomes unrealistic. For images, this may involve pixel changes, visual similarity, size, or lighting. For audio, it may involve noise or sound limits. For text, it can involve synonyms, spelling changes, or encoding.

Researchers may use measures such as L0, L2, L-infinity, SSIM, and PSNR to describe changes. The chosen limit should match real inputs. It must also fit the controls that check them.

The tests should also account for how data is captured. A low-resolution camera may not record tiny pixel changes. A real room may have glare, motion, and noise. An audio test may need to meet sound limits. A change that looks valid in a file may not work through the real device.

Keep two checks separate: can the attacker make the input, and would the system accept it? Review human approval, input filters, and anomaly alerts that could stop a crafted sample before the model sees it.

Search for effective examples

Depending on what the attacker can access, testers may use:

  • White-box testing: work with the model’s parameters and gradients when full access is realistic or available to the internal team.
  • Black-box testing: try inputs based on model responses without seeing its internal code or weights.
  • Transfer attacks: build adversarial inputs on a similar model, then test whether they work on the target.
  • Heuristic testing: use domain knowledge to try changes likely to confuse the model.

Some white-box tests use Projected Gradient Descent. Some black-box tests use Square Attack. These are examples, not proof that one method fits every model.

Tests may use both familiar inputs and inputs outside the model’s normal data range. If an attack works in the lab, test it against real input checks. Can it reach the system in practice?

Use this order when it fits the attacker model:

  1. Perfect-knowledge search: if model access is realistic, use its parameters or gradients. Begin with inputs it classifies well. Try small changes across the whole input and changes in one local area. Keep them inside the allowed limits.
  2. Zero-knowledge search: if the attacker sees only the output, change the input based on each response. Count how many tries the search takes. Test whether rate limits or monitoring stop it.
  3. Surrogate-model transfer: if direct searches fail, try a similar model or a model built from available task data. Generate attacks there and check which ones transfer to the target.
  4. Domain-based changes: use likely real-world edits, such as image rotation, scaling, text typos, or a change in wording.

Do not assume every stage must run. If an earlier test already shows a feasible attack with serious harm, the result may be enough to fail that check. If no attack is found, record which methods and effort limits were used. Stop searching when the time and resources go beyond what the relevant attacker could use.

For example, a team may test a model that estimates an animal’s age from a photo. The target model may be private, making direct gradient attacks unrealistic. The tester can train a similar model on available labeled photos, craft altered images, and check if the target also fails. The test must still show that the altered image is realistic and the wrong result would cause unacceptable harm.

Check impact and repeat

A test fails when a feasible evasion input leads to harm above the accepted risk level. In some systems, one wrong result may be enough. In others, the risk may depend on repeated errors. Apply a fix and run the tests again, including valid inputs.

When feasibility is unclear, run a separate check. Can the real input device capture the change? Do detection systems flag it? Could a person make the change at all? Testing tools use known attack methods, so a clean result does not rule out new methods. Test normal inputs after each fix to make sure valid use still works.

Agentic AI Security Testing

AI agents can use tools, keep state across sessions, and pass messages to other agents. That makes their test scope wider than a single chatbot prompt.

Map the four agent security layers

LayerWhat to test
LLM reasoningPrompt injection, goal changes, and misleading instructions
Tool executionUnauthorized calls, altered parameters, weak validation, and excess permissions
InfrastructureAPI controls, keys, credentials, logs, and access to services
Inter-agent communicationMessage tampering, spoofed identities, and trust-boundary failures

The agent’s tools need security checks outside the prompt. A rule in a system prompt is not enough. The tool itself must check access before it runs.

Key agent tests

  • Goal hijacking: can an external instruction shift the agent away from its assigned task?
  • Unauthorized tool use: can it call a tool or change a setting without the right permission?
  • Parameter tampering: can attacker-controlled text affect a tool’s arguments?
  • Persistent memory poisoning: can hostile content affect later sessions?
  • Multi-agent propagation: can one compromised agent influence another or the orchestrator?
  • Delegation-chain abuse: can an agent gain more power through a chain of handoffs?
  • Human approval bypass: can urgent wording or repeated requests lead a reviewer to approve a harmful action?
  • Protocol flaws: do Model Context Protocol (MCP), agent-to-agent (A2A), or similar interfaces have weak implementation controls?
  • Supply-chain changes: could a changed tool or model version bypass earlier checks?
  • Failure modes: what happens when the context window fills, a tool fails, a task stops halfway, or the orchestrator chooses the wrong route?
  • Log integrity: can the agent hide or change the record of what it did?

Use single-turn and multi-turn tests. A model that resists an attack on the first message may fail after a long sequence. Record which layers and tools were tested and which were not.

When ranking an agent flaw, check what it can do on its own. Ask whether the effect lasts or can spread to other agents. Check if the action can be reversed.

A hands-on agent testing sequence

A hands-on agent testing sequence

First, map the system. List each agent, the orchestrator, its tools, memory, data sources, and shared message channels. Mark which content comes from a user, document, tool reply, or another agent. Note where that content changes from untrusted input into a proposed action.

Next, prove the normal controls work. Run safe tasks with the expected permissions. Confirm which tool calls are allowed, which need approval, and which should be denied. Without a baseline, it is hard to tell an attack failure from a broken feature.

Then, attack each entry point. Try one-message injections and long conversations separately. An attacker may slowly shift an agent’s goal over many turns. Repeat the same pattern in a retrieved page, a tool response, and an agent message. Check whether the bad instruction spreads to peer agents or changes what the orchestrator does.

Test the tool gate without the LLM. Send approved test calls straight to the API gateway or permission layer. Include wrong roles, changed tool parameters, and attempts to use another user’s resources. The gate must reject a forbidden action even if an LLM asks for it.

Test goal-level attacks. Give the tester a clear objective, such as making an agent expose a fake secret, use a higher role, or change a task. See whether the goal can be reached through a series of steps. Test copied or changed tool versions as well. A new tool may not follow the same security rules.

Check the handoffs and human reviews. Attempt message spoofing between agents and unsafe use of delegated rights. Test urgent or confusing approval requests. Too many prompts for approval can make a reviewer less careful. If the workflow uses MCP or A2A, check those protocol interfaces like any other API. Include conventional risks such as SSRF, SQL injection, and cross-site scripting (XSS) in connected servers.

Finally, check failures and evidence. Fill the context window, return tool errors, stop a task midway, and force an unexpected route. Review data changes, tool logs, and alerts. Record each test’s steps and how often it worked. State how many agents, tools, turns, and attack cases were covered. Treat untested paths as gaps, not safe paths.

My tip: When an agent fails, check the tool logs before you read its final reply. The reply may sound harmless. But a tool may already have opened a file or changed a record. It may even have sent data elsewhere.

RAG Security Testing: Check Retrieval Before the Answer

Retrieval-augmented generation (RAG) gives a model outside content to help it answer questions. This creates another input path. The model might read a company file, search index, ticket, wiki page, or uploaded document.

RAG tests check what the system retrieves. They also check what the model does with it.

Five layers of RAG security testing

RAG layerMain tests
IngestionConnector access, file handling, parser flaws, malformed uploads, and source integrity
IndexingChunking, embeddings, index write access, metadata, and stale caches
RetrievalUser and tenant permissions, ranking changes, and access to protected chunks
Prompt assemblySeparation of trusted instructions from untrusted retrieved content
Generation and outputIndirect prompt injection, data disclosure, and unsafe output handling

Test retrieval access on its own

The retriever must not return data outside the user’s access rights. Test with different users, tenants, and roles. Check access at document and chunk level, not just by document name.

Do this test before the model generates an answer. If a retriever gives a private chunk to the wrong user, access control has failed. The model’s later refusal does not fix that leak.

Permission changes need tests too. Reduce a user’s role, remove a permission, or delete a source item in the test system. Then check whether the index and caches respect that change.

Automate RAG permission checks without an LLM

Retrieval access can be tested with a clear pass or fail. A query by one identity should return only chunks that identity may read. You do not need a model-generated answer to judge that rule.

Set up the check this way:

  1. Describe each test user by subject ID, tenant, and role. Mark which documents and chunks that user can read. Access to one chunk must not grant access to the rest of the file.
  2. Use a small test adapter to send each query to the retriever. Collect the returned chunk IDs and access labels. Check these items before they enter the model prompt.
  3. Test tenant isolation apart from document IDs. The same file name or document ID in two tenants must not let one tenant read another’s data.
  4. Test permission changes over time. Downgrade a role, change an access-control list (ACL), or revoke access. Query again and confirm the result changes at retrieval time.
  5. Prove the test can catch failure. Run it against a purposely weak test retriever as well as the real index. If it misses the known weak case, the test needs work.

This approach also works as a repeatable regression test when a search engine, vector store, or ranking setup changes.

Test untrusted and poisoned content

Place approved test content into a real input path. This could be a document upload or source update. Check whether it reaches the index and is later retrieved. Then test whether harmful instructions inside it change the answer or trigger a tool.

Test forged source details such as the author, date, or confidence label. These fields can make untrusted content seem more reliable than it is. Test both likely search results and content that may surface only after ranking changes.

Use realistic entry paths: a wiki edit, a shared-drive file, a new support ticket, or a crawled page. Follow the content through parsing, chunking, embedding, and retrieval. Test whether a low-ranked or weakly related item can be pushed into the model’s context by changes in search ranking or reranking. Repeat tests across keyword, vector, and hybrid search if the system uses them.

Try several fake source labels in a batch of test files. Change the claimed author, date, or confidence level. Then measure whether those labels affect where a result appears or how much the model trusts it. A forged label must not turn outside text into a higher-priority instruction.

Test the vector store and embedding data

Check who can read or write embeddings. Check whether stored vectors can reveal source text. Consider what happens if a database or backup is exposed. This is separate from prompt testing because the risk exists in data storage and retrieval.

In an approved lab, give the tester read-only access that represents a stolen database credential or exposed backup. Check how much of the original content can be inferred from the vectors. Do not assume embeddings are safe to share just because they are numbers.

Test parsers, rendering, and downstream actions

File parsers, connectors, vector APIs, and search endpoints still face normal software attacks. Test them for issues such as path traversal, injection, server-side request forgery (SSRF), and unsafe file handling.

Include PDF and Office files, OCR input, and HTML. Use malformed, oversized, and oddly structured test files. Check for parser crashes, unsafe deserialization, and service slowdowns. Test each path: direct uploads, connector sync, and web crawling.

Also check what happens when generated content is shown in a browser or passed to another service. Links, code blocks, and HTML may carry risk. The next system must check how it handles that output.

If RAG feeds an agent, test the full path from a planted document to a tool call. This combines three risky parts: hostile content, private data that the agent can reach, and a tool that can send or change data. A harmless chat answer does not rule out a risky tool action.

Measure RAG test coverage

Record which sources, indexes, retrieval settings, and access boundaries were tested. The team should also list any source it could not test. If the RAG system can call tools, check the impact. What data could leak? What action could follow?

Measure coverage by source type, number of test documents, user roles, tenants, and ranking modes. Rank findings by the access boundary crossed, the number of files or users affected, and any action taken by an agent. Search and retrieval engineers should work with security testers. Many RAG flaws come from a broken pipeline or access rule, not from the LLM itself.

Manual vs. Automated AI Security Testing

Both methods are needed. They solve different parts of the problem.

Manual testingAutomated testing
Finds creative or unusual attack pathsRuns many known cases at scale
Uses expert judgment to assess harmUses repeatable checks and scoring rules
Helps explain why a control failedTracks results across model and app versions
Works well for hard-to-judge outcomesWorks well for clear checks, such as blocked tool calls or leaked markers

Automation can generate adversarial prompts, run scans, compare outputs, and calculate ASR. Human testers still judge complex harm. They also find risks that the test set missed.

Run automated attacks in a safe, production-like test setting. Use the real filters and access rules where possible. Do not use live private data when controlled test data can show the same flaw.

AI Security Testing Tools for Predictive and Generative AI

No single tool covers every threat. Some tools focus on adversarial inputs for predictive models. Others probe LLMs for prompt injection and unsafe behavior. A few are better suited to runtime checks or enforcing guardrails.

Open-source tools for predictive AI testing

ToolMain useFit and limits
Adversarial Robustness Toolbox (ART)Evasion, model robustness, model inference, and related adversarial ML testsBroad framework and data-type support; works with common ML and deep-learning libraries
ArmoryAdversarial ML evaluations, especially repeatable attack testsUsed in security-focused work; has a narrower integration scope than broad toolkits
FoolboxAdversarial examples and evasion testsUseful for research with major deep-learning frameworks; less focused on large enterprise rollouts
DeepSecDeep-learning security and evasion researchMore research-focused, with a smaller community and narrower integration options
TextAttackAdversarial attacks on natural language processing modelsStrong focus on text; connects with common NLP model and dataset libraries

These tools mainly help find model weaknesses. They do not replace infrastructure security checks or guarantee that a model will resist future attacks.

Technical details for predictive AI tools

The following details reflect the 2024 comparison. They show the listed scope at that time, not a promise about the latest release.

ToolLanguage and licenseInput typesFrameworks and main test scope
ARTPython; MITText, image, audio, video, tabularTensorFlow, Keras, PyTorch, MXNet, scikit-learn, XGBoost, LightGBM, CatBoost, and GPy; evasion, development-stage model poisoning, model theft through use, model inversion, and membership inference
ArmoryPython; MITText, image, audio, video, tabularTensorFlow and PyTorch; adversarial evaluations across classification, object detection, and speech tasks; model poisoning and direct injection checks were also listed
FoolboxPython; MITText and imageTensorFlow, PyTorch, and JAX; adversarial examples and evasion, with support for classification and vision tasks
DeepSecPython; Apache 2.0Text and imageTensorFlow, PyTorch, and TorchVision; focused on deep-learning evasion research
TextAttackPython; MITTextPyTorch and natural-language model libraries, including Transformers and datasets; NLP evasion and adversarial data tests

ART covers a broad set of machine learning tasks. Its listed use cases include classification, object detection, and speech recognition. Foolbox and DeepSec are narrower and focus on evasion. TextAttack is designed around text models and changes to language inputs. Armory is aimed at repeatable security evaluations.

All five tools are listed as supporting vulnerability detection or testing, but not direct attack prevention. None replaces a fix in the deployed system.

Open-source tools for generative AI testing

ToolMain useFit and limits
PyRITAutomated LLM red teaming, attack prompts, scoring, and reportsWorks with different model endpoints; useful for repeatable red-team exercises
GarakLLM vulnerability scanning and generated attack probesSupports several LLM interfaces and focuses on finding model failures
Prompt FuzzerDynamic tests against a generative app’s system promptNarrower focus on prompt defenses; check current maintenance before choosing it
Guardrails AIOutput checks and safety rules around LLM behaviorCombines detection with protective controls; not a replacement for red teaming
PromptfooPrompt evaluation and red-team testingUseful for repeatable LLM tests and CI workflows; check feature and license limits for the version used

For example, teams can use PyRIT’s project resources to understand how automated attacks, scores, and reports fit together. Tool support, licenses, integrations, and maintenance can change. Check current details before selecting a product.

Technical details for generative AI tools

ToolModel or service supportMain security role
PyRITLLM APIs and model services, including hosted and local endpointsRuns attack prompts, scores results, measures attack success, and builds reports; focuses on detecting weaknesses, not blocking them
GarakSeveral LLM APIs, Hugging Face models, and local model routesScans for unsafe model behavior, evasion, and direct prompt injection; detects risks rather than enforcing a block
Prompt FuzzerLLM API and prompt-based workflowsTests system prompts with changing inputs; checks whether prompt defenses fail
Guardrails AICommon LLM API services and model frameworksUses rules and validators to detect and prevent unwanted inputs or outputs; can serve as a runtime control
PromptfooLLM APIs, hosted models, and local model routesRuns prompt evaluations and red-team tests; supports repeated test runs in development workflows

The tools do not all serve the same purpose. PyRIT, Garak, and Prompt Fuzzer mainly search for failures. Guardrails AI also enforces checks at runtime. Promptfoo helps teams test prompts and app behavior as part of development. The comparison also lists detection and prevention support for Promptfoo. Check how that support works in the release you plan to use.

License detail: The reviewed Promptfoo codebase used the MIT license for its main code. Its red-team directory had a separate enterprise license. Do not assume that every feature is covered by the same terms. The review also noted that Prompt Fuzzer had seen a period with few updates in 2024; that is not a claim about its maintenance today.

What the historical project figures show

The same assessment recorded GitHub activity in 2024. These are dated snapshots, not live download counts or a verdict on present quality.

ToolStars in the 2024 snapshotForks in the snapshotOpen issues reported
ARTAbout 4,900About 1,200About 131
ArmoryAbout 176About 67About 59
FoolboxAbout 2,800About 428About 21
DeepSec209About 70About 15
TextAttackAbout 3,700About 455About 130
PyRITAbout 2,000About 384About 63
GarakAbout 3,500About 306About 303
Prompt FuzzerAbout 427About 56About 10
Guardrails AIAbout 4,300About 326About 40
PromptfooAbout 4,300About 320About 108

Issue counts can be high because a project is used often, or because work remains open. Stars alone say little about the attacks a tool can find. The review also looked at contributor activity, docs, response times, and ways to get help. Use these old figures only to understand how that comparison was made.

How to compare AI security testing tools

A tool comparison can use four factors:

  1. Popularity: the size and use of the project community.
  2. Community support: contributors, documentation, issue response, and help channels.
  3. Scalability: whether the tool can support larger tests and more than one model framework.
  4. Integration: whether it works with the models, data types, APIs, and pipelines you use.

The 2024 ratings used high, medium, and low bands. High popularity meant more than 3,000 GitHub stars. Medium meant 1,000–3,000 stars. Low meant fewer than 1,000. High community support meant more than 100 contributors and fast replies. Low support meant a smaller community and slower replies. Scalability and integration were judged by deployment fit and framework reach.

The review scored the tools as follows:

ToolPopularityCommunity supportScalabilityEase of integration
ARTHighHighHighHigh
ArmoryLowMediumMediumHigh
FoolboxMediumMediumLowMedium
DeepSecLowLowLowLow
TextAttackHighHighMediumHigh
PyRITMediumHighHighMedium
GarakHighMediumMediumMedium
Prompt FuzzerLowLowMediumMedium
Guardrails AIHighHighMediumHigh
PromptfooHighHighMediumMedium

These ratings came from a 2024 assessment. They are historical, not live product rankings. Check current project health and features before making a choice.

Features to expect from an AI security testing platform

Features to expect from an AI security testing platform

The software market also highlights four core feature groups:

  • Model behavior checks: test for data leaks, biased replies, toxic content, and unsafe actions.
  • Adversarial prompt testing: create and run prompts that try to mislead or bypass the model.
  • AI component scanning: inspect model files, libraries, frameworks, notebooks, and other components.
  • Reporting and remediation guidance: rank findings, explain impact, and suggest fixes.

Other features may cover runtime monitoring and agent actions. Some tools also check supply chains or connect with development systems. A product listing or user rating is not proof that a tool fits your threat model. Test its coverage against your own system.

Best Practices for Effective AI Security Testing

Validate data and its sources

Check where training and inference data came from. Review labels, access rights, data lineage, and signs of poisoning. Check data from outside suppliers and live feeds as well as stored datasets. Poor data can create unfair results, hidden triggers, or leaks.

Use automated checks for unusual records, but also review a sample of labels and source files by hand. Test whether a change can pass through the cleaning pipeline without being caught. Keep a record of the source and changes made to each dataset. Review suppliers and data brokers before trusting their input.

Keep behavior testing active

Run tests as models and apps change. Use normal inputs, edge cases, known attacks, and new patterns. Watch for drift, unusual output, or weaker controls after updates.

Track both input changes and model results. Add stress tests and tests for new conditions that users may face. A new training set, prompt, API setting, or model release can change behavior. Put repeatable tests into the release cycle rather than waiting for a major audit.

Use defense in depth

Do not rely on one prompt, classifier, or filter. Use layers of controls across data ingestion, model use, APIs, infrastructure, and output handling. These may include validation, access controls, encryption, rate limits, isolation, anomaly checks, and runtime guardrails.

A guardrail may block an unsafe answer. It does not replace permission checks at the API or tool layer.

Test the layers on their own. A content classifier may stop a harmful reply. Input checks may stop a malformed file. A container or virtual environment can limit what code can reach. Encryption helps protect stored and sent data. Rate limits may slow brute-force probing. Check that each layer still works if a nearby layer fails.

Enforce least privilege

Give people, apps, agents, and services only the access they need. Check role-based access, strong login for privileged tasks, approvals for sensitive actions, and audit logs. If a key or one tool is misused, limited permissions can help limit the harm.

For high-risk tasks, consider just-in-time access. This grants extra rights only when needed and removes them after the task. Test that rights expire and that a tool cannot keep using them later. Review both user access and service-to-service access.

Monitor drift, misuse, and odd outputs

Track changes in model performance and input patterns. Review alerts, outputs, and logs with both automated checks and human judgment. Connect important findings to incident response so teams know when to act.

Watch for new prompt patterns, sudden changes in result quality, unusual tool calls, or shifts in input data. Monitor both security alerts and model quality. Audit model decisions for fairness and policy issues where relevant. Logs should give responders enough facts to act, not just a large list of warnings.

Include privacy, safety, and compliance checks

Security tests should check whether private information can be exposed or sent to the wrong place. Data tests should also review minimization, consent, and use rules. Some services must meet more than one privacy law. For Swiss and EU processing, a separate FADP vs GDPR review may be useful. It can sit beside AI attack tests. A security test alone is not a full legal audit.

Use governance and risk controls to guide what harms matter. The AI Risk Management Framework is one related resource for that work.

My tip: After fixing a prompt injection, keep the failed case in your test set. Run it after every model, prompt, or tool change. Otherwise, the same weakness can return in a later release.

How to Measure and Report AI Security Testing Results

Good reports help teams decide what to fix. They should also make the test repeatable.

For each finding, record:

  • A test ID, date, and the system or feature tested.
  • The model version, prompt setup, tools, and permissions used.
  • The input and any relevant external content.
  • The actual output and actions taken.
  • The expected safe result.
  • How many times the test was run and how many times it succeeded.
  • The data or system that could be harmed.
  • The impact, likelihood, and risk priority.
  • The fix, owner, and retest result.

Attack success rate (ASR)

An automated red-team exercise may report ASR as:

ASR = successful attack attempts ÷ total attack attempts × 100

This gives a useful measure for a defined test set. It does not describe every possible attack. The score depends on the test prompts and model settings. Detection rules and the meaning of success matter too.

Some outcomes need human review. For agent tests, check tool calls and changes to state. For model tests, check whether a wrong result has real impact. For high-risk failures, do not ignore a rare event only because the overall success rate is low.

Reports should highlight the top issues and link to raw evidence. They should also explain the next tests. Credit the red-team members involved. They should be clear about what was not tested.

Include the testing plan for the next round. Keep screenshots or traces when they help someone repeat a finding. An example of a harmful result proves that a failure is possible; it does not show how common that failure is. That needs a defined test set and measured results. Red teaming supports risk measurement, but it does not replace it.

Common AI Security Testing Mistakes

Testing only the model

A model may be safe on its own. A search index, database, or agent tool may add new risks. Test the full application and its data paths.

Testing only with hostile prompts

Normal users can trigger harmful outputs too. Include benign cases and check whether filters block valid work.

Trusting one successful defense

AI output can vary across runs. Repeat tests, try variations, and measure results instead of using one pass or fail.

Ignoring RAG access checks

If a retriever gives private data to the wrong user, access has failed. That happened before the model answered. Test the index with different roles and tenants.

Treating prompt rules as access controls

Instructions can guide a model, but tool permissions need enforcement outside it. Check gateways, APIs, roles, and logs directly.

Using only automated scans

Scanners cover many known patterns, but human testers can find new attack paths and judge complex harm. Use both methods.

Failing to test after changes

New models, prompts, tools, data, and libraries can create new risks. Retest after changes and keep regression cases current.

Reporting findings without business impact

A technical output alone may not show the real harm. Explain the data, action, or system at risk and how the failure could affect users or operations.

Subscribe to our Newsletter

Stay updated with our latest news and offers.
Thanks for signing up!

Frequently Asked Questions

1. What is AI security testing?

AI security testing checks whether AI models can be fooled or misused. It also tests for data leaks. It includes tests for prompt injection, poisoning, evasion, model theft, weak APIs, and unsafe outputs. It also tests the data and tools connected to the model, not just the model itself.

2. How is AI security testing different from penetration testing?

Traditional penetration testing looks for flaws in software, networks, APIs, and infrastructure. AI security testing adds model behavior and AI input attacks. For example, a harmful instruction in a file might change an LLM’s actions. A small input change might fool a predictive model. Strong assessments use both methods.

3. What are the most common AI security risks?

Important risks include prompt injection, poisoning, evasion, extraction, and data leaks. Also test unsafe output handling and unauthorized agent actions. The main risks vary by system. A predictive image model, a RAG chatbot, and a tool-using agent need different test cases.

4. How do you test an LLM for prompt injection?

Start with clear rules for what the LLM must not reveal or do. Build direct and indirect test cases. Run them through the real app. Check the answers and any tool actions. Change wording and format, run the cases more than once, review the harm, then retest after fixes. Include safe user requests to catch false positives.

5. Which tools are used for AI security testing?

For predictive models, examples include ART, Armory, Foolbox, DeepSec, and TextAttack. For generative AI, examples include PyRIT, Garak, Prompt Fuzzer, Guardrails AI, and Promptfoo. They differ in the models and attacks they support. Choose tools that match your main risks. Check their current support and fit with your system.

Final Thoughts

AI security testing is an ongoing process. It brings together software security and model checks. It also tests hostile inputs, data, and agent actions. Each part matters. A flaw in one layer may lead to a data leak or an unwanted action.

Start with a clear threat model. Test both the model and the full app. Use manual red teaming and automated checks. Track attack results and fix the risks that could cause harm. Then keep testing as the system changes

This page was last edited on 4 November 2026, at 12:17 pm