2026-08-13

Third-Party AI Risk Assessment: A Checklist a Mid-Market Team Can Run

A repeatable third-party AI risk assessment for security teams without a red team: what to ask a vendor, which answers are red flags, and how to weigh claims

A vendor walks into your procurement process with an AI feature bolted onto a product you already wanted to buy. The demo is clean. The sales engineer answers everything. Someone on the business side has already written the PO in their head. And you, the person who signs off on security, have about two weeks to decide whether this thing is safe to put next to your data.

That is the spot most mid-market security teams are in right now. You do not have a dedicated AI red team. You probably do not have a data scientist who can pull apart a model. You have a small crew, a stack of other work, and a vendor who is very motivated to tell you everything is fine. I want to give you a way to run a real third-party AI risk assessment inside those constraints, so you can sign with your eyes open and say no when saying no is the right call.

I have sat on both sides of this table. I helped write the OWASP Top 10 for LLMs and contributed to the Agentic Security Initiative, and I advise boards and security teams on exactly this decision. The pattern I see over and over is a team that treats an AI vendor like any other SaaS vendor, runs the usual SOC 2 checklist, and misses the parts that make AI different. So let me start there, because if you get the framing wrong the rest of the questionnaire is theater.

Why an AI vendor is a different risk than the last SaaS vendor you bought

NIST says this plainly in the AI Risk Management Framework. The risks AI systems pose are unique in ways that traditional software risk practices do not cover. A model can be trained on data that shifts over time, sometimes a lot and without warning, and that drift changes how the system behaves in ways that are hard to see from the outside. The system is socio-technical, so its behavior depends on who operates it, what context it runs in, and how it interacts with other systems. You can read the framing yourself in the AI RMF 1.0.

Here is the part that bites you specifically as a buyer. NIST calls out that risk from third-party data, software, and hardware is harder to measure, because the metrics the developer used may not line up with the metrics you care about as the deployer, and the developer may not tell you which metrics they used at all. The organization that built the model and the organization that runs it in production see different risks and often do not talk to each other about them. You are the deployer. The risk of how this model behaves in your context lands on you, and the developer may have no visibility into your context whatsoever.

BSA, the software industry group, made this same point to NIST during the drafting of the framework. They pushed NIST to add guidance for the case where a company deploys an AI system acquired from an external vendor, and for the case where the customer customizes or retrains that system. Their comment is worth reading because it is the vendor side asking the standards body to spell out that risk management is a shared responsibility split across the developer and the deployer. When even the vendors are saying the responsibility is shared, you should take the hint. Signing the contract does not move the risk to them. It splits it, and your half is the half that shows up in your incident report.

So the goal of a third-party AI risk assessment is narrow and it is worth stating out loud. You are trying to figure out what this vendor's AI does to your data, your users, and your obligations, how much you can verify versus how much you are being asked to trust, and what you will do when it fails. That last one matters because it will fail in some way, and the difference between a bad day and a bad quarter is whether you planned for it.

Who eats this risk

Before the checklist, get clear on who owns the outcome, because a questionnaire with no owner is a document that dies in a shared drive.

If the model leaks customer data through a prompt injection, your customers eat the first harm and your company eats the breach cost, the notification duty, and the trust damage. If the model produces a biased or wrong output that drives a decision about a person, that person eats it and your legal team eats the claim. If the vendor goes down or changes their model under you with no notice, your operations team eats the outage. The CISO answers for all of it to the board, and the board answers for it to regulators and shareholders. None of that lands on the vendor's account manager.

Name these owners inside your own shop before you send a single question to the vendor. It changes how seriously people take the answers.

The framework: a repeatable third-party AI risk assessment in six steps

Here is the operator sequence I hand to teams. Run it in order. Each step has a named owner and a way to tell you are done, so it does not turn into an open-ended research project that never closes. Adapt the titles to your org, but keep the owner and the done-check on every step.

  1. Scope the use and set a risk tier. Owner: the security lead who will sign off, working with the business sponsor. Write one paragraph that says what the AI does, what data it touches, whether a human reviews its output before anything happens, and whether it makes or influences a decision about a person. Use that to drop the vendor into a tier: low (internal drafting help, no sensitive data, human always in the loop), medium (customer data in, human reviews high-impact output), or high (regulated data, automated decisions about people, or an agent that can take actions on its own). Done when the tier is written down and the business sponsor has signed the same paragraph. This step maps directly to the NIST idea that you assign oversight and resources based on risk tolerance, so higher-risk systems get more scrutiny. Do not run the full battery on a low-tier tool. You will burn goodwill and miss the one that matters.
  2. Send the questionnaire and demand documents, not adjectives. Owner: the analyst running the assessment. Send the tiered questionnaire below and ask for evidence attached to each answer. A claim with no artifact is an unanswered question. Done when every question in the tier has either a document behind it or an explicit written "we do not do this," and you have logged which is which.
  3. Weigh each claim against independent evidence. Owner: the analyst, escalating to the security lead on anything that does not reconcile. For every material claim, ask what would prove it and whether that proof exists outside the vendor's marketing. A third-party audit report, a published model card, a penetration test from a named firm, a contract clause. Done when each high-tier claim carries a confidence label (verified, partly verified, or vendor assertion only) and you can defend the label.
  4. Map the shared responsibility. Owner: security lead with legal. Write down, in plain language, which risks the vendor controls, which you control as the deployer, and which fall in the gap between you. The gap is where breaches live. Done when you have a one-page split that both your team and the vendor have seen and neither disputes.
  5. Decide, with conditions. Owner: the CISO or the person who carries the sign-off. Choose one of four: approve, approve with contract conditions, approve for a limited pilot with a kill switch, or decline. Tie any condition to a clause in the contract, so the promise survives the sales cycle. Done when the decision and its conditions are recorded and the conditions are in the redline, not in an email.
  6. Set the recheck. Owner: whoever owns the vendor inventory. AI vendors change their models, and a model change can undo everything you assessed. Put the vendor on a review cadence tied to its tier (high tier every quarter, medium twice a year, low once a year) and require the vendor to notify you of material model changes. Done when the vendor is in your inventory with a next-review date and a named owner, matching the NIST Govern practice of inventorying AI systems and reviewing them on a set frequency.

That is the whole loop. It is boring on purpose. A third-party AI risk assessment that runs the same way every time is one you can teach to a new analyst, defend to an auditor, and finish before the PO expires.

The questionnaire: what to ask, sorted by what you are worried about

A questionnaire is only as good as the answers you refuse to accept. Group your questions by the thing you are trying to protect, and ask for the artifact alongside the answer. I will give you the questions and, right after, tell you what a good answer looks like versus a red flag.

Data: where does my data go and what does it train

Ask these of any vendor above the low tier.

What data do you send to the model, and does our data ever leave our tenant or region. Do you use our inputs or outputs to train or fine-tune your models, and can we turn that off in writing. Who are your model subprocessors, meaning which foundation model providers or hosting services sit behind your product. How long do you retain prompts and outputs, and can we set that retention. If we delete our account, what happens to any data already used in training.

A good answer names the subprocessors (for example, which foundation model API they call), says our data stays in our region, confirms in the contract that our data is not used for training by default, and gives a concrete retention number we can change. A red flag is any of these: they cannot name their model subprocessor, training on customer data is on by default and buried in the terms, retention is "as long as necessary," or the answer to the deletion question is a shrug. If they train on your inputs and will not carve you out, treat that as a hard stop for anything touching sensitive data. Once your data is in a training set you cannot pull it back out.

Security of the model itself

This is the part the standard SaaS questionnaire skips, and it is where AI-specific harm hides.

How do you defend against prompt injection and jailbreaks, and have you tested it. Do you have input and output filtering, and what does it catch. If the product is agentic, meaning it can call tools or take actions, what actions can it take without a human approving them, and how are those actions scoped and logged. Have you run adversarial testing or red teaming against this system, who ran it, and can we see a summary. How do you detect and handle model output that is wrong in a way that causes harm.

A good answer describes specific controls, names who tested the system (an internal team with a real method, or an outside firm), and shares at least a summary of findings and fixes. For an agentic product, a good answer shows tight scoping, a human approval gate on anything with real-world effect, and full logging of tool calls. The red flags: "our model does not hallucinate," "prompt injection is not a concern for our architecture," or an agent that can send email, move money, or change records with no human in the loop and no audit trail. NIST's Generative AI profile exists precisely because these risks are novel to or made worse by generative systems, and a vendor who waves them off has told you they are not tracking the same risks you are. That mismatch is the exact measurement gap NIST warned about.

Governance and accountability

Ask who is responsible when the model is wrong.

Do you have a named owner for AI risk, and does your leadership take responsibility for model decisions. Do you maintain model cards or documentation of intended use and known limits. What is your process when a customer reports a harmful output. Do you test for bias where the model influences decisions about people, and how. What legal and regulatory requirements have you mapped this product against.

A good answer points to real documentation, a named accountable person, an incident path with timelines, and evidence they have thought about the laws that apply to your use. This lines up with the NIST Govern function, which asks whether legal and regulatory requirements are understood and documented and whether executive leadership takes responsibility for AI risk decisions. A red flag is a vendor who cannot name anyone accountable, has no documentation of intended use, and treats "is this legal in our industry" as your problem alone. It is a shared problem, and a mature vendor knows that.

Reliability and change

Ask what happens when the model moves under you.

How often do you change or update the underlying model, and will you tell us before you do. What is your uptime commitment and your plan when the model provider behind you has an outage. How do you monitor for drift, meaning the model getting worse over time as the world changes. Can we test the product against our own data before we commit.

A good answer commits to change notification in the contract, has a fallback when their upstream provider fails, monitors production behavior rather than only lab results, and lets you run a pilot on your data. NIST makes the point that a model measured in a lab can behave differently in real operational settings, so a vendor who only cites benchmark numbers and offers no way to test in your context is asking you to trust the least reliable kind of evidence. The red flag is a vendor who changes the model silently, has no fallback, and will not let you pilot.

How to weigh a vendor claim against independent evidence

Here is the skill that separates a real third-party AI risk assessment from a filled-out spreadsheet. Every answer a vendor gives is a claim, and claims come with wildly different amounts of proof behind them. Your job is to sort them and to size your trust to the evidence, the same way you would with any risk call.

Start by labeling each material claim with one of three tags. Verified means you have proof from outside the vendor's own mouth: an audit report from a named firm, a model card the vendor published and will stand behind in the contract, a penetration test you can read, a clause you can enforce. Partly verified means there is some independent signal but gaps remain, for example a SOC 2 that covers their infrastructure but says nothing about the model, or a red team summary with no methodology. Vendor assertion only means the sole evidence is the vendor telling you it is true.

Then apply a simple rule. The higher the risk tier, the more of your material claims need to be verified or partly verified, and the fewer you accept on assertion alone. For a high-tier system touching regulated data, I want the data-handling and model-security claims verified or in the contract, full stop. For a low-tier internal tool, assertion plus a decent contract is often fine. Write the confidence label next to each claim in your assessment, because six months later when something breaks, the label tells you and the board whether you took a known bet or got blindsided.

Watch for the specific ways vendors inflate a claim. A benchmark score is evidence about a lab, not about your data, so treat it as partly verified at best until you pilot. A compliance certification is often scoped to the company's general operations and does not cover the AI feature at all, so read the scope statement and do not let a SOC 2 logo stand in for a model security review. "We use enterprise-grade security" is an adjective, not evidence. Arvind Narayanan's talk on how to recognize AI snake oil, which NIST itself cites in the Manage playbook, is a good gut check here: the more a claim leans on the mystique of AI and the less it offers you a way to test it, the more skeptical you should be.

One more move that costs you nothing. When a vendor makes a claim you cannot verify, ask them what evidence would prove it and why they cannot share that evidence. A confident, mature vendor will either produce the artifact or give you a straight reason it is confidential and offer a workaround like an NDA review. A vendor who gets defensive or vague about why they cannot show you proof has answered the question in the way that matters.

A worked example so this is not abstract

Say a mid-market fintech is buying a customer-support AI that drafts replies to customer emails and can pull account details to personalize them. Walk it through the loop.

Step one, scope and tier. It touches customer financial data and personalizes using account records, so even though a human agent reviews each draft before sending, it lands in the high tier because of the data and the regulated context. The security lead writes the paragraph, the support VP signs it.

Step two, questionnaire. The answers come back. Data stays in region, good. Training on customer inputs is on by default, and the toggle to turn it off is only on the enterprise plan the fintech is buying, so that is workable if they get it in writing. The vendor names their foundation model subprocessor, good. On model security, they say they have output filtering and ran an internal red team, and they attach a two-page summary. On agentic actions, the tool only drafts and never sends, so the human gate is real. On change notification, they offer thirty days notice on model changes in the enterprise contract.

Step three, weigh the claims. Region and no-training get tagged verified because they go into the contract. The red team summary is partly verified, because it exists but was internal and thin on method, so the analyst asks for the methodology and gets a reasonable answer, keeping it partly verified. The output filtering claim stays vendor assertion only until the pilot, because there is no test the fintech can read. Everything material is now labeled.

Step four, map responsibility. The vendor controls model hosting, filtering, and the base model. The fintech controls who at the support desk can see which account fields, and the human review gate. The gap: what happens if a drafted reply contains another customer's data because of a prompt injection in an inbound email. The fintech owns catching that at the review gate, so they add a rule that agents cannot approve a draft that references an account number the customer did not provide. That gap discovery is the whole point of the exercise.

Step five, decide. Approve for a sixty-day pilot with a kill switch, conditioned on the no-training clause and the thirty-day change-notice clause being in the signed contract. During the pilot the security team seeds a few test emails with injection attempts and watches whether the filter and the human gate catch them. That turns the assertion-only filtering claim into real evidence.

Step six, recheck. High tier, so quarterly review, and the vendor is on the hook to notify on model changes. Into the inventory it goes with an owner and a date.

Notice that no step required a data scientist or an in-house AI red team. It required clear scoping, a demand for artifacts, honest confidence labels, and a pilot designed to test the one claim that mattered most. That is a third-party AI risk assessment a small team can run.

The mistakes I see teams make

The most common one is treating the AI questionnaire as a longer version of the SaaS questionnaire. You end up with forty questions about SOC 2 and zero about whether the model trains on your data. Cut the redundant infrastructure questions for AI vendors and spend the saved effort on data, model security, and change.

The second is accepting a benchmark or a certification as the end of the conversation. Both are useful inputs and neither tells you how the model behaves on your data in your context. Pilot the high-tier tools.

The third is skipping the shared-responsibility map. Teams assume the vendor's controls cover the whole risk, sign, and then discover during an incident that the gap between vendor and deployer was theirs to close all along. The one-page split is cheap insurance.

The fourth is running the assessment once and never again. A model you cleared in March can be a different model by September with no new contract and no announcement unless you required one. The recheck cadence and the change-notice clause are what keep the assessment true over time.

And the fifth, the quiet one, is running a beautiful assessment that no owner acts on. If the CISO does not sign the decision and the conditions do not make it into the redline, you produced a document, not a control.

Where this fits in your broader AI governance

A third-party AI risk assessment is one function of a governance program, and it works best when it plugs into the rest. NIST's Govern function asks you to keep an inventory of AI systems, to have leadership take responsibility for AI risk decisions, and to make sure your policies cover third-party AI systems specifically. The vendor loop above feeds that inventory and gives leadership something concrete to sign. If you are building this program from scratch and want a hand, that is the kind of work I do with security teams through advisory engagements and as a fractional Chief AI Officer, and for teams that need ongoing security leadership there is the virtual CISO route. If you just want to talk through where your program stands, reach out.

The honest bottom line is this. You will not eliminate AI vendor risk, and no questionnaire will make a vendor perfectly safe. What you can do is know what you bought, size your trust to your evidence, close the gaps you can, and plan for the failure you cannot prevent. Do that consistently and the next time a slick demo shows up two weeks before the PO expires, you will have a real answer instead of a nervous yes.

FAQ

We do not have a data scientist or a red team. Can we really assess an AI vendor?

Yes, and the framework above is built for exactly that. The parts that need deep model expertise (breaking the model yourself, auditing training data) are the parts you push onto evidence: a red team summary from a named firm, a model card, a pilot on your own data. Your job is to scope the use, demand artifacts, label your confidence honestly, and design a pilot that tests the one or two claims that matter most for your tier. That is a security and governance skill, and it is one a small team has. For the rare high-stakes case where you need hands-on model testing, bring in an outside specialist for that single question rather than trying to build a red team.

What is the single biggest red flag in a vendor's answers?

A vendor who cannot or will not tell you what evidence would prove their claims, and who gets vague about why they cannot share proof. Close behind it: training on your data by default with no way to turn it off, and an agentic product that can take real-world actions with no human gate and no logs. Any one of those on a system touching sensitive data is a reason to decline or to make approval conditional on a contract fix. The pattern underneath all three is a vendor tracking different risks than you are, which is the measurement gap NIST names in the AI RMF, and that gap is where your incidents come from.

How is this different from our existing vendor security review?

Your existing review covers infrastructure, access, and general security posture, and you should keep it. The AI-specific layer adds four things a standard review misses: where your data goes and whether it trains the model, security of the model itself against prompt injection and jailbreaks and unsafe agent actions, governance and accountability for model decisions, and what happens when the model changes under you. NIST is explicit that AI risks differ from traditional software risks, so a review built for traditional software will pass a vendor whose model quietly trains on your customer data. Add the AI layer, do not replace the base.

How often should we reassess a vendor we already approved?

Tie it to the risk tier. High tier every quarter, medium twice a year, low once a year, and always require the vendor to notify you of material model changes in the contract. The reason is specific to AI: a model can change behavior through an update or through drift with no new contract and no announcement, so a point-in-time assessment goes stale faster than it does for ordinary software. Put every approved vendor in your inventory with a named owner and a next-review date, which is the NIST Govern practice of inventorying AI systems and reviewing them on a set frequency.

The business wants to sign now and the assessment is not done. What do I do?

Offer the limited pilot with a kill switch instead of a full production yes. It lets the business start, buys you time to turn assertion-only claims into tested evidence, and keeps you able to walk away cleanly if the pilot surfaces something bad. Make approval conditional on the two or three contract clauses that carry the most risk (no training on your data, change notification, data residency) so the promises survive the sales cycle. A conditional, time-boxed yes is almost always available, and it beats both a blind yes and a flat no that gets you overruled.

Sources