AI Vendor Due Diligence Checklist

The first task in any AI vendor evaluation is understanding what you are actually buying. This sounds obvious, but it is routinely skipped. Many commercial AI products are built on top of foundation models provided by a small number of large technology companies. The vendor's contribution may be a user interface, a domain-specific prompt configuration, or a fine-tuned layer trained on specialist data. That is entirely legitimate, but it means you need to ask two separate questions: what is the vendor's product, and what does it sit on top of? This is the question of model provenance: the origin and lineage of the model your product depends on. This matters because if the underlying model changes, your system's behaviour may change without warning. If the foundation model provider updates their terms of service, your vendor's ability to serve you may be affected. And if something goes wrong, attributing responsibility across that chain requires you to understand it first. Worth asking: is this product built on a third-party foundation model, and if so which one and what version? How are updates to the underlying model communicated, and what is the process for regression testing before they reach your environment? What is the vendor's own contribution to the product? The second layer of scrutiny concerns performance claims. Benchmarks in sales materials reflect best-case conditions: curated test sets, favourable configurations, and tasks the model was likely exposed to during training. They are not a substitute for testing the tool against your own data and use cases. Think of it like a credit agreement versus a bond rating: the borrower's contractual promises must be read alongside independent evidence of their creditworthiness. Ask for results on tasks comparable to your use case, run on data similar in type and quality to your own. If the vendor cannot provide this, run the pilot yourself before committing. ISO 27001, an international standard for information security management, and SOC 2, a framework for assessing service organisations' controls, are worth having, but understand what they cover: infrastructure and data-handling controls, not AI accuracy, fairness, or reliability. A vendor can hold an ISO 27001 certificate and a clean SOC 2 report and still produce an AI system that performs poorly on your specific tasks.