AI for compliance document review: 8 questions to ask

AI can read a supplier file, a policy set or a contract bundle far faster than a person can. That is not the hard part. The hard part is knowing which of its answers you can rely on, and being able to show an auditor why.

These are the eight questions we would ask of any tool, including our own. Our answers are at the end.

1. Does every finding cite its exact source?

A citation should take you to the passage: the file and the page or section, or for a spreadsheet, the sheet and the rows. A citation to a whole document is barely better than none, because the reviewer still has to find the sentence. In a demo, click through from a finding to its source, and check that the passage says what the finding claims.

2. What does it do when the evidence is not there?

This is the most important question. Language models are built to produce plausible text, and a plausible answer to “does the supplier hold ISO 14001?” is easy to generate whether or not a certificate exists. A tool fit for compliance work needs an explicit outcome for “no supporting evidence located”, and it has to use it. Test this directly: run a requirement against a document set you know does not support it.

3. Does it keep related material apart from support?

An environmental policy is related to an ISO 14001 requirement. It is not evidence of certification. Tools that blur the two produce findings that look supported and are not. Ask how the tool labels material that is relevant but does not answer the requirement.

4. Who makes the decision, and is it recorded?

An AI tool can locate and prepare evidence. The decision that a requirement is met should belong to a person, and the record should show who decided, when and on what evidence. Be wary of tools that issue their own compliance verdicts or scores: an auditor will ask who made the decision, and “the software” is not an answer that holds up.

5. How does it handle old versions and conflicting documents?

Supplier files contain superseded certificates, earlier policy revisions and documents that disagree with each other. Ask whether the tool picks the current version of a record, and whether it tells you when two documents conflict. Many tools do neither reliably. That need not rule a tool out, but you should know, because it decides how much of the file the reviewer still has to read.

6. Which AI provider sees your documents, and on what terms?

Ask which model providers receive your content, what exactly is sent (whole files or passages), where processing happens, and whether your data can be used to train models. Ask whether there is a zero data retention agreement with the provider and, if there is not, what the provider’s retention terms are. In the UK, the ICO’s guidance on AI and data protection is a useful companion to these questions when documents contain personal data.

7. What does deleting a document actually do?

Deletion often means hiding a document from the interface while the file stays in storage and in backups. That can be reasonable, but you should know which it is, how long copies persist and how to have data removed for good.

8. Can you test it on your own documents first?

Demos use documents chosen because they work. Your files will have scans, spreadsheets, missing pages and documents in more than one language. A short paid pilot on a real file tells you more than any number of demos.

How DataStruct answers these questions

We also hold no SOC 2 report. The full answers, with the detail behind each one, are on the security and data handling page, and the product’s scope is on features. If citation checking is new territory, why document AI needs citation validation explains the mechanics.