An AI assistant for compliance assessments

The first time my document scanner read a genuinely good policy, it told me the policy covered nothing.

Zero controls fully covered. Dozens marked as partial. A well-written business continuity policy, the kind an auditor is happy to read, looked like a failure.

The scanner wasn't broken. It was answering the wrong question.

And fixing that question turned out to be the most important design decision in the whole project.

Here's what I built, what the AI does and doesn't do, and why the lesson goes far beyond compliance.

First, what a compliance assessment actually is

If you've never sat through one, here's the short version.

A compliance assessment measures an organization against a framework. For cybersecurity, that's often the NIST Cybersecurity Framework. For business continuity, it's ISO 22301. Each framework is a long list of controls: things a well-run organization should have in place. NIST CSF 2.0 alone has 106 of them.

For every control, a consultant asks questions, looks at evidence and gives a score. Then the scores turn into a plan: what to fix first, who owns it, and what "better" looks like.

Evidence usually means documents: policies, plans, procedures, records of tests. Reading them carefully is a big part of the job.

My client is an IT advisory firm that does this for its own clients, inside a web portal its consultants already use.

The portal handled the structure well. What it didn't do was help with the thinking. And the thinking is where the hours go.

What does this control actually mean for a company this size? How do I score it? What evidence should I ask for? Does the policy the client just sent me cover it, or only mention it?

Multiply those questions by a hundred controls and you get the real cost of an assessment.

Not the clicking. The reading and the judgment.

Help where the consultant already is

I wanted the help to live inside the portal, next to the work. No separate tool, no copy-pasting into a chatbot in another tab.

So that's where it went.

Next to every question, there's now an Ask AI button. It knows which control you're looking at, so you can ask what it means, how to score it or what evidence to look for, and get an answer in context.

It also speaks the portal's language. For business continuity, it uses the same maturity levels the consultant sees on the screen, so its answers line up with the scores they're about to give.

On the whole assessment, an analysis pulls out the weakest areas, the gaps and a set of recommended activities. A consultant can add any of them to the improvement plan, with an owner. Or ignore them.

On the summary page, a small helper next to each score explains why it sits where it does, which controls are holding it back, and what it would take to reach the next level.

And for documents, a scan: upload a client's policy or process and check it against the framework.

The first three were about putting good answers in the right place.

The scan is the one that taught me something.

The day the scanner failed a good policy

The first version of the scan did what seemed obvious. Take the document, take every control in the framework, and check the document against each one.

Thorough. Systematic. Completely wrong.

A business continuity policy is not supposed to contain your incident procedures, your supplier contracts or your exercise reports. It sets direction. It states commitments. It says who is responsible for what. That's its whole job.

So when you check a policy against every control in a standard, it falls short on almost all of them. Not because it's bad, but because it was never meant to cover them. And the scanner marked every one of those as partial or missing.

That's where the zero came from.

The result was technically defensible and practically useless. Nobody trusts a checker that fails good documents. And no consultant can hand a client a report saying their solid policy covers nothing.

An AI that fails good documents is worse than no AI at all.

Asking the right question

The fix wasn't a better prompt. It was a better question.

Not "does this document satisfy the framework?" but "what is this document for, and does it do that well?"

So the scan now works in two ways.

Against one control. The consultant picks a control, or clicks the scan icon right next to it, and the document is judged only against what that control requires. It matches how a consultant actually thinks: I have this policy, does it evidence this control?

Against the whole document. If you don't pick a control, the scan first works out what kind of document it is and which controls it was written to evidence. It assesses those, and only those. Everything else is listed as not in scope for this document.

Not in scope is not a gap. That distinction alone changed how the results read.

Each result reads like a short review: a verdict, how many of the control's required elements were found, a quote for each one, and a single line on what's missing to fully meet it. Good-practice tips appear too, but as advice. They never lower the verdict.

There was one more trap, and it's a subtle one.

A policy will usually say that a continuity plan must exist. Early on, a test showed the scan reading that line and happily declaring the plan control met.

It isn't. A policy that says you'll have a plan is not the plan.

So now, when a document only requires something to exist, that control shows as supported by this document, with no verdict. The evidence for it is the plan itself, and the result says so in plain words.

What the AI never decides

This is the part I care about most, in every project.

The AI reads fast, explains, drafts and points. It does not decide.

It never sets a score. Scores belong to the consultant.

It never approves a document. Every scan result says so: it's an AI-assisted pre-check, not an audit opinion.

Nothing goes on the improvement plan unless a consultant adds it.

And when the scan says a document covers something, it has to prove it. Every quote it gives as evidence is checked word for word against the document. If the sentence isn't really there, it's flagged for a person to check, never shown as proof.

Same for the verdict. Meets, partially meets or does not meet is not the AI's opinion. It's worked out by fixed rules from the required elements that were actually found, so the same findings always give the same verdict.

Consistency mattered as much as accuracy. In early runs, the same policy could come back with slightly different results from one run to the next. That kills trust: scan a document twice, get two answers, and you'll believe neither. Now each control is always checked against the same list of required elements, and the verdict comes from the rules, not from the model's mood.

That's not distrust of AI. It's knowing what it's good at.

A language model is a brilliant reader and an overconfident guesser. You want the first without the second.

Where it stands

The assistant is now with the client, on their own server, being tested on real assessments.

In testing, the same good policy that once covered nothing came back close to meeting its control: all but one of its required elements clearly there, a quote for each, and the last one only implied, named as the thing to fix. I ran it three times. Same answer three times.

A policy on a completely unrelated topic came back as not fitting the standard at all, instead of being forced into matches. An earlier version wasn't so careful. It matched that same document to eight controls, and four of its quotes did not exist anywhere in the document.

That's exactly what the word-for-word check is for.

For a typical policy, a focused check against one control takes under a minute and costs about three US cents in AI usage.

None of that replaces the consultant. It gives them a first read they can verify, and more of their time for the part only they can do.

This has nothing to do with compliance

Strip out the frameworks and the vocabulary, and the pattern is everywhere.

A quality manual checked against a certification standard. HR policies checked against a new law. Supplier questionnaires checked against your security requirements. Contracts checked against a checklist. Any time a person reads a document to decide whether it meets a list of requirements, the same approach works.

Let the AI do the reading and the explaining. Let code check the evidence and apply the rules. Let a person make the call.

The hard part is rarely the AI. It's deciding what the AI is allowed to say, what it has to prove, and what stays with a person. Get that right, and the rest is engineering.

And judge each document on what it's for. Not against everything at once.