Before your AI acts on a number, check it against your own

If you’re a CEO or CFO getting ready to let AI act on your company’s numbers, you can have each of those numbers checked against your own systems before it does. It takes your leaders agreeing what each number means, a check against something independent of whatever produced the number, and their sign-off before any AI acts on it.

A finance team would hold any new system to those controls. AI makes them easier to skip, because it builds fast and can report on its own work with confidence, and far more of the teams that build data pipelines are prioritizing it for building than for checking. In a 2026 survey by dbt Labs, which makes data tools, 72% of data practitioners and leaders said they now prioritize AI-assisted coding, while only 24% prioritize using AI to manage their pipelines, including testing and monitoring them.

The test report said the math was right

I saw what happens when the checking falls behind on a build of our own. Debt Hacker is a fintech app I invented for people paying down credit card debt, and it hasn’t launched yet. At Signal Scout, the firm I founded, we built the data pipeline and AI that turn a person’s card and bank statements into a plan for the month, with AI agents writing most of the code, and an AI-written test report said every step had been tested and the math was accurate to the penny.

I had AI agents build a second calculation, written a different way from the pipeline’s own, and audit the pipeline against it. The audit found a $12,000 test balance being told it could save approximately $181,835 in interest, when the true figure was $7,089. The pipeline’s answer was 26 times too high, and the step behind it was one the AI-run tests had themselves listed as never tested. Debt Hacker had no users and nothing had been sent, so the figure never reached anyone.

The audit traced that error, and the others it found, to copies of the payoff math scattered through the pipeline. We replaced them with one shared calculation, and in the 1,684 checks we ran afterwards it never disagreed with the second one.

What caught the error was a check built independently of the thing it checked. The test report grew out of the same build as the numbers it vouched for, while the second calculation was built apart from the pipeline, a different way. On a company’s own data, the independent source closest to hand is the system each number came from. The Debt Hacker story shows how we built the pipeline.

Agree what each number means first

A check needs a right answer to compare against, and for a company’s own numbers the right answer is whatever its leaders agree each number means. Gartner’s Rita Sallam warned in May 2026 that without “a clear understanding of the specific relationships and rules within an organization’s data,” AI agents “cannot operate accurately and are far more likely to hallucinate, introduce bias and produce unreliable results.” If sales and finance each count revenue their own way, your AI has no way to know which one you mean.

At Hideaway Report, a members-only luxury travel guide where I led digital marketing, marketing analytics and technology, teams that read the same numbers and worked from the customer journey paved the way for a 100X increase in the visitor to paid-subscriber conversion rate.

Agreeing what each number means is where our work with a company starts. Your leaders settle it while AI agents read every source and find where the numbers disagree, and you come away with the definitions your leaders signed off and the first use of AI we agree on, picked for what it pays.

Let AI read, and keep the math where a person can check it

I decide where AI belongs and where plain code is safer, and a number someone will act on should come from code that can be checked. In Debt Hacker, AI reads each person’s statements, the person checks and corrects the card details it found before anything is calculated, and the plan itself runs in plain code.

At a title research company, I designed the next phase of the software we built so that AI would turn each title search into data, and every examination decision would come from a rule a person can read and audit.

The numbers a company already trusts live somewhere today, in spreadsheets, an old database or the reports its finance team sends out, and a new pipeline’s numbers have to match them, or your leaders need to know why they don’t. I’ve built a mortgage lender’s data warehouse, moving its loan and hedging math out of spreadsheets into numbers refreshed several times a day, and moved a health sciences company from Oracle to Snowflake. On your data, AI agents build the pipelines from your apps, documents, emails and call transcripts, and senior builders like Patrick Hrabos, who join when the work needs them, check each number against the system it came from before your AI relies on it.

What you have when it goes live

Before we start, we agree what will be live within ninety days. If it isn’t, we keep building at no charge. Nothing goes live until your leaders sign off, and when it does, you have the pipelines, the tests behind them and your first use of AI running in your own accounts, all of it yours. After that, if you choose, AI agents watch the pipelines for breaks, and each month you get a short report in your own numbers on what broke, what we fixed and what changed.

If there’s a number your teams already disagree on and you want AI acting on it soon, bring it to a thirty-minute call, and we’ll start there.

Read next

What has to happen around an AI agent before it pays off

All writing