Why AI Agents Get Financial Figures Wrong, and the Four Fixes

The short answer. Language models generate the most plausible next token, not the true one, so an unsupported financial figure is produced by exactly the same mechanism as a supported one and looks identical. The four things that measurably help are: retrieve the figure rather than recall it, show the source row next to it, let the system decline, and verify numbers with arithmetic rather than with more language. Nothing eliminates the failure; these four move it from common to rare.

The word hallucination is a bad name for it, because it suggests a malfunction. Producing a confident wrong number is the system working exactly as designed. That is the uncomfortable part and it is why "we use a better model" is not a fix.

Why financial data is unusually dangerous ground

Three properties of this domain make it worse than most.

Numbers are precise, so a wrong one is specific and therefore convincing. "Revenue was $4.31bn" reads as researched whether or not it is true, in a way "revenue was large" does not.

Formats are near-identical across thousands of companies, so a figure from the wrong filing looks perfectly at home in the right one.

And the reader usually cannot check cheaply. Verifying one number means opening a document, finding a table and reading a footnote. Most people will not, which means the error survives. It is the same reason a headline free cash flow figure gets repeated across the internet for years after the filing it came from was restated.

Read that chart again, because it reframes the problem. Roughly six in ten of the errors we have investigated were a real number in the wrong place. That is not a reasoning failure at all. It is a retrieval and labelling failure, and retrieval and labelling have known engineering answers.

The four fixes, in order of how much they buy you

One: retrieve, never recall. Any figure a system states must have been fetched during the request from a source it can name. A model asked what a company earned will always produce something. A model handed the income statement and asked to read a line is doing a different and much easier task.

Two: show the row. Print the source next to the figure by default, not behind a hover. This does more than let the reader check. It changes what the system is allowed to output at all, because a claim with no row to attach cannot be rendered.

Three: let it decline. Every system needs a path that ends in "not established", and that path must be as easy to take as any other. In practice this is the hardest of the four to build, because a hedged sentence is always available and always scores better on a naive quality metric than a refusal.

Four: check arithmetic with arithmetic. If a system says margin was 38%, compute 38% from the two retrieved numbers and compare. Do not ask a second model whether it looks right. Two language models agreeing is not verification, it is correlation.

What ours does and does not do

Every figure on a SteadyShares company page is retrieved, not recalled, and carries the source it came from. That is the one architectural decision in the product we would defend without qualification, and it is why our failure mode is a blank rather than a wrong number.

Where we are weaker: our agent still prefers a hedge to a refusal, and our arithmetic verification runs on the headline figures rather than on everything. Both are named in the roadmap and both are in build. We are writing them here rather than waiting to announce the fix, because a page about honest failure modes that hides its own would be a strange page to publish.

What this cannot tell you

None of these four fixes gets you to zero, and any vendor implying otherwise has either not measured or is not telling you. Retrieval can fetch the wrong document. Sources can be printed next to a figure that does not come from them if the plumbing is wrong. The correct posture is not trust or distrust; it is knowing which numbers you would want to check before acting, and checking those.

The bottom line

The category talks about hallucination as if it were a reasoning problem, because reasoning problems sound like they need a smarter model, and a smarter model is a thing that can be sold. Most of it is plumbing: right number, wrong period. Ask any tool you are considering to show you where a figure came from. If it cannot, the number is a guess with good posture. Ours shows it on every company page, which is the check we would want if we were you.

Educational information, not financial advice. Where this page states a figure about our own coverage it is read from the database as the page loads, so it is today's number rather than the day this was written.