Keeping an LLM from making up numbers.

The problem

Ask a language model "is this token safe?" and it will answer. It will say the top holders own 12% of supply, that liquidity looks healthy, that the dev hasn't sold. All of it reads well and none of it has to be true. A model's job is to produce a likely next word, and a likely-sounding percentage is easy to produce.

For a token checker that's the worst failure there is. A wrong "looks fine" costs someone money; a confident wrong number is worse than "I don't know". So Tick is built on one rule: the model decides what to look up and how to say it, and never what the numbers are.

How a question flows

Tick runs on a Cloudflare Worker at /tick. It calls a model (Claude Haiku by default) with Fivebar's MCP tools, the same tools AI agents use at fivebar.xyz/mcp, called in-process rather than over the network. The tools read chain data directly: Robinhood Chain transfers, Solana accounts through Helius, swap routes through 0x, Hyperliquid's public API. Code below is lightly trimmed from the production source.

Rule 1: tools are the only source of numbers

The system prompt opens Tick's rules with this, and every other rule bends around it:

- Every number, and every claim about a specific token, wallet, agent, seller
  or price, comes from a tool result in this conversation. Never guess or fill
  in. If a tool fails or finds nothing, say so.

A prompt rule is a request, not a guarantee, which is why the rest of this article exists. But it does two useful things. It makes "the tool found nothing" an acceptable answer, which removes the pressure to fill a gap. And it gives us something testable: if every number must come from a tool, then every number in an answer should be findable in the tool results. The measurements below check exactly that.

The tools are shaped to make this easy. They return the figures a person would quote, already computed and rounded: top10Share, roundTrip.lossPct, dev.sold. The model never has to add up holders or convert units, so it has no arithmetic to get wrong. The heavier checks also return a method string saying how the numbers were made, so Tick can explain a figure without inventing an explanation.

Rule 2: the model never draws

A bubble map with 40 holders is 40 numbers. If the model produced them, any one could be wrong and nobody would notice. So charts don't go through the model at all. When a tool returns, the Worker builds the drawing from that tool's own JSON and sends it to the page next to the model's text:

const res = await deps.mcp({ method: "tools/call", params: { name: c.name, arguments: c.args } });
const text = res?.result?.content?.[0]?.text ?? "no result";
// drawn by the page from the tool's own numbers, never from the model's
if (c.name === "wallet_flows") visual = flowsVisual(text) ?? visual;
if (c.name === "token_holders" || c.name === "bundle_check") visual = bubblesVisual(c.name, text) ?? visual;
if (c.name === "token_report") visual = reportVisual(text) ?? visual;
out = text.slice(0, 12_000); // what the model sees

This changes what the model needs to say. The page already shows the four report sections and the bubble map, so the prompt tells Tick to lead with the single most serious reason and leave the rest to the drawing. Fewer numbers in the text means fewer chances to misquote one.

Rule 3: the output is a contract

Tick's last message must be one JSON object: a short say, an optional verdict stamp, up to five facts and up to three follow-up questions. The Worker parses it and drops anything outside the contract:

export const STAMPS = ["PASSED", "LOOKS OK", "CAREFUL", "FLAGGED", "WASH", "NOT YET", "OPERATOR"];

const stamp = STAMPS.includes(j.stamp?.toUpperCase()) ? j.stamp.toUpperCase() : null;
const facts = (j.facts ?? []).slice(0, 5).map((x) => ({ k: clean(x.k, 40), v: clean(x.v, 40), bad: x.bad === true }))
                             .filter((x) => x.k && x.v);
const follow = (j.follow ?? []).slice(0, 3).map((x) => clean(x, 40));
const say = clean(j.say, 600);

The stamp list matters most. A model left to its own words will invent verdicts like "SAFE" or "LEGIT". Tick can only pick from seven, each tied to a tool outcome in the prompt (token_risk high → FLAGGED, low → LOOKS OK), and "LOOKS OK" is the strongest thing a clean check can earn. There is no "SAFE". A stamp the model makes up is replaced with none.

Rule 4: a bounded loop

The model can call tools for up to four rounds, three calls a round. On the fifth round the tools are taken away, so it has to answer with what it has:

for (let round = 0; round < 5; round++) {
  const last = round === 4; // out of tool rounds: answer with what's there
  const m = await chat(turns, last ? null : tools);
  const calls = m.calls.slice(0, 3);
  if (!calls.length || last) return send({ type: "answer", ...parseAnswer(m.text), visual });
  ...
}

The limits keep cost and latency predictable, and they also block a quieter failure: a model that keeps calling tools until one returns something that fits the answer it has already settled on. Tool results are capped at 12,000 characters, and only read-only tools are on the list: Tick can quote a swap, never build one.

Rule 5: decide what's being asked before reaching for tools

The first version of Tick was too eager. Every message was treated as something to look up. Here is a real exchange from before the fix:

Nothing in it is invented, so Rule 1 held. It just answered a question nobody asked. "Tools are the only source of numbers" had quietly become "tools are the only way to answer". The fix was a routing step at the top of the prompt: work out whether the message is a check (an address, a mint, a ticker, "check this"), a conversation (Fivebar, other products, how something works) or unclear, and only then pick tools. Conversation answers can use general knowledge but must say when it may be out of date, and still may not invent another product's features, prices or dates. The same question now:

Changing a prompt changes everything it touches, so we didn't trust one good example. We measured it.

Measuring it

The eval runs Tick's real code (the same handleTick the site runs) with the real model and Fivebar's live tools, on 26 questions in four kinds:

• Checks (10): a Solana mint or a Robinhood Chain token with "is this safe?", "who holds it?", "was it bundled?", "is the dev selling?"; a wallet; perps and stock tokens. Each names the tools that must be called and the ones that may be.
• Conversation (8): "What is x402?", "How do you detect a bundled launch?", "Does Fivebar hold my money?", the Coinbase question. No tool should be called.
• Unclear (6): "is pump legit?", "check coinbase", "is GATO safe?" (a ticker with no address). No name may be looked up as if it were a token or an agent.
• Follow-ups (2): "who holds it?" after a safety check. The right tool must be called with the token from the earlier turn.

Every question ran 3 times with the current prompt and 3 times with the prompt from before the fix (156 runs in all), because the same question doesn't always get the same answer. The old prompt is swapped in on the request to the model; everything else is identical. Each run saves every tool call with its full result, so any number can be checked by hand later.

The grounding audit

Routing says whether Tick asked the right question. Grounding says whether it reported the answer honestly. For every check answer, the audit pulls each number out of the text and the facts table, removes addresses first so their digits don't count, and looks for it in that conversation's tool results and in the question itself. A number counts as traced if some source number rounds to it at the precision Tick wrote: "3.3%" matches a tool's 3.32, "$13.8M" matches 13814537, and a percentage may match a 0-1 share times 100. Dates must appear as written.

const near = (n) => src.some((v) =>
  Math.abs(Math.abs(v) - Math.abs(n.value)) <= n.tol ||          // 3.32 → "3.3"
  Math.abs(Math.abs(v) * 100 - Math.abs(n.value)) <= n.tol);     // 0.0332 → "3.3%"
// n.tol is half a unit at the precision Tick wrote: "3.3" → 0.05, "$13.8M" → 50,000

The 11 numbers the audit couldn't trace

We read each one against the tool output of the run it came from. None was invented: 6 unit conversions, 3 formats the audit can't read, 2 small sums. The format cases are the audit's own blind spot. The conversions and sums are Tick doing small arithmetic the tools didn't do for it. They were all correct, and they're the cases a mechanical audit is built to surface: a number that isn't copied straight from a tool is a number worth a second look.

What this doesn't catch

• A tool can be wrong. Tick quoting a tool faithfully is only as good as the tool. While building the Solana checks we found our own safety check flagging busy pump.fun tokens as "possible honeypots": its buy-then-sell test only tried USDC, and those tokens trade against SOL. Tick reported the tool correctly; the tool was wrong. The fix was in the tool, with a test, not in the prompt.
• Traced isn't the same as right. The audit proves a number exists in the tool output, not that it was taken from the right field. A model that quotes a holder's share as the top-10 share would pass. Small answers and named fields make this less likely; it isn't impossible.
• Conversation answers aren't audited. They have no tool results to trace to, so they're held to a different rule (say when knowledge may be out of date, never invent another product's features or prices) that this eval only checks by reading.
• 3 runs per question is a small sample. It catches regressions like the one in Rule 5. It doesn't put a tight bound on rare failures.

What we'd tell anyone building this

1. Make tools return the sentence-ready figure, rounded and labelled, so the model copies rather than computes.
2. Never let the model produce a chart's data. Draw from the tool result on a separate path.
3. Constrain the output to a schema, and keep any verdict to a closed list with no word stronger than the evidence.
4. Give "nothing found" a respectable place in the answer.
5. Route before you retrieve: not every question is a lookup.
6. Audit numbers mechanically, run every case several times, and keep the old prompt around to compare against.