In Part 1, I argued that token consumption is useful information, but not a sufficient measure of the economics of AI. The cost of producing an outcome can depend on model choice, infrastructure, retrieval, tools, retries, human intervention and the architecture connecting them.

The logical next step is therefore to decide what we actually want to cost. Before Finance can compare alternatives, calculate a meaningful unit cost or decide whether something is becoming more economical, we need to agree on the unit itself. Is it a model call, a completed task, a broader use case, a business process, or ultimately an acceptable outcome?

Once we start thinking of the problem in that way, our choice of application architecture also becomes part of the economics.

The Architecture Is Part of the Economics

A frontier model may be the best option, but that does not mean every task needs it. Smaller models might do the trick for simple or routine work, whilst more difficult tasks are performed elsewhere. That is the basic logic behind the router assignment and services such as OpenRouter: use the capability you need rather than automatically buying the most capable model available.

The same applies to where the model runs. A proprietary API, an open-weight model sitting on cloud infrastructure, reserved capacity or local inference can all perform similar work but with different cost structures. API services are largely variable-cost; dedicated infrastructure makes utilisation more important; self-hosting may reduce external model charges but replace them with infrastructure, engineering and energy costs.

Higher up the stack, caching, retrieval and agent design create the same sort of trade-offs. They can reduce cost or enhance performance, though they also have their own consumption to be considered. A cheaper model that creates more retries or human checking may not produce a cheaper process overall.

For Finance, the key comparison is not token prices between providers, but the total economics of different methods for completing the same work.

What Should We Cost?

Once we move beyond token prices, a familiar management-accounting problem appears: deciding what it is we actually want to cost.

The most obvious answer may be the original use case, particularly because this is often how investment decisions were originally framed. A company funds a customer-service assistant, a finance copilot or a document-review tool and, quite reasonably, wants to know what that use case costs and whether it is delivering value.

The use case is a helpful starting point, but not always the right endpoint. A broad use case can include diverse activities. A “Finance Copilot”, for example, might summarise documents, investigate variances, draft commentary and answer policy questions. These activities may use different models, require different amounts of context and carry very different quality or control requirements. One average cost for the whole application could hide most of what management actually needs to understand.

There is also a practical problem in that an AI use case will change over time. What was approved as one idea may become something rather different once people start using it. That does not make the original business case irrelevant, but it does mean that forcing every future cost and benefit back into the categories chosen at the start can become artificial.

I think it is useful to think of it as different layers: technical activity or event, the task, the bigger use case, the business process or product it supports, and finally, the outcome. We may want costs at several of these levels rather than trying to nominate one perfect unit in advance. The lower levels help explain what is driving consumption and cost, whilst the higher levels support investment appraisal, for example.

Several vendors or deployment models make this even more important: one provider may bill by tokens, another through provisioned capacity, while an internally hosted model creates infrastructure and support costs elsewhere. Those charges still need to be reconciled back to the accounts, but for management purposes, they also need to be translated into a common view of the work being performed.

The point is not to build an elaborate allocation model on day one, but to make sure the information captured while experimenting leaves us with enough flexibility to understand the economics later.

Cheap Is Not Always Economical

Once we start looking at unit costs, there is a danger of becoming too pleased with the cheapest answer.

A model that costs less per call may also fail more often, require more retries or create more work for the person reviewing the result. The ICLR Cost-of-Pass paper helpfully looks at inference cost together with the probability of getting the answer right, rather than treating cost as an isolated factor. That is much closer to the way Finance would normally think about an operating process.

But in reality, just deciding if an answer is 'right' or 'wrong' can be too simple. An acceptable outcome may also need to be timely, secure, resilient, traceable and compliant. In some cases, it may need human approval regardless of how accurate the model appears to think it is.

In financial services, for example, where I’ve spent a large part of my career, this distinction matters. A cheap model that occasionally produces a poor marketing summary is one thing. A cheap model that produces an unreliable cash-flow forecast, misses a control failure or mishandles sensitive data is quite another. The expected cost of failure can easily outweigh any cost savings from optimised token consumption.

That points to a more useful measure than cost per call: the cost of producing an acceptable outcome. It may include model usage, tools, retries, infrastructure, human review and, where relevant, the expected cost of failure.

Agentic systems make this slightly more complicated again because the cost of completing the same nominal task may vary quite a lot. One case may be completed in two calls; another may take several tool interactions, retries and an escalation to a person. In that environment, averages can hide useful information. Median cost, higher-percentile costs, retry rates and human-intervention rates may all tell us more about the economics than a single average cost per task.

The implication is that cost optimisation has to be done alongside performance and risk, not in isolation. The cheapest architecture is not necessarily the most economical one.

Sources:

1. Cost-of-Pass: An Economic Framework for Evaluating Language Models by Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yuksekgonul, James Y Zou: https://proceedings.iclr.cc/paper_files/paper/2026/hash/399d54cce01e782192b3d7ecbe607a96-Abstract-Conference.html