Your AWS Bill Is Not an Observability Tool: Token Usage Monitoring for Amazon Bedrock

Every single day
And every word you say
Every game you play
Every night you stay
I'll be watching you

The Police, I’ll be watching you

A team running generative AI workloads would love to be able to answer this question at any time of day: how many tokens are we consuming, and how much do they cost?

If your answer is "let me check the billing console," this article is for you.

I recently learned, in the most stressful way possible, that the billing console is a reconciliation tool, not an observability tool. It's delayed, it's aggregated, and, in rare but very real cases, it can be wrong.

How wrong? In my case, wrong by a factor of one thousand. I woke up (well, technically, I finished lunch) to a bill of $58,000 for a workload that should have cost about fifty bucks. The only reason I could prove the error, get it fixed in days, and get the underlying bug patched for everyone was that I had application-side token telemetry that didn't depend on AWS metering.

The incident

Let me set the scene. You might already know that I test new services and share what I learn: in February 2026, as a Community Builder (now I’m an AWS Hero!), I was experimenting with autonomous agents on Amazon Bedrock using OpenClaw, comparing different foundation models: DeepSeek v3.2, Amazon Nova Lite, and, later, Kimi K2.5 by Moonshot AI, which landed in the Bedrock catalog right on time.

Kimi's pricing looked great: $0.60 per million input tokens and $3.00 per million output tokens. But the OpenClaw Node.js library didn't get along with Kimi's tool-calling implementation over the native Bedrock Converse APIs: I got failing calls, inconsistent responses, and actions weren't happening silently.

I started fiddling around and debugging the library, but I resorted to another method: reading the documentation (a shockingly effective debugging technique), which led me to find a workaround: Project Mantle, Bedrock's distributed inference engine announced at re:Invent 2025, which exposes OpenAI-compatible API endpoints as a drop-in replacement without changing application code.

Point your existing OpenAI-style client at https://bedrock-mantle..api.aws/v1, generate a Bedrock access token, and everything just works. And it did: I switched on February 17th, and the tool-calling issues disappeared.

I was being a responsible cloud citizen. Daily billing checks in the organization's management account. A $100 budget alarm on the sandbox account. A forecast alarm on top. On February 23rd, my whole organization's monthly bill sat at roughly $140, comfortably covered by AWS credits, with the Bedrock experiments contributing their usual $3 to $5 per day.

Then, on February 24th, right after lunch, two AWS Budgets notifications arrived at once: the actual spend was $56,265.59. The forecasted month-end was $70,161.62, with no gradual ramp and no intermediate alerts. The billing jumped from $140 to over $56,000 overnight, then settled above $58K, including taxes.

After the physiologically mandatory moment of panic and a heart attack, I did what we do at beSharp when a customer calls with a scary bill: I started measuring.

Finding the issue

The billing breakdown for us-east-1 showed the problem, but in a subtle way, and it took a moment to believe it. Kimi K2.5 input tokens through Mantle were 77,228,206 units of 1K tokens, billed at $0.0006 per 1K, for a total of $46,336.92. Output tokens were 195,679 1K-token units at $0.003, totaling another $587.04.

77 million thousand tokens. That's 77 billion tokens: that's an amount of text you could not realistically push through an interactive agent workflow in a week, even if you tried.

Application-side telemetry saved me because OpenClaw tracks token usage on its own, from the usage fields returned by the API. Its numbers for Kimi K2.5 through Mantle were about 72.1 million tokens across 547 messages. Not 72 million thousand tokens, but plain 72 million.

To make sure OpenClaw's accounting was trustworthy, I checked token usage against the models I had used accessing Bedrock native APIs:

Model	API path	App-side telemetry	Billing console	Match?
DeepSeek v3.2	Native Bedrock (Converse)	~61.8M tokens / 844 messages	61,910 x 1K tokens	Yes
Amazon Nova Lite	Native Bedrock (Converse)	Consistent	Consistent	Yes
Kimi K2.5	Mantle (OpenAI-compatible)	~72.1M tokens / 547 messages	77,228,206 x 1K tokens	Off by ~1,000x

It became clear to me that something in Mantle billing was treating tokens as 1 K units, multiplying my bill by 1000; so, instead of $46.34, I was being billed $46,336.92. It was a simple misplaced comma that was costing me like a new car (and not even a cheap one!).

The inflated bill was the bug. What turned it into a week of silent damage was the absence of signals:

• CloudWatch token metrics weren't populating for Mantle invocations. Invocation activity was visible, but the token usage metrics that normally appear for Bedrock were empty for the new endpoint.
• Billing data was dumped all at once, going past my $100 budget alarm and notifying me when my spending was already at $56K.
• Cost Explorer eventually showed the anomaly started on February 17th, the exact day of the switch to Mantle. But "eventually" meant a full week later.

Every safety net I had was downstream of AWS's own metering. When the metering itself was broken and delayed, all of them failed together. This is the correlated-failure mode that should worry you: budget alarms, forecast alarms, anomaly detection, and Cost Explorer all share the same upstream data source.

How it got resolved

I built a report with the comparison you saw above: native APIs were correct, while Mantle costs were inflated, the 1,000x factor was consistent across the dataset. I opened a support case (even on the basic support plan, you can open a billing issue) and described the math. The conversation happened on a Saturday; by Tuesday, the billing was corrected, and the bug was addressed at the service team level. Fixed for me, and fixed for everyone who would have hit it next.

Two things made this outcome possible: data and good faith. I wasn't asking AWS to trust my feelings about the bill; I was showing numbers from an independent measurement system. 

To be fair to the service, Project Mantle is a genuinely beautiful piece of engineering. It is a distributed inference infrastructure that solves real migration problems in the time it takes to change a configuration file. After a little time had passed since my incident happened, AWS has published dedicated documentation for monitoring the bedrock-mantle endpoint. 

Early adopters find bugs; that's part of the deal. The question is whether you find them, or your CFO does.

You can find the full story on my personal blog. In this article, I want to focus on the approach that lets you catch this kind of problem before you have a look at Cost Explorer.

Tokens are the new billable unit

We've spent 15 years building observability for the classic billable units of the cloud: instance hours, GB-months, and requests. We alert on them, tag them, and assign them to teams and cost centers.

Generative AI introduces a new billable unit, the token, with some uncomfortable properties:

• Consumption is decided by the model, not just by you: output length varies, reasoning tokens exist, and also agent loops make things more difficult to estimate. We also have to keep in mind that two similar requests can differ significantly in cost.
• A traditional APM is not designed to track token usage: a request that costs $0.002 and one that costs $0.40 share the same latency profile and the same HTTP 200 response status code.
• Multipliers hide everywhere. Cache reads and writes, burndown rates against TPM quotas, per-model pricing differences, and cross-region inference, mapping "what my app did" to "what I pay" is not trivial.
• Agents amplify everything. An autonomous agent stuck in a loop doesn't throw errors. It just quietly consumes tokens with a 0% error rate until something (hopefully an alarm, possibly your credit card limit) stops it.

We have to treat token usage with the same approach we use for any other production workload: gather provider-side metrics, application-side telemetry, and perform financial reconciliation. 

Let's build it, layer by layer.

Layer 1: AWS-native visibility (necessary, not sufficient)

Everything in this layer comes directly from AWS. It's the foundation, it's cheap, and you should enable all of it. Just remember what my incident proved: this layer shares fate with AWS's own metering.

CloudWatch metrics in the AWS/Bedrock namespace. Bedrock automatically publishes Invocations, InvocationLatency, InputTokenCount, and OutputTokenCount per model. These are your baseline: alarm on token counts, not just on invocation counts, because in the token economy, volume and cost are only loosely correlated.

The newer operational metrics. Bedrock now also emits TimeToFirstToken for streaming APIs and EstimatedTPMQuotaUsage, which reflects how your requests actually burn down your tokens-per-minute quota, including burndown multipliers. The quota metric is particularly interesting for cost awareness: if your estimated quota consumption differs significantly from your application's token accounting, something is wrong somewhere, and both possibilities (your bug or theirs) are worth investigating.

Model invocation logging. It's disabled by default because Bedrock will write metadata, requests, and responses for every invocation to CloudWatch Logs and/or S3, including per-invocation input and output token counts. This is the closest thing to a provider-side audit trail, and CloudWatch Logs Insights lets you query it: top sessions by token usage, requests grouped by input size, and per-identity attribution. Check if you really need to enable this: you will have to deal with a lot of data, as well as potential PII or, in the worst case, sensitive data.

CloudWatch generative AI observability. The pre-built Model Invocations dashboard gives you token counts by model, daily token trends, throttles, and a curated invocation table without building anything. If you run agents on Bedrock AgentCore, the same capability extends to session-level token usage and traces, following OpenTelemetry conventions.

Application inference profiles and cost allocation tags. If multiple applications or teams share a Bedrock account, create application inference profiles and tag them. Untagged Bedrock usage is a single undifferentiated line in Cost Explorer, and "who spent this?" becomes archaeology.

AWS Budgets and Cost Anomaly Detection. If you don't have them in your AWS account, log on now to the console and enable them, and layer them with intermediate thresholds: don't jump from $100 straight to panic, and also internalize their limitation: they trigger on billing data, and billing data for a brand-new API layer may be delayed, batched, or, once in a very long while, wrong.

One honest note from experience: for new endpoints and services, verify that these signals are actually being populated before trusting them. In my case, the Mantle endpoint produced no token metrics at all for days. An empty dashboard looks a lot like a quiet, healthy workload.

Layer 2: OpenTelemetry GenAI conventions (your independent witness)

This is the layer that saved me, but in a more formalized form. It’s based on the principle that your application sees every API response, and every API response contains a usage block. If you capture it, structure it, ship it to a backend you control, you have a token accounting system that does not depend on your provider's metering pipeline.

You can do this with custom logging (which is essentially what OpenClaw's built-in tracking does), but the industry has converged on the OpenTelemetry GenAI semantic conventions. It is a standard developed by the OTel GenAI SIG that defines a common vocabulary for LLM telemetry. This makes it easier to gather uniform metrics and spans whatever platform you are using: a Bedrock Converse call, a LangChain agent, or an OpenAI-compatible endpoint will always have a common representation.

At the core, every model call becomes a span carrying attributes like:

gen_ai.provider.name        = "aws.bedrock"
gen_ai.request.model        = "moonshot.kimi-k2.5"
gen_ai.operation.name       = "chat"
gen_ai.usage.input_tokens   = 3412
gen_ai.usage.output_tokens  = 287
gen_ai.response.finish_reason = "tool_use"

Plus standard metric instruments: gen_ai.client.token.usage as a histogram of tokens per request, and gen_ai.client.operation.duration for latency. From these two, you can derive everything that matters: tokens per session, cost per feature (token counts multiplied by your pricing table), p95 latency per model, and, crucially, a continuously updated "what I should be billed" figure.

The architectural beauty is the decoupling. Your applications export OTLP to an OpenTelemetry Collector, which fans out to whatever backends you want: CloudWatch (via the AWS Distro for OpenTelemetry), a self-hosted stack, Langfuse, all of them simultaneously. Instrument once, route anywhere, switch backends without touching application code.Three practical notes for the real world:

1. Auto-instrumentation gets you 80% for free. Libraries like OpenLLMetry and OpenLIT instrument popular LLM SDKs and emit gen_ai.* spans automatically. If you use an OpenAI-compatible endpoint (like Mantle, or LiteLLM as a gateway), the OpenAI instrumentation applies with a base URL change, exactly like the endpoint itself.
2. Emit a cost metric, not just token counts. A small piece of pipeline logic that multiplies token usage by your per-model pricing turns telemetry into a near-real-time spend dashboard. My "$3 to $5 a day" intuition was a mental version of this; make it a metric and an alarm instead.
3. Mind the maturity. Most gen_ai.* conventions are still under development, so attribute names can shift between versions. The core token usage attributes are the most settled part; pin your instrumentation library versions and review upgrades.

Here's the part that should make you sit up: with this layer in place, my incident would have been detected the same day (if billing data were not delayed and batched). A scheduled comparison between sum(gen_ai.usage.input_tokens) and the provider-reported figures would have screamed on February 18th, not the 24th. Not because the app-side numbers stop the wrong bill, but because a 1,000x divergence between two measurements of the same thing is the loudest possible alarm.

Layer 3: Langfuse (making token data explorable)

Metrics tell you that you're burning tokens. When you're running agents, you also need to know why, and that's a tracing problem. This is where an LLM-specific observability platform earns its place. If you use Langfuse, you can self-host it, keeping prompts and completions inside your own account. Please note that the same reasoning for model invocation logging applies here: you have to take care of potential PII and sensitive data.

Langfuse acts as an OpenTelemetry backend: it exposes an OTLP endpoint, natively ingests gen_ai.* spans, and turns them into a purpose-built UI. On top of the raw traces, you get:

• Full agent traces: every step of a session as a hierarchy of spans, with prompts, completions, tool calls, and per-step token usage.
• Cost tracking with model pricing: Langfuse maintains per-model prices (and lets you define custom ones, useful the day a new model like Kimi K2.5 lands and you try it on day one), so every trace carries a dollar figure.
• Attribution dimensions: user IDs, session IDs, tags, and metadata, so token spend can be sliced by customer, feature, or experiment.
• Loop and anomaly spotting: the pathological agent patterns (a session consuming 50,000 tokens for a task that normally takes 3,000, retry storms, runaway tool loops) are visually obvious in a trace tree and trivially queryable.

Deployment-wise, Langfuse runs happily on ECS or on the Kubernetes flavor of your choice, and the sizing for most teams is modest. If self-hosting isn't a constraint for you, the managed cloud version removes even that effort.

Putting it together: a reference architecture

The complete picture, from an application on ECS/EKS/Lambda calling Bedrock:

1. Application layer: LLM SDK calls instrumented with OTel GenAI conventions (auto-instrumentation where possible). Every invocation emits a span with token usage and a metric data point.
2. Collector layer: an OpenTelemetry Collector (sidecar, DaemonSet, or Gateway) receives OTLP and fans out: traces to Langfuse, metrics to CloudWatch via ADOT (or managed Prometheus/Grafana if that's your stack).
3. Provider layer: Bedrock model invocation logging enabled to CloudWatch Logs; native AWS/Bedrock metrics and the generative AI observability dashboards on top; application inference profiles with cost allocation tags.
4. Financial layer: set up AWS Budgets using different thresholds, Cost Anomaly Detection, and Cost Explorer for cost attribution.
5. The reconciliation loop: a job that periodically compares application-side token usage against provider-side figures. It can also compare billed usage from Cost Explorer. Divergence beyond a tolerance threshold pages someone.

Layers 1 and 4 you probably have. Layer 2 is a few days of engineering that pays for itself the first time anything goes weird. Layer 3 makes your AI team faster every single day. Layer 5 is twenty lines of code and the difference between finding a metering bug in one day and finding it when the invoice arrives.

Trust, but verify

My story ended well because I had the data to demonstrate that the bug was real, and the good part was that the fix was made available to everyone. Project Mantle remains a solid addition to Bedrock, and I keep using it. Early adopters find bugs; that's the tax we pay for playing with new toys, and honestly, I keep paying it gladly.

But the lesson generalizes beyond a single bug in a single API layer. In the token economy, your provider's metering is a measurement, not a ground truth, and any single measurement can be delayed, incomplete, or wrong. The teams that will operate AI workloads confidently are the ones that measure independently, reconcile continuously, and can put a number on the table when something doesn't add up.

Your bill should be a confirmation of what your telemetry already told you. If it's ever a surprise, in either direction, your observability has a gap.

Have you already instrumented token usage in your GenAI workloads? Ever caught a discrepancy between what your application measured and what you were billed? Let us know in the comments, and if you want help designing the observability layer for your AI workloads, you know where to find us!

---

About Proud2beCloud

Proud2beCloud is a blog by beSharp, an Italian APN Premier Consulting Partner expert in designing, implementing, and managing complex Cloud infrastructures and advanced services on AWS. Before being writers, we are Cloud Experts working daily with AWS services since 2007. We are hungry readers, innovative builders, and gem-seekers. On Proud2beCloud, we regularly share our best AWS pro tips, configuration insights, in-depth news, tips&tricks, how-tos, and many other resources. Take part in the discussion!