For a while, Amazon and AWS teams relied on Claude Sonnet 4.6 to tag tickets in an inner system without breaking the bank.
Sonnet 4.6 fit the goldilocks choice of balancing intellect alongside cost. Considering it dealt alongside one of the world’s largest inner ticketing systems, the nine duration old example did well. However, two Amazon engineers lately exchanged out Sonnet 4.6 for Strands Decider, a decision example we launched a duration ago, that performed fair as fine at 98% small cost. This alter volition preserve millions of dollars of token expenses in a year.
Carr and Mehul, who activity on the inner ticketing system, stroll through how this happened below.
Replacing an LLM’s job alongside a decision model
We had Sonnet 4.6 labeling tickets at enormous scale. The tickets could be client escalations, capacity requests, automated alarms, safety findings, or item alternatively throughout the business that needed to fit within 20 category types. The delegate ran alongside a lengthy scheme immediate on all ticket, generating JSON and reasoning against a fixed set of labels.
A decision example can’t create text. It’s archetypal stated several province and a inquiry alongside a set of options, afterward the decision example returns a probability for all choice affirmative a confidence score. We saw this method fit absolutely alongside ticket labeling: the province is the ticket, the options are the label catalog.
We started alongside the first Strands Decider 2B. On a test set of tickets, the E4B Gemma 4 build got plainly improved results, so we switched to it. Afterwards, we tuned how assured the example needed to be before applying a label. The results showed E4B beating manufacturing Sonnet on a distinct set of our tickets.
We ran E4B on SageMaker ml.g5.xlarge instances (one A10G, 24 GB each). The example used concerning 19 GiB, and one petition took concerning 640 ms on one instance. The endpoint autoscales alongside traffic, but its network was isolated, preventing outbound admission to the container. SageMaker copies the example weights in from S3 before start-up.
Customers can do this too! Apart from our ticketing APIs, everything in the way is a norm AWS service. Ticket events district in an SQS queue that afterward triggers a Lambda function. It says the ticket data and calls the model, a SageMaker AI Endpoint alongside Strands Decider. AppConfig holds the switch (described below), and CloudWatch carries the logs, metrics and alarms. The stack is defined in CDK with deployment stages for Beta, Gamma, and Prod.
Running Sonnet 4.6 and Strands Decider on the identical tickets
We didn’t desire to flip a toggle and hope. The labeling Lambda says a three-way environment from AWS AppConfig on all message:
- BEDROCK: Sonnet labels all ticket, but Strands Decider isn’t called.
- DUAL: Both the Sonnet and Strands Decider models are called in parallel. The results from the Sonnet example are applied to the ticket and compared alongside Strands Decider results.
- DECIDER: Strands Decider labels all ticket and Bedrock isn’t called.
What we measured
| Claude Sonnet 4.6 | Strands Decider E4B | % Improvement | |
|---|---|---|---|
| Average period per call | 4,215 ms | 897 ms | 78% |
| Input tokens per ticket | 6,554 | 2,009 | |
| Output tokens per ticket | 170 | 1 | |
| Total tokens, per ticket | 6,724 | 2,010 | 70% |
The token autumn had two sources. Strands Decider produced one output, the choice, alternatively of a JSON answer. Its immediate was additionally much shorter than Sonnet 4.6’s. Production’s tag descriptions solitary are longer than Strands Decider’s 4,096-token window, so we rewrote all of the 20 labels as a one- or two-sentence criterion and sent them as the options of a sole choice question, affirmative a “none” option. The ticket data is the identical as what Sonnet gets, apart from that on lengthy tickets Strands Decider drops the oldest comments to fit.
Below is a chart of our Lambda durations on the day we turned off the Sonnet usage, showing the average Lambda implementation duration dropping 74%.
What it costs
At published us-east-1 on-demand prices:
| Claude Sonnet 4.6 (Bedrock, earth endpoint) | Strands Decider E4B (SageMaker, ml.g5.xlarge) | |
|---|---|---|
| Price | $3.00 per 1M input, $15.00 per 1M output tokens | $1.408 per instance-hour |
| Per ticket | $0.0222 | about $0.0005 |
| Per myriad tickets | about $22,200 | about $500 |
Sonnet’s disbursal per ticket comes direct from the token counts above. Strands Decider’s depends on how busy all case is. At concerning 640 ms per request, one case can grip concerning 5,600 tickets an hour, which would be $250 per myriad tickets at complete use. We keep headroom for bursts, and in custom it comes to concerning $500 per million. That is concerning 44 times cheaper than Sonnet, a 98% reduction.
These are published catalog prices. They depart out the Lambda, which additionally runs for small period per ticket now.
One item that amazed us was that JSON key command changed the answer. Sending the identical satisfied with keys in a distinct command moved the output probabilities by up to 0.19. We now dispatch all petition with sorted keys, so if you assist a decision example rearward an API, inspect this before you difference runs.
Our lessons
This worked since the project is a closed choice from a known catalog, the quantity is high, and the latency of the old way mattered. If your labels need clarification text, if the catalog changes weekly, or if you can’t fit the helpful environment into a few thousand tokens, a decision example likely can’t replace the LLM’s job.
The part we’d copy despite of example is the three-way switch. Running the two on genuine traffic, afterward judging all disagreement, told us additional than any offline eval would have. It additionally meant going rear was a 45-second config change.
The earth of agents gets a lot additional breathtaking (and small costly) alongside decision models. If you’re running a classification agent, try the Strands Decider repo and its serving notes.
