Friday November 20, 2026
Larassatti D.

The best-scoring AI example isn’t continually the one you should ship. When we benchmarked the AI rearward our AI website builder for WordPress, a smaller, faster example outperformed the top scorer in leaf design.
We ended up alongside norm improving throughout all endpoint we upgraded, yet expenses decreased. Getting to this item started alongside a problem. In first September, we replaced the builder’s template-based motor alongside a pipeline that generates all website from scratch.
In A/B testing, customers preferred the new output, but it was considerably additional costly to run. The dozens of LLM tasks rearward all location additionally motionless relied on a few models chosen whenever the merchandise was archetypal built.
But there was no dependable way to inform whether switching models would really help.
“We could alter a example and eyeball a few results, but there was no proven way to cognize it would be okay,” says Eylül Güleryüz, AI Engineer on our WordPress team.
She set out to alter that by construction a way to measure how fine all AI project performs, turning example choice into an informed decision.
Here’s how she did it.
How all AI example earned its spot
Eylül archetypal created a stream that lets all example go through the identical test.
One benchmarking formula for all endpoint
She began by shortlisting about 10 to 12 applicant models, ranging from low-cost options to top-tier frontier alternatives.
She afterward put all one through the identical six-step benchmarking process, scoring all endpoint and phase on its own:
- Collect. Build a test set from genuine website requests, filtered for WordPress and alongside duplicates removed.
- Freeze. Select 100 cases per endpoint using a fixed random seed, afterward pin all dataset alongside a hash so that all example faces the exact identical test and results remain comparable.
- Run. Test all example independently, without fallbacks, using manufacturing timeouts. Each example runs through the identical 100 cases three times, producing 300 outputs per endpoint. Testing models individually matters because, inner a fallback chain, a backup can quietly conceal the chief model’s failures.
- Score. Combine deterministic checks, specified as whether the output uses the accurate format, language, and fields, alongside an LLM fairness that rates output norm on a 1–5 scale. Report all mark alongside a 95% confidence interval, so small differences aren’t mistaken for genuine gains.
- Publish. Send all run to an inner benchmarking dashboard, tagged alongside its dataset hash. This keeps results comparable throughout experiments and complete time.
- Ship. Roll out the victor as the endpoint’s chief model. Backup models, tested on the identical icy cases and, ideally, from another AI providers, stage in if the chief example fails.

Scoring all part on its own has one blind spot: it can’t display how the pieces activity together. For that, Eylül added a final inspect on complete websites, built end to end and visually reviewed, which we shield in the afterward section.
Eylül applied the formula throughout 18 endpoints and website-generation stages, evaluation concerning seven models on all one on average. She used genuine requests and kept evaluation new models as they became available.
The dashboard turns those results into an inner LLM leaderboard, ranking the tested models for all endpoint by quality, cost, and speed.
Other teams in the business can additionally admission the dashboard, so they can choose models for their own products according to these benchmarking results.
Measuring what “good” looks like
Not all AI tasks create the identical benevolent of output. For some, there’s a dependable expected answer that correctness can be checked against. For others, multiple distinct outputs can be likewise valid, and norm depends on judgment.
This difference resolute how Eylül evaluated them. She used two methods, depending on the endpoint. Some outputs needed lone one, during others required both.
First, deterministic checks measure outputs against predefined rules. They verify whether the outcome is valid, follows the expected format, and contains all required fields. When correctness can be defined in advance, these checks provision an goal measure.
Second, an LLM fairness evaluates outputs that necessitate qualitative judgment. The LLM fairness says the user’s petition and the generated output, afterward rates it from 1 to 5 against predefined criteria, specified as how faithfully the outcome follows the concise and how fine it fits the brand. There’s no sole accurate answer in these cases, so using the identical fairness to mark all example keeps the comparisons consistent.
Choosing the LLM judge
The fairness itself is an AI model, so Eylül had to measure it too. A fairness that’s too small power young female crucial norm differences, during a larger example could add unnecessary costs.
To choose the finest LLM judge, she started alongside the default fairness in the team’s codebase, afterward experimented alongside larger models to see whether they could create additional dependable evaluations.
The largest example turned out to be additional than most endpoints needed. After benchmarking the judges against all other, she switched to a newer, stronger default example at no additional cost.
The elimination was leaf design, anywhere judging norm takes item nearer to a human’s eye. In this aspect, the largest example was the strongest fairness available, but Eylül’s end-to-end pipeline test showed that it had limits.
She additionally watched for possible bias. A fairness from the identical example family as the example it’s scoring may favor that model’s style. When that happened on one endpoint, Eylül kept spot-checking the results against genuine outputs.
Scoring the output norm from distinct AI models
To see how these evaluation methods activity in practice, let’s appearance at three endpoint examples, from outputs alongside plainly defined answers to those requiring subjective judgment.
1. Deterministic inspect method: website-type detection
This endpoint says the user’s petition and identifies the category of website they desire to build.
For example, a petition to create an online shop should come back the corresponding website-type label. Since there’s a predefined accurate answer, the output can be evaluated objectively.
A top-tier example archetypal drafted the expected labels, which Eylül afterward reviewed manually. Deterministic checks compared the endpoint’s output against these approved labels to verify its correctness.
2. Qualitative judgement method: project catalogs
For businesses that obtain bookings, this endpoint generates a catalog of services customers can book.
Consider a massage studio. There’s no sole accurate catalog of services it should offer. One example power propose Swedish massage and profound tissue massage, during another power contain aromatherapy and hot rock treatments. Both combinations could be relevant, depending on the endeavor and its customers.
Because multiple answers can be likewise valid, predefined expected outputs wouldn’t be adequate to measure quality. Eylül hence relied on the LLM fairness for scoring during additionally manually reviewing the generated results.
3. A blend of the two methods: color palette generation
This endpoint generates six color palette options according to a brand description. For example, a concise for a cozy, vintage bakery should create six palettes, all containing a set of hex codes.
Some parts of the output can be checked objectively, during others necessitate interpretation.
Deterministic checks verified that the six palettes were distinct, covered a light-to-dark range, and included at smallest one dreary option. But whether those colors really suited the bakery’s brand was a distinct question.
That’s anywhere the LLM fairness came in. It evaluated how fine all palette matched the brand brief, assessing the subjective norm that code solitary couldn’t capture.

Picking the genuine winners
A higher benchmark mark didn’t automatically create a example win. Eylül used four criteria to decide whether a example was value switching to.
1. Is it plainly better?
A higher average mark wasn’t enough. Eylül set a deliberately traditional rule: a new example was considered improved lone if its 95% confidence duration didn’t overlap alongside the current model’s.
If the intervals overlapped, she treated the outcome as a tie, marking it as “within noise” fairly than claiming an improvement.
2. Is it accelerated and dependable enough?
The AI website builder operates in genuine time, so all example was tested under the identical period limits it would visage in production. Models that scored fine but were too dilatory or unsuccessful too frequently didn’t create the cut.
3. Is it value the cost?
Any norm betterment had to validate the additional expense. Some top-tier models performed fine but were too costly to use.
Self-hosted models faced the identical scrutiny. One appeared cheap at first, but its quoted cost didn’t document for the GPUs needed to run it. It additionally frequently timed out during longer generations, making it an impractical choice notwithstanding its apparently low cost.
4. Does it really appearance right?
Finally, Eylül didn’t depend on benchmark scores alone. She compared the LLM judge’s ratings against the genuine outputs and, for complete websites, reviewed screenshots against scheme principles she’d learned from a decorator on her team, specified as accordant spacing on an 8-point grid, readable row lengths, and adequate color contrast.
After all, a example can mark fine on document and motionless create a website that doesn’t appearance or awareness right.
What changed for us and for our customers
With the benchmarking procedure in place, Eylül began rolling out the winning models. She started alongside the smaller endpoints that create foundational decisions concerning all website, afterward moved on to the chief website-generation stages.
The results display how much our WordPress squad improved norm and speed during reducing costs, and what those improvements average for our customers.
The construction blocks: improved and cheaper at the identical time
Before a website can obtain shape, the AI website builder needs to create multiple foundational decisions: what benevolent of website to create, which colors and fonts to use, and what satisfied to write.
Eylül started by benchmarking the models rearward these tasks. Every endpoint she switched to enhance norm got better, and most additionally became cheaper to run.
Here’s how the changes compared:
| Endpoint | Quality before → following (1–5 mark range) |
| Product descriptions | 3.24 → 3.78 |
| Font pairing | 3.47 → 3.92 |
| Colors | 4.19 → 4.72 |
| Prompt enhancer | 3.44 → 3.93 |
| Post and leaf generation | 2.82 → 3.31 |
Post and leaf generation was the exception: it became additional costly to run, but the norm betterment was ample adequate that our WordPress squad accepted the trade-off was value it.
One another construction obstacle isn’t included in the array since it’s evaluated using accuracy fairly than a 1–5 norm score.
Website-type detection determines what benevolent of website the builder should create, so its output is either correct or wrong. Its accuracy risen from 95.3% to 98.7%. The new example takes approximately twice as lengthy to respond, but motionless stays comfortably inside its period budget.
Not all endpoint needed a new model, though. The benchmarks additionally helped Eylül acknowledge anywhere switching wouldn’t bring meaningful improvements.
Page suggestions stayed alongside the existing example since no challenger plainly outperformed it, and it was already among the cheapest and fastest options.
Image keywords additionally stayed as they were. Every tested example scored low, ranging from 2.28 to 2.41, indicating that the prompt, not the model, was the problem.
Site construction was a distinct case. Its norm improved lone inside noise, so Eylül switched models according to disbursal and speed instead, without claiming a norm win.
Rebuilding website generation, phase by stage
The construction blocks grip the groundwork for website creation. The chief AI motor afterward takes those decisions and turns them into a complete website through four stages:
- Discovery plant out what the endeavor does, who it serves, and what the website needs to achieve.
- Planning maps out the website, deciding which pages to contain and what goes on all one.
- Page design lays out all web page, division by section, prepared for Elementor or Gutenberg.
- Catalog generation fills the website alongside starter satisfied whenever needed. It comes in three types: merchandise catalogs for online stores, project catalogs for bookable services, and blog catalogs alongside first posts.
Eylül benchmarked all phase independently, including all three catalog types, so all project could have its own best-performing model.
Here’s what changed:
| Website innovation stage | Quality before → following (1–5 mark range) |
| Discovery | 3.78 → 4.60 |
| Planning | 3.71 → 3.77 |
| Page design | 2.99 → 3.25 |
| Product catalog | 3.12 → 3.27 |
| Appointment catalog | 3.28 → 3.42 |
| Blog catalog | 3.13 → 3.58 |
Quality improved at all stage, but the gains came in distinct forms.
- Discovery and catalog generation delivered several of the biggest disbursal funds during improving quality.
- Page design became much cheaper and faster. Its p95 reply period (the period inside which 95% of requests finish) dropped from 37.9 seconds to 12.4 seconds.
- Planning saw lone marginal improvements in norm and cost, but its p95 reply period cut from approximately 98 seconds to 71 seconds.
Together, these changes made website generation considerably cheaper during improving norm throughout all stage.
The new setup is additionally additional resilient. Each phase has backup models from another providers, tested against the identical icy cases. This reduces reliance on a sole provider and makes website generation small exposed to outages.
What customers really see
Here’s how those improvements display up in the websites customers get:
1. Better understanding of website requests
The builder now identifies the correct category of website additional reliably, giving all consequent generation phase a improved starting point.
For example, a petition for a endeavor website alongside services, a gallery, and online booking is now correctly identified as a multi-page endeavor site.

2. More assortment in color options
The builder now generates options that extend a wider range, from ray to dark. Around 99% of palette sets additionally contain a dreary option, giving customers additional distinct directions to choose from.

3. Better handling of multilingual requests
Generated satisfied now follows the tongue of the user’s petition during sticking to the data provided.
For example, whenever asked in Spanish to compose a merchandise clarification for a h2o bottle, the builder now responds in Spanish without inventing merchandise features.

4. Font suggestions that fit the brand
Font recommendations now improved indicate the business’s personality.
A bakery gets a warm, traditional type, during a application tool gets a clean, contemporary one, fairly than the identical generic typefaces for all business.

The issue that slipped former the benchmarks
Benchmarking idiosyncratic stages of website generation helped Eylül acknowledge the best-performing example for all task. But whenever she tested operating the complete pipeline end to end, she discovered that the website output actually looked worse.
The leaf scheme victor had scored highest in its secluded benchmark, but in Eylül’s judgment, its pages felt raw and the layout looked unbalanced. Meanwhile, a example classified sixth produced designs much nearer to what was shipped before.
This is since the LLM fairness could measure whether a leaf included the correct sections and followed the brief, but it couldn’t inform whether the leaf really looked good.
The example that ended up shipping in the AI website builder was a smaller, faster one, released during this activity was underway. It scored fine on the benchmark and held up visually.

To verify the new setup held up, Eylül tested the two the old and new setups using the identical website requests throughout seven languages, alongside the two Elementor and Gutenberg, in a genuine hosting environment. She afterward compared screenshots of the generated pages flank by side. The results showed that:
- Pages looked additional eye-pleasing: The new setup produced additional proportional champion sections, photo cards, and clearer headlines.
- Costs went down: The new setup was cheaper to run throughout most of the tested websites.
- Generation speed remained comparable: The old setup took an average of 106 seconds, compared alongside 99 seconds for the new one.
- Website generation remained reliable: Both setups applied successfully, alongside no missing images or sections.
The live test revealed a limitation of secluded benchmarks: the example alongside the highest mark doesn’t necessarily create the finest outcome whenever it’s part of a larger system.
That’s why Eylül additionally reviews the genuine websites, not fair the scores. She acknowledges that this ocular judgement is motionless subjective.
“I can lone fairness what looks finest to me, so I leaned on the scheme principles our decorator taught me during we built the engine,” she says. “The next stage is turning additional of those rules into automatic checks.”
Turning example upgrades into drag requests
Testing the next example now takes a drag petition and a benchmark run, not a sprint. Eylül set up two feedback loops to create that possible:
- On all drag request, in minutes. When an engineer touches an endpoint, an automated eval runs the current example and the challenger flank by side. A comment on the drag petition reports quality, speed, cost, errors, and format checks, and all endpoint gets a pass/fail position that can obstacle a merge.
- The complete benchmark. For the final decision, the challenger runs on the identical 100 icy test cases as all before model, three times, and the results appear on the dashboard alongside all former runs.
In one genuine example, the drag petition inspect came rear in under a minute. The challenger scored higher on norm (3.87 → 4.03), responded faster, and disbursal noticeably small per request.
When a new example is released, validating it takes four steps:
- Swap the chief example in the endpoint’s path and open a drag request.
- Read the automated difference on the drag request.
- Run the complete benchmark on the icy dataset.
- Merge lone if it wins.
That’s how a newly released example reached our preparedness phase inside days. Adding a brand-new endpoint to this scheme needs fair a small spec document and a set of test scenarios.
The finest AI example depends on the job
Teams construction AI-powered products all visage the identical question: alongside new models arriving all few weeks, how do they cognize which one is correct for all task? Swapping a example and checking a few outputs lone goes so far.
Eylül built a repeatable evaluation procedure according to person requests, icy test sets, and accordant criteria for comparing models.
Along the way, the activity challenged a few average assumptions:
- Better results don’t continually necessitate additional costly models. Some of the biggest norm gains came alongside lesser costs. And in leaf design, a smaller, faster example produced better-looking websites than the top scorer.
- Different tasks need distinct models. Understanding a request, choosing colors, and laying out pages all reward distinct strengths, so no sole example is the finest default for everything.
- Measurement strikes guesswork, but it motionless needs a individual eye. With a fixed benchmarking procedure in place, example upgrades stopped being risky one-off projects and became a regular part of the betterment process. But scores lone grasp what they’re designed to measure. When norm comes downward to taste, akin how a website looks, a individual motionless needs to assessment the genuine output.
Our WordPress squad now has improved models and a scheme for evaluation the next ones, keeping our AI website builder elastic as new models emerge.
If you’re construction alongside AI, you can use the identical approach. Freeze a test set built from genuine requests, so all example faces the identical exam. Test all example on its own, without fallbacks hiding its failures. And anywhere an automated fairness can’t reliably inform fine from bad, appearance at the output yourself before you ship.
