Best AI Model for A/B Testing and CRO (Updated for GPT-6.1 Sol)

The short answer: As of September 30, 2026, use Claude Opus 5.5 for the analysis side of A/B testing: reading funnel data, comparing explanations, and writing the test brief. Test GPT-6.1 Sol against it: OpenAI's new model costs half as much per token and OpenAI reports it beats Opus 5.5 on document and business-workflow benchmarks. Use Gemini 3.8 Flash when you need to audit many ads and landing pages cheaply, including video. Use Opus 5.5, GPT-6.1 Sol or Grok 4.6 to build the variant in code. GPT-6 Astra is now hard to justify for CRO work at five times Sol's price. The model matters less than the test design: a clear primary metric, enough conversions, and a human approval step.

We update this page when a new model is released. This version adds GPT-6.1 Sol and moves our Opus 5.5 notes into the job sections.

TL;DR

• Pick a model for the job, not for the leaderboard. CRO has three distinct jobs: investigate, audit, and build.
• GPT-6.1 Sol is OpenAI's new value pick: near-Astra results on OpenAI's own tests at one fifth of Astra's price.
• Every benchmark cited below is reported by the model's own vendor. Run a bake-off on your own funnel before switching.
• A model can make a wrong test faster. Score every experiment on revenue per eligible visitor, not clicks or signups alone.

The five models compared

Prices are standard API rates taken from each vendor's documentation on the dates listed under Sources. Subscription plans, caching, batch discounts and long prompts change the real cost. Compare the cost of an accepted brief or a merged variant, including review time, rather than token prices alone.

What's new: GPT-6.1 Sol for A/B testing

OpenAI released GPT-6.1 Sol on September 29, 2026, one week after GPT-6 Sol. It is available in the API as gpt-6.1-sol and in ChatGPT Work and Codex on Plus, Pro, Business, Enterprise and Edu plans. It is not yet in regular ChatGPT chat. Four changes matter for CRO work.

It handles documents with tables and charts well. OpenAI reports that on GDP.pdf, a benchmark of professional questions answered from complex PDFs, Sol scores higher than Claude Opus 5.5 at less than half the cost per task, and approaches GPT-6 Astra at about one fifth of Astra's cost. Funnel exports, ad platform reports and customer research decks look a lot like that test. It is still OpenAI's benchmark, run by OpenAI.

It completes multi-step business workflows more reliably. On AutomationBench, which runs end-to-end tasks across 47 tools in sales, marketing and operations, OpenAI reports Sol scores 2.2 points above Opus 5.5 at medium effort for about a third of the cost. That is the closest public proxy we have for "read the analytics, draft the brief, open the ticket."

It makes fewer factual errors than the model it replaces. On deliberately difficult prompts, OpenAI says the share of answers with a factual error fell from 11.4% to 7.7% at low effort compared with GPT-6 Sol. That is a comparison with Sol, not with Claude. For a test brief, an invented baseline conversion rate is worse than a missing one, so keep asking the model to cite the source row for every number.

It costs one fifth of Astra. Input and output tokens are $2 and $10 per million, against $10 and $50 for GPT-6 Astra. Cached input is $0.10 per million, and batch processing is half price. If you rerun the same analysis prompt over the same funnel export every week, the cache discount is the bigger saving.

Practical notes from the documentation:

• The default effort level is medium. Supported levels run from low to max; none and minimal are not supported. For a funnel investigation, set effort to high and compare against a few past analyses before you rely on the default.
• Tool calling needs the Responses API. Chat Completions works without tools. If your agent reads analytics through MCP or function calls, use Responses.
• Long prompts cost more. Requests with more than 272,000 input tokens are billed at twice the input rate and 1.5 times the output rate for the whole request. Send a summarized funnel table, not a raw event dump.
• Image input, no video. Sol reads screenshots and page captures but not screen recordings. Keep Gemini 3.8 Flash for video audits.

OpenAI also announced a faster tier, GPT-6.1 Sol Ultrafast, "in the coming days." We will add it once its pricing is published.

Job 1: Investigate the funnel and write the test brief

Use: Claude Opus 5.5 first; GPT-6.1 Sol as the lower-cost comparison.

We keep Opus 5.5 as the default here because of how it behaves on open-ended analysis. In Anthropic's launch post, the analytics company Hex describes Opus 5.5 finding a second problem in a data task where Opus 5 stopped at the first answer. In an internal Anthropic test of report writing from hard-to-find sources, Opus 5.5 avoided invented figures in 16 of 18 attempts. Both are vendor-published results. GPT-6.1 Sol's benchmark wins over Opus 5.5 are also vendor-published, which is why the right move is a side-by-side test on your own past analyses rather than a switch.

Give the model one commercial question, not "audit our website." For example: why do paid mobile visitors reach pricing but rarely start a trial?

Supply a small evidence packet:

• Funnel counts by step, device and campaign, with the same date range and event definitions.
• Current desktop and mobile page captures, plus the ad that sent the traffic.
• Customer objections from support or sales, with personal details removed.
• Trial and purchase outcomes, including how long each group has had to convert.
• Recent changes to the page, campaigns, offer or tracking.

Then make the model compare explanations before it proposes a change. A drop-off tells you where to look, not why people left. Paid mobile traffic with weak trial starts could mean the ad attracts the wrong buyer, the page doesn't explain setup, or the mobile button is hard to use. Each explanation needs different evidence.

Investigate [page and funnel] for [audience and campaign].
Use the attached evidence and connected read-only analytics.
Keep date ranges, timezone, visitor definitions and cohort maturity explicit.

First list missing or inconsistent data.
Then compare up to three explanations for the drop-off in a table:
observation, possible explanation, evidence against it, next check.
Cite the source row, customer note or page observation for each.
Do not invent customer motives, conversion lift, lost revenue
or statistical confidence.

Recommend one experiment. Return: audience, control, proposed change,
why it should work, primary metric, guardrail metrics, eligibility rules,
and the evidence that would prove it wrong.
If the evidence is not enough, name the next measurement to collect.
Prepare the brief for review. Do not launch anything.

If the model can't name evidence that would weaken its own recommendation, the recommendation is probably too vague.

Opus 5.5 setting notes: its default effort is medium, so set it to high for this job. Thinking can't be turned off; if you previously disabled it for speed, lower the effort level instead.

Job 2: Audit the ad-to-page experience

Use: Gemini 3.8 Flash for volume; Claude Opus 5.5 or GPT-6.1 Sol for the pages that matter most.

A page can look polished and still answer the wrong question. If an ad promises an easy migration and the landing page opens with a general product overview, the visitor has to work out whether the promise still applies. That is a candidate mismatch, not proof that a new headline will lift sales.

Gemini 3.8 Flash accepts text, image, video, audio and PDF input, so it can review a screen recording of the journey from ad click to form. At its introductory price it is the cheapest way to run the same audit across dozens of campaigns. Opus 5.5 and GPT-6.1 Sol both take images but not video. Anthropic reports Opus 5.5 reads values off dense charts much more precisely than Opus 5, which makes either model a good second pass on your highest-spend pages.

Require evidence a reviewer can find:

Review the supplied ad, page screenshots, journey recording
and campaign funnel table for [audience].

Map the ad's promise to the page's answer and the next action.
Return up to five findings, each with: screenshot label or timestamp,
the mismatch, the affected segment, the possible consequence,
and the evidence needed to confirm it.

Say "not visible in the supplied material" when evidence is missing.
Do not infer emotions, attention or cause from images.
Pick one finding with enough support for an A/B test and propose
the smallest page change that tests it.

Keep campaigns in separate rows. Averaging several audiences into one audit hides the mismatch you are looking for.

Job 3: Build the variant and check the tracking

Use: Claude Opus 5.5, GPT-6.1 Sol or Grok 4.6 inside a coding agent.

Anthropic reports that Opus 5.5 leads its agentic coding benchmarks and needs fewer steps per task than Opus 5. OpenAI reports GPT-6.1 Sol matches GPT-6 Astra on DeepSWE, a test of long software tasks in real codebases, at about one fifth of the cost, and it is available directly in Codex. Grok 4.6 is cheaper per output token and xAI positions it for sustained coding and self-testing. All three work when the brief is tight. These are vendor claims; none shows that a model builds pages that convert better.

Write the experiment contract before the build prompt: eligible audience, control, the one change being tested, primary metric, guardrails and stopping rule. Include the approved product facts. If the agent invents a plan feature or changes billing copy, the page no longer tests your hypothesis.

Implement the approved experiment brief for [page] in a preview.
Read the project's instructions and reuse existing components.

Change only what the brief describes. Keep prices, plan features,
checkout links, consent behavior, analytics IDs and experiment
assignment unchanged.

Check desktop and mobile layouts, keyboard navigation, the main button,
form errors and loading states. Confirm that exposure and conversion
events fire once each through the full journey.
Say which checks you ran and which you could not run.

Return the preview link, changed files, check results, open issues
and rollback steps. Do not deploy.

A page that renders correctly can still break the test. Before launch, confirm visitors get a stable assignment, exposure is recorded when they actually see the change, and the conversion survives any handoff to another domain, such as a checkout.

Score the test on revenue, not the model's confidence

Whichever model you use, decide the primary metric before launch. A hypothetical example shows why:

The variant wins on signups and loses on revenue. These invented numbers are too small to call a winner either way, but a team measuring only signups would never see the question.

Check feasibility before you build. The conversions you need depend on the smallest lift worth detecting, not on visitor counts alone. Use the sample size calculator and the minimum detectable effect guide. If the test can't reach enough conversions in a useful time, more AI-generated variants won't fix it.

How to choose for your team

1. Start with the job that's blocking you. If ideas are cheap but briefs are weak, test Opus 5.5 against GPT-6.1 Sol on investigation. If audits are the bottleneck, try Gemini 3.8 Flash. If briefs wait on developers, try a coding agent.
2. Run a small bake-off on your own work. Give two models the same evidence packet for three past analyses where you already know the answer. Count unsupported claims and the edits needed before the brief was usable.
3. Measure the cost of an accepted output. Include review time. A cheap first pass that needs heavy correction costs more than it looks.
4. Re-check when a new model ships. Model rankings change weekly this autumn: OpenAI shipped two Sol versions seven days apart. Your evaluation set is the asset that lasts.

Connect the model to your analytics

Humblytics exposes analytics, funnels, click data and experiment actions through its MCP server, and reports test results in revenue when Stripe is connected. In Claude Code or Codex, an agent can read your funnel, prepare a test for your approval and read the result. See how to use Claude Code with Humblytics for setup. In Codex, select GPT-6.1 Sol; in Claude Code, select Opus 5.5.

If you use OpenAI's new always-on agents, see what OpenAI dots change for A/B testing. For platforms rather than models, see our ranking of the best AI A/B testing tools.

Connect your agent to Humblytics, start with one funnel and one question, and review the brief before anything goes live.

Sources and freshness

• OpenAI, Introducing GPT-6.1 Sol, published September 29, 2026; retrieved September 30, 2026. Supports release date, availability, pricing, the GDP.pdf, AutomationBench, DeepSWE and factuality results, and the Ultrafast announcement.
• OpenAI, GPT-6.1 Sol model documentation, retrieved September 30, 2026. Supports input types, reasoning effort levels, the Responses API note, cache, batch and long-prompt pricing.
• Anthropic, Introducing Claude Opus 5.5, published September 22, 2026; retrieved September 28, 2026. Supports release date, pricing, the 40% cost claim, the report-writing test, benchmark claims and the Hex quote.
• Anthropic, What's new in Claude Opus 5.5, retrieved September 28, 2026. Supports default effort, always-on thinking and chart and screenshot reading.
• OpenAI, GPT-6 Astra model documentation, retrieved September 6, 2026, and the GPT-6.1 Sol launch post above for the September 3 release date and current pricing.
• Google, Gemini 3.8 Flash announcement, published September 2, 2026, and developer guide, retrieved September 6, 2026. Support release date, input types and introductory pricing ($1.50 / $7.50 from January 1, 2027).
• xAI, Introducing Grok 4.6, published August 12, 2026, and API pricing, retrieved September 7, 2026. Support release date, positioning and the short-context and long-context rates.
• Humblytics, MCP and skills, retrieved September 30, 2026. Supports the analytics, test-approval and revenue-reporting workflow.

All benchmark and customer claims come from the model vendors. The prompts and examples are proposed workflows, not measured customer results. Prices and availability change; check each vendor's page before budgeting.