Google Ads A/B test: how to set up and read experiments properly

An A/B test in Google Ads sounds simple: two variants run against each other and the better one wins. In practice, two things decide whether the result is worth anything: which tool you use for the test, and whether enough data comes in before you judge. Here are the three tools Google Ads offers, an honest calculation of how much data you need, and the three experiments I would start with.

1. Google Ads A/B test: three tools for three questions

Experiments live under Experiments in the Campaigns menu of Google Ads. That’s where you’ll find two of the three tools. The third is the classic route of two ads in one ad group.

Custom experiments: a campaign against its copy

According to Google, a custom experiment is a copy of an existing campaign with changes, running alongside the original. Part of the traffic and budget goes to the experiment.

• Setup: Campaigns → Experiments → in the “All Experiments” table, the plus button → Custom.
• Campaign types: Search, Display, Video and Hotel Ads campaigns. Custom experiments aren’t available for App and Shopping campaigns.
• Split: you set what percentage of the budget goes to the experiment. Google recommends 50/50 for a clean comparison. For Search campaigns you also choose between a cookie-based split, which Google recommends because each user always sees the same version, and a search-based split, which can reach data faster but may show one user both versions.
• Success metrics: up to two conversion goals, highlighted in the report.
• Wrap-up: if you’re happy, you apply the changes to the original campaign or create a new campaign from the experiment.

Use this tool for anything at campaign level: bid strategy, match types, a different landing page.

Ad variations: change text across many ads

With ad variations you test changes to responsive search ads, in one campaign or across all campaigns at once. Google offers three kinds: find and replace (for example “Book now” against “Call now”), updating text (adding, removing or pinning headlines and descriptions), and updating URLs. Here too a cookie decides whether a visitor sees the variation or the original. You’ll find them under Campaigns → Experiments → the Ad variations tab.

Use this tool when the same copy question affects many ads, such as a different offer or a different call to action.

Two ads in one ad group: simple, but skewed

The oldest route: two ads in the same ad group, see which does better. The problem is ad rotation. With the default Optimize setting, Google favours the ad that’s likely to perform better. And if you use Smart Bidding focused on conversions, Google’s help says it automatically uses “Optimize”. The two ads then don’t get the same chance, and part of what you measure is Google’s distribution rather than your copy. The alternative, Do not optimize, spreads impressions more evenly, but Google doesn’t recommend it for most advertisers.

What about pinning in responsive search ads?

Pinning isn’t a test, it’s a control. Google warns that pinned headlines reduce the number of possible combinations and that pinning isn’t recommended for most advertisers. If you do pin, then, following Google’s own advice, pin two or three different headlines or descriptions per position so the system still has a choice. I counted which copy mistakes happen most often in the RSA error report.

2. How much data an A/B test needs before you judge

The most common mistake in A/B tests isn’t the tool, it’s patience. After a week one variant is ahead and the test gets stopped. Whether the lead is real or chance only a calculation can tell you.

How the A/B test calculator works

The free A/B test calculator on this site uses Wilson intervals at 95% confidence. Such an interval is the range in which a variant’s true conversion rate plausibly lies. With little data it’s wide, with lots of data it’s narrow. The calculator has fixed rules:

• Below 100 impressions per variant it doesn’t calculate at all.
• Conversion rate only decides once both variants have at least 15 conversions from at least 100 clicks. Before that it compares click-through rate and points out that this answers a different question.
• There’s only a clear winner once the intervals no longer overlap and the lead is above 20%. Otherwise the verdict is “trend”.

A worked example

The numbers are made up; the maths is real. Two variants get 400 clicks each. A brings 20 conversions (5%), B brings 28 (7%). So B is 40% ahead.

• A’s Wilson interval runs from about 3.3% to 7.6%.
• B’s runs from about 4.9% to 9.9%.

The ranges overlap clearly. Despite a 40% lead, it’s only a trend. If the rates stay the same, the intervals in this example only separate at around 2,400 clicks per variant, roughly 120 against 168 conversions. For an account with 20 conversions a month, that takes a very long time. Hence the most important rule for small accounts: test big differences, not single words.

Duration

Google doesn’t specify a fixed duration for custom experiments; you set start and end yourself. I would always run a test over full weeks, so weekdays and weekends appear equally often, and not start it at a seasonal peak. The calculator itself points out that it can’t account for seasonality, budget changes or learning phases.

3. Three experiments worth running first

These are the three tests I would start with in a Search account. They aim at big differences, because small ones are hardly measurable on normal budgets.

1. The landing page

With a custom experiment you send half the clicks to a different page. The lever is big: at €2 per click, if the conversion rate rises from 5% to 8%, the cost per enquiry falls from €40 to €25 without any change to the bid. Test a genuinely different page, not just a new button colour.

2. The offer in the ad copy

With ad variations you test one core message across all campaigns: free first consultation against a fixed price, “Request a quote” against “Call now”. That collects data from the whole account instead of one small ad group. How strong simple, quick variants can be is the topic of my talk in Why ugly ads outperform.

3. The bid strategy

Before you switch from manual bids or “Maximize clicks” to a conversion strategy, test the switch in a custom experiment with a 50/50, cookie-based split. That way you see what the change brings, instead of interpreting a before-and-after comparison that also contains seasonality and competition. The prerequisite is conversion tracking you trust. How to check it before launch is in How to run Google Ads.

Reading the result

• Decide on conversions or revenue, not clicks. More clicks can mean fewer enquiries.
• Wait until both variants reach the minimum amounts, even if one is ahead early.
• Apply the winner using the apply function rather than uploading the ad again. The calculator gives the same advice: keep the original instead of recreating the winner, because recreating it splits the impressions.
• Write down what you tested and why. Otherwise, three months from now nobody will know why the campaign looks the way it does.

The bottom line

• Google Ads experiments live under “Experiments”. Custom experiments compare a campaign with its copy; ad variations test copy changes across many ads.
• Two ads in one ad group aren’t a clean A/B test once Google optimises rotation, and with conversion-focused Smart Bidding it does so automatically.
• Pinning is control, not a test. If you pin, use two or three options per position.
• A lead is only a result once the intervals separate. In the example, a 40% lead at 400 clicks per variant isn’t enough.
• Start with big differences: landing page, offer, bid strategy.

Have a test result and aren’t sure you can trust it? Whether a test is enough, what to test next or whether to switch is a question for the Clarity Session: from €390, a 60-minute video call with up to three links or documents reviewed first. If you’d rather have your whole account and measurement checked, the Google Ads audit from €1,900 covers that. Whether the session fits your question is something you can check first in a free intro call. Book a free intro call.

FAQ

Where do I find experiments in Google Ads?

In the Campaigns menu under Experiments. You create custom experiments with the plus button, and ad variations have their own tab.

How long should a Google Ads A/B test run?

Google doesn’t set a fixed duration. What matters is the amount of data: in the A/B test calculator, each variant needs at least 15 conversions from 100 clicks before conversion rate decides, and there’s only a winner once the intervals separate. Run tests over full weeks.

Can I test Shopping or Performance Max campaigns this way?

According to Google, custom experiments are only available for Search, Display, Video and Hotel Ads campaigns, not for App and Shopping campaigns. For other types Google offers separate experiment types that aren’t covered here.

Why does Google Ads give one ad far more impressions than the other?

Because rotation is set to “Optimize”. Google then favours the ad that’s likely to perform better, and with conversion-focused Smart Bidding that happens automatically. For a fair comparison, use ad variations or a custom experiment.