How Much Traffic Do You Need to A/B Test on Shopify?
The sample-size math behind Shopify A/B tests, what Shopify Rollouts can test natively, why most tests don't win, and what low-traffic stores should do instead.
Key Signal
~15,600 per variant
The posts in this archive are written to help Shopify teams identify what is weakening buying momentum, what is suppressing AOV, and what deserves action first.
Most Shopify stores don't have enough traffic to A/B test small changes, and many test results shared online come from tests too small to trust. You can still test, but you need to know your numbers before you start, so you test changes big enough to detect and stop calling winners after four days.
This guide walks through the math, what Shopify's native Rollouts feature can do, and what to do if your store is too small for classic split testing.
Quick answer
At a 2.5% conversion rate, detecting a 20% relative lift (2.5% to 3.0%) needs roughly 15,600 visitors per variant, or about 31,000 sessions for a two-variant test. Detecting a 10% lift needs about four times that.
These figures use the rule of thumb from Evan Miller's article, which corresponds to roughly 95% significance and 80% power: sample size per variant ≈ 16 × p(1 − p) ÷ δ², where p is the baseline conversion rate and δ is the absolute change you want to detect.
| Baseline conversion | Lift to detect (relative) | Visitors per variant | Orders per variant (approx.) |
|---|---|---|---|
| 1.5% | 10% | ~105,000 | ~1,580 |
| 1.5% | 20% | ~26,300 | ~390 |
| 1.5% | 30% | ~11,700 | ~175 |
| 2.5% | 10% | ~62,400 | ~1,560 |
| 2.5% | 20% | ~15,600 | ~390 |
| 2.5% | 30% | ~6,900 | ~175 |
| 4.0% | 10% | ~38,400 | ~1,540 |
| 4.0% | 20% | ~9,600 | ~385 |
| 4.0% | 30% | ~4,300 | ~170 |
Halving the lift you want to detect roughly quadruples the traffic you need. And at any baseline, detecting a 10% lift takes around 1,500 orders per variant.
What minimum detectable effect means
Minimum detectable effect (MDE) is the smallest lift your test can reliably detect. You choose it before the test starts, and it drives the sample size.
Optimizely's documentation gives a worked example: with a 15% baseline, a 10% MDE, and 95% significance, you'd need about 8,000 visitors per variation. It also suggests limiting variations to changes that directly influence the primary conversion event. For a low-traffic store, the math points the same way: fewer variations and bolder changes.
For a Shopify store, that means a button color or a microcopy tweak is rarely worth a formal test. A different PDP layout, a new offer structure, or a change to cart type might move conversion enough to detect.
Why peeking breaks your test
Checking results every day and stopping when the dashboard turns green inflates false positives badly. Evan Miller's well-known analysis shows that checking for significance after every observation can push the false-positive rate to 26.1% against a nominal 5%.
Two ways to avoid this:
- Fix the sample size in advance and don't make a decision until you reach it.
- If you want to check results as the test runs, use a testing tool whose statistics are designed for that, and confirm it with the vendor.
You'll often hear that 100 conversions per variant is enough. The table above suggests otherwise: detecting a 20% lift at common Shopify conversion rates takes roughly 390 orders per variant.
Most tests don't win
Expect most of your tests to lose or come out flat. Optimizely's analysis of 173,000 experiments found that only 12% won on their primary metric. A healthy program in that data had a 35% to 40% conclusive rate, meaning a clear win or a clear loss, and the median company ran 34 tests a year.
That changes how you plan. If one test in eight wins, a store that can run four tests a year shouldn't expect many wins. Prioritize tests where the research gives you a strong reason to expect a large effect.
What Shopify Rollouts can test natively
Shopify now has a native tool for scheduling and testing changes, called Rollouts (under Markets in the admin). Shopify's help center describes three types:
| Rollout type | What it does |
|---|---|
| Launch | Publishes a change permanently at a scheduled time |
| Event | Applies a change for a set period, then reverts |
| Experiment | Splits traffic between a treatment and a control |
Rollouts can change the main theme, checkout and customer account pages, discounts, and catalogs. A June 2026 changelog update added A/B testing between two different themes or checkout configurations, plus mutually exclusive experiments.
Availability and limits, per Shopify's requirements page:
- Rollouts are available on the Basic plan or higher. Experiments are available "on the Grow plan or higher," in Shopify's wording.
- Vintage themes aren't supported, and Liquid template changes can't be rolled out.
- Experiments, and rollouts that reach less than 100% of visitors, apply to the online store only.
Shopify's documentation doesn't say how much traffic an experiment needs, so the sample-size math above still applies. Rollouts makes a test easier to run, but a small store still needs the same traffic to get a valid result.
Google Optimize, which many Shopify stores used for free testing, shut down on September 30, 2023. Stores that relied on it moved to paid testing apps, and more recently some have moved to Rollouts.
What to test first
Pick tests where the expected effect is large and the change sits close to the purchase decision:
- Offer structure. Bundles, subscription defaults, or free shipping thresholds (see how to set a free shipping threshold).
- Cart format. Drawer vs. page, which our cart drawer guide covers.
- Product page structure. Moving proof, delivery information, and variant clarity above the fold (see product page gaps).
- Landing pages for paid traffic. A dedicated page against the PDP for one campaign, covered in our paid social landing page guide.
Test primary metrics that reflect revenue, such as conversion rate or revenue per visitor. Add-to-cart rate is easy to move and easy to misread.
What to do if your store is too small to test
At 15,000 sessions a month and a 2.5% conversion rate, even a test looking for a 20% lift takes about two months to reach its sample size. Some better options:
- Fix what's clearly broken without testing. A shipping fee that appears only at checkout, a hidden checkout button on mobile, or a broken variant selector doesn't need a split test.
- Use qualitative research. Session recordings, customer surveys, post-purchase questions, and support tickets show where people get stuck.
- Make bigger, bolder changes. If you test, test changes large enough to show up at your traffic level.
- Use before-and-after comparisons carefully. Compare equal periods, avoid sale weeks, and read results by device and traffic source.
- Audit first. A structured audit of the storefront finds the friction worth fixing, so tests go to the questions that are still open.
Our Shopify conversion audit checklist is a good starting list.
Don't test during peak season
Holiday traffic looks attractive for testing because volume is high, but shopper intent, discount depth, and traffic mix all change. A variant that wins during Black Friday week may not win in February. Schedule tests to finish before your peak trading period, and use Rollouts' Event type for time-boxed promotional changes instead. Our BFCM 2026 guide covers what to change, and what to leave alone, in the final weeks.
FAQ: A/B testing on Shopify
How many visitors do I need to run an A/B test on Shopify?
It depends on your conversion rate and the lift you want to detect. At a 2.5% conversion rate, detecting a 20% relative lift needs about 15,600 visitors per variant at 95% significance and 80% power. Smaller lifts need far more traffic.
Does Shopify have built-in A/B testing?
Yes. Shopify Rollouts includes an Experiment type that splits traffic between a treatment and a control. Shopify says experiments are available on the Grow plan or higher, and they don't support vintage themes or Liquid template changes.
How long should I run an A/B test?
Run it until you reach the sample size you calculated in advance, and for at least two full weekly cycles so weekday and weekend behavior are both included.
How many conversions do I need per variant?
The "100 conversions" rule of thumb is too low for most tests. Detecting a 20% lift typically needs around 390 orders per variant, and a 10% lift needs about 1,500.
Is it OK to check A/B test results early?
You can look, but don't stop the test because it looks significant. Repeated peeking with a fixed-horizon test inflates false positives well above 5%. Use a fixed sample size or a sequential testing method.
What percentage of A/B tests win?
In Optimizely's analysis of 173,000 experiments, 12% won on their primary metric. Most tests are flat or lose, which is why prioritizing large expected effects matters.
Test less, test bigger
A small store that runs one well-chosen test a quarter will learn more than one that runs a dozen underpowered tests. Calculate the sample size before you build the variant. If the number is out of reach, fix the obvious friction first and save testing for the decisions you can't settle any other way. A HiveSense audit gives you a prioritized list of what to fix and what to test, and the sample report shows an example.
Sources: Evan Miller: How not to run an A/B test, Evan Miller: sample size calculator, Optimizely: minimum detectable effect, Optimizely: takeaways from 173,000 experiments, Shopify Help Center: Rollouts, Shopify Help Center: Rollouts requirements, Shopify Changelog: A/B test themes and checkout, Google Optimize sunset.
Next Step
Turn these patterns into a real storefront audit.
If you want a faster read on conversion blockers, AOV gaps, checkout friction, and the issues most likely to cost revenue, run a HiveSense audit on your store.