A/B Testing When You Do Not Have Much Traffic

Split testing is presented as the responsible way to make website decisions. For most small and mid-sized sites it is not available: the traffic required to detect a realistic improvement is far higher than people assume, and running an underpowered test produces confident, random answers.

This guide covers how to know whether you can test, and what to do when you cannot.

The arithmetic, briefly

To detect a change you need enough conversions in each variant, not enough visitors. The smaller the effect you want to detect, the more you need, and the relationship is steep. Roughly halving the effect size quadruples the sample required.

A store with a few hundred orders a month can realistically detect only very large changes - the kind you would notice anyway. Detecting a modest lift would take many months, over which season, campaigns, and product mix all change, which invalidates the comparison.

Before running any test, calculate the required sample from your actual conversion rate and the smallest effect worth acting on. If the answer is longer than about four weeks, do not run the test.

What to do instead, in order

1. Fix the outright defects

Before optimising anything, remove the things that are simply broken. These do not need a test:

• A pay button covered by a sticky element on mobile
• Delivery costs revealed only at the final step
• Forms that reject valid phone numbers
• Images that push the price below the fold
• Pages that shift while loading

Nobody needs an experiment to justify fixing a broken checkout. Our product page checklist and mobile checkout checklist list the usual suspects.

2. Watch real people

Five session recordings of real visitors tell you more than an underpowered test ever will. Watch for hesitation, repeated scrolling, rage clicks, and abandonment points.

Add:

• Site search queries that returned nothing
• Support questions that repeat
• The questions your sales team answers on every call

These are direct evidence of confusion, and confusion is the thing you are trying to remove.

3. Ship one coherent change at a time

Sequential testing - change one thing, watch a stable metric, keep or revert - is weaker than a split test but far better than nothing, provided you are disciplined:

• Change one thing at a time
• Establish a baseline over a comparable period first
• Avoid periods with sales, festivals, or campaign spikes
• Compare like with like: same traffic sources, same day-of-week pattern
• Write down the prediction before you ship

If you cannot state what you expect to happen and by roughly how much, you are not testing; you are redecorating.

4. Use bigger swings

If you can only detect large effects, only attempt large changes. A button colour test is exactly the kind of small-effect experiment low traffic cannot resolve. A restructured product page, a new pricing presentation, or removing a required step are big enough to show up.

5. Test on the aggregate, not the page

If a single page has too little traffic, test a change applied across all product pages, or all category pages. You gain sample by pooling similar surfaces.

When a test is worth running

Run a proper split test when all of these hold:

• The change is expensive or risky to get wrong
• You can reach the required sample within about four weeks
• The metric is a real business outcome, not a proxy like clicks
• You can run both variants simultaneously, so seasonality affects both equally
• You have agreed in advance what result would make you revert

Otherwise, ship the better-reasoned version and monitor.

Guardrails to watch either way

Whatever method you use, watch the metrics that reveal a "win" that is actually a loss:

• Refunds and returns
• Support contacts
• Page performance
• Order quality and margin, not just order count
• Unsubscribe rate, if the change involves capture

A change that lifts conversion by pushing people into the wrong purchase shows up here.

Common mistakes to avoid

• Stopping a test when it first looks significant. Peeking inflates false positives dramatically.
• Running two tests on the same journey. You will not be able to attribute the result.
• Testing during a sale. The traffic mix is not representative.
• Declaring a winner from a handful of conversions. That is noise with a number attached.
• Never writing down the hypothesis. Post-hoc explanations always sound convincing.

Frequently asked questions

How much traffic do I need for an A/B test?

It depends on your conversion rate and the smallest effect worth detecting, but most small sites need far more than they have. Calculate the required sample first; if it exceeds about four weeks of traffic, choose a different method.

Can I run an A/B test with 100 visitors a day?

Only for very large effects. At that volume, a realistic test of a modest improvement would run for months, during which seasonality and product mix would invalidate the comparison.

Is sequential testing valid?

It is weaker than a split test because time-based factors are not controlled. It is still useful if you change one thing at a time, compare against a stable baseline, avoid campaign periods, and treat the result as directional rather than proof.

What should I optimise first with low traffic?

Defects and clarity: broken mobile interactions, hidden costs, confusing forms, and unanswered objections. These have large effects and need no statistical justification.

Next steps

Calculate the sample your current conversion rate would require for a realistic lift. If it is out of reach, spend that effort on watching five real sessions and fixing what makes them hesitate.

Start free and remove the friction you can already see.