Preview
What it is.
You give it your baseline conversion rate and your traffic, and it tells you whether the test you want is possible before designing it: a hypothesis with a prediction, one variable, and the test type your volume supports.
The sample size table answers the first question. At a 1% baseline a 10% lift needs about 150k visitors per variant and a 50% lift about 6k; at a 10% baseline those fall to about 12k and 550.
What you get.
- A sample size table by baseline rate and target lift
- A hypothesis structure, so a test predicts an outcome instead of watching one
- Test type guidance across A/B, A/B/n, multivariate and split URL, with traffic needs per type
- A one-variable-per-test rule, with the reason
- A read on whether your traffic supports testing at all
How to set it up.
Install with npx skills add coreyhaines31/marketingskills --skill ab-testing -a claude-code.
Get your baseline conversion rate and your weekly traffic before anything else, because together they decide whether a valid test is possible at all.
Look your numbers up in the sample size table. At a 1% baseline chasing a 10% lift you need roughly 150k per variant, which most pages do not have.
Write the hypothesis as a specific prediction with reasoning behind it. Let us see what happens is not a hypothesis.
Change one variable. Multivariate needs very high traffic, which the table makes plain.
Pick the test type from the traffic column, not from what sounds thorough.
Pricing Plans
Free. MIT license.
Checked against the repo in July 2026.
Use cases
A go or no-go call
Someone proposes a headline test. Hand over the baseline conversion rate and your weekly traffic, and the sample size table says whether the lift you're chasing is reachable on your volume.
Post-mortem on flat results
A year of testing produced no winners. Run the numbers through the table and you'll usually find the tests were underpowered and called early, with the traffic each one needed shown next to what it got.
Choosing the test type
The team wants multivariate because it sounds thorough. The traffic requirement per test type turns the choice into math, and multivariate usually loses it.
When testing is off
The math says your page can't support a valid test. The stated fallback comes back with the verdict: make a larger change and measure it as a before-and-after, without pretending significance.
Best for
Deciding whether to test at all
The sample size table answers this in a minute, and for most B2B pages the honest answer is no.
A testing program with no wins
Usually underpowered tests called early. The table shows how much traffic the lift you wanted would have needed.
Choosing between A/B and multivariate
The traffic column per test type turns that choice into math instead of preference.