Reading Data and A/B Testing

Data Without Interpretation Is Just Numbers

A dashboard full of charts means nothing until a PM asks the right questions of it: Is this change real, or noise? Is it caused by what we shipped, or something else entirely? This lesson covers how to read data critically and how to run A/B tests — the most reliable way product teams answer "did this actually work?"

---

Correlation Is Not Causation

Imagine Flutterwave notices that transaction volume rose 20% the same week a new dashboard redesign shipped. It is tempting to credit the redesign. But that same week might also have been salary week, a public holiday shopping period, or a marketing campaign running independently. Without a controlled comparison, a PM cannot separate the effect of the redesign from everything else happening at the same time.

This is exactly the problem A/B testing solves.

---

What Is an A/B Test?

An A/B test (also called a split test or randomized controlled experiment) randomly divides users into two or more groups:

• Control (A): sees the current experience
• Variant (B): sees the new experience

Because users are assigned randomly, the two groups should be statistically similar in every way except the one thing being tested. Any meaningful difference in outcomes can then be attributed to the change itself, not to external factors like salary week.

A Simple Example

Konga wants to know if adding a "Only 3 left in stock" urgency label increases purchase conversion.

• Group A (50% of visitors): sees the product page as normal
• Group B (50% of visitors): sees the same page with the urgency label

After two weeks: Group A converts at 4.1%, Group B converts at 4.6%. Is that difference real, or could it have happened by chance?

---

Statistical Significance, in Plain English

Statistical significance tells you how confident you can be that an observed difference is real and not due to random chance.

Key ingredients:

The most common mistake new PMs make: peeking at results after two days and declaring a "winner" before enough users have gone through the test. Small samples produce noisy, unreliable percentages that often reverse a week later. Always calculate the required sample size before launching a test, and let it run to completion.

---

Designing a Good Experiment

1. Write a clear hypothesis. "We believe adding urgency labels will increase conversion by at least 0.3 percentage points, because scarcity signals reduce hesitation at checkout."
2. Pick one primary metric. Testing ten metrics at once increases the odds one looks significant purely by chance (a problem called p-hacking).
3. Randomize properly. Users should be assigned to a group in a way that avoids bias, and each user should consistently see the same variant throughout the test.
4. Calculate sample size up front. Tools like Optimizely, Statsig, or even a simple online A/B test calculator can tell you how many users you need for a given baseline rate and expected lift.
5. Run for full business cycles. A test that runs Monday to Wednesday misses weekend behavior. Most teams run tests for at least one to two full weeks.
6. Watch for interaction effects. If Access Bank runs a pricing test and an onboarding test on the same users at the same time, the two experiments might interfere with each other's results.

---

Beyond Simple A/B Tests

• Multivariate tests test several changes at once (e.g. headline + button color + image) to see which combination performs best, but need much larger sample sizes.
• Phased or gradual rollouts release a new feature to 5%, then 25%, then 100% of users, catching major problems early without a full controlled experiment.
• Holdout groups keep a small percentage of users on the old experience permanently, so a team can measure the long-run cumulative impact of many small changes over months, not just days.

---

The PM's Job in All of This

A PM does not need to personally compute p-values by hand, but must understand enough statistics to ask good questions of a data scientist or analyst: "Was this sample size big enough?" "Could this result be a fluke?" "What did we NOT measure that might explain it?" That healthy skepticism, paired with a bias toward testing rather than debating, is what separates evidence-driven product teams from ones running on opinions.