Statistical Thinking for Analysts

What is Statistical Thinking?

Statistical thinking is a mindset for interpreting data with appropriate uncertainty, understanding variation, and avoiding common reasoning errors. It is the difference between an analyst who reports numbers and one whose analysis actually drives better decisions.

Many business decisions fail not because of bad data, but because of flawed reasoning about data. Statistical thinking helps you ask the right questions, interpret results correctly, and communicate findings with the right level of confidence.

Understanding Variation

All data contains variation. Some variation is signal (meaningful patterns), some is noise (random fluctuation). The central challenge of data analysis is distinguishing one from the other.

Common and Special Cause Variation

Common cause variation is the natural, expected randomness in any process. If a website gets 1,000 visitors per day on average, some days will be 900 and some 1,100 -- just from chance. This does not require action.

Special cause variation is an unusual change that indicates something real happened. If traffic drops to 200, that warrants investigation.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Simulate daily website traffic
np.random.seed(42)
days = 90
traffic = np.random.normal(loc=1000, scale=100, size=days)

# Introduce a real event on day 70
traffic[70:] *= 0.6  # 40% drop

mean = traffic[:70].mean()
std = traffic[:70].std()

# Control chart
plt.figure(figsize=(12, 4))
plt.plot(traffic, color='steelblue', linewidth=1.5)
plt.axhline(mean, color='green', linestyle='--', label=f'Mean: {mean:.0f}')
plt.axhline(mean + 2*std, color='orange', linestyle=':', label='+2 SD')
plt.axhline(mean - 2*std, color='orange', linestyle=':',label='-2 SD')
plt.axhline(mean + 3*std, color='red', linestyle=':', label='+3 SD (control limit)')
plt.axhline(mean - 3*std, color='red', linestyle=':')
plt.legend(); plt.title('Website Traffic: Control Chart')
plt.show()

Points outside 3 standard deviations (control limits) are almost certainly special cause variation and warrant investigation.

The Base Rate Fallacy

One of the most common statistical errors in business. Ignoring how rare an event is before interpreting conditional probabilities.

Example: A data science model flags customers as "likely to churn" with 90% accuracy. Your company has a 2% monthly churn rate. If the model flags 1,000 customers, how many will actually churn?

# Base rate fallacy calculation
population = 100000
churn_rate = 0.02
actual_churners = population * churn_rate        # 2,000
actual_non_churners = population - actual_churners  # 98,000

sensitivity = 0.90   # correctly identifies 90% of churners
specificity = 0.85   # correctly identifies 85% of non-churners

true_positives = actual_churners * sensitivity     # 1,800
false_positives = actual_non_churners * (1 - specificity)  # 14,700

precision = true_positives / (true_positives + false_positives)
print(f"Of all flagged customers, only {precision:.1%} actually churn")
# Result: ~10.9% -- even a '90% accurate' model is wrong most of the time when events are rare

Lesson: When events are rare, even accurate models produce many false positives. Always calculate the positive predictive value alongside accuracy.

Regression to the Mean

When you measure an extreme value, the next measurement tends to be closer to the average -- not because of any intervention, but simply due to chance.

Business example: You identify your ten worst-performing stores and provide intensive coaching. Next month, all ten improve significantly. Was the coaching effective?

Not necessarily. The stores were likely selected because they had an unusually bad month. Even without coaching, they would have regressed toward the mean. This is why control groups are essential for evaluating interventions.

# Demonstrating regression to the mean
np.random.seed(42)
true_performance = np.random.normal(100, 10, 50)  # true store performance
measurement_noise = np.random.normal(0, 15, 50)   # random monthly variation

month1 = true_performance + measurement_noise
month2 = true_performance + np.random.normal(0, 15, 50)

# Select 'worst' performers from month 1
worst_10_idx = np.argsort(month1)[:10]
print(f"Month 1 avg (worst 10): {month1[worst_10_idx].mean():.1f}")
print(f"Month 2 avg (same stores): {month2[worst_10_idx].mean():.1f}")
# Month 2 is higher -- regression to mean, not coaching effect

Simpson's Paradox

A trend can appear in several groups of data but disappear or reverse when the groups are combined.

Classic example: Hospital A has a higher survival rate than Hospital B. But Hospital A treats more severe cases. When you control for case severity, Hospital B is actually better at treating both mild and severe cases. The aggregated result misleads.

# Simpson's Paradox example
data = {
    'Group': ['Mild', 'Mild', 'Severe', 'Severe'],
    'Hospital': ['A', 'B', 'A', 'B'],
    'Survived': [81, 234, 192, 55],
    'Total': [87, 270, 263, 80],
}
df = pd.DataFrame(data)
df['Rate'] = df['Survived'] / df['Total']

print(df[['Group','Hospital','Rate']])
# Hospital B is better in both groups

# But aggregate...
agg = df.groupby('Hospital')[['Survived','Total']].sum()
agg['Rate'] = agg['Survived'] / agg['Total']
print(agg)
# Hospital A looks better overall due to patient mix

Lesson: Always check whether aggregate trends hold within meaningful subgroups. Segment your data before drawing conclusions.

Survivorship Bias

You only see the data that survived a selection process, which creates a misleading picture.

Examples:

• Studying successful startups to learn what makes companies succeed (you never see the failures with identical strategies)
• Analysing customer reviews to understand product quality (unhappy customers often stop purchasing and reviewing)
• Looking at published academic studies to understand a topic (studies showing no effect are less likely to be published)

# Survivorship bias in customer analysis
# Imagine you analyse only active customers
active_customers_satisfaction = [7, 8, 9, 8, 7, 9, 8, 6, 9, 8]  # high satisfaction

# But churned customers also exist -- you just can't easily survey them
churned_customers_satisfaction = [2, 3, 1, 4, 2, 3, 2, 1, 3, 2]  # low satisfaction

# If you only look at active customers:
print(f"Active-only satisfaction: {np.mean(active_customers_satisfaction):.1f}")
# True picture when you include all:
all_satisfaction = active_customers_satisfaction + churned_customers_satisfaction
print(f"True satisfaction: {np.mean(all_satisfaction):.1f}")

Correlation Does Not Imply Causation

This cannot be stated enough. Strong correlations are often coincidental, driven by confounding variables, or the result of reverse causation.

Framework for evaluating causal claims:

1. Plausibility: Is there a logical mechanism by which A could cause B?
2. Temporality: Does A precede B in time?
3. Dose-response: Does more A lead to more B?
4. Experiment: Has a controlled experiment confirmed the relationship?
5. Confounders: Is there a third variable causing both A and B?

The gold standard for establishing causality is a randomised controlled experiment (A/B test). When experiments are not possible, techniques like difference-in-differences, instrumental variables, and propensity score matching can help.

Thinking About Sample Size and Power

Before collecting data, ask: "Is my sample large enough to detect the effect I care about?"

from scipy import stats

def minimum_sample_size(effect_size, alpha=0.05, power=0.80):
    """
    Estimate minimum sample size per group for a two-sample t-test.
    effect_size: Cohen's d (0.2=small, 0.5=medium, 0.8=large)
    """
    from statsmodels.stats.power import TTestIndPower
    analysis = TTestIndPower()
    n = analysis.solve_power(effect_size=effect_size, alpha=alpha, power=power)
    return int(np.ceil(n))

# How many samples needed?
print(f"Small effect (d=0.2): {minimum_sample_size(0.2)} per group")
print(f"Medium effect (d=0.5): {minimum_sample_size(0.5)} per group")
print(f"Large effect (d=0.8): {minimum_sample_size(0.8)} per group")

Running an underpowered study wastes resources and produces unreliable results. Running an overpowered study is also wasteful. Power analysis helps you right-size your data collection.

Practical Statistical Thinking Checklist

Use this before finalising any analysis:

1. What is the population I am drawing conclusions about? Is my sample representative?
2. What is the base rate? Rare events require much larger samples and produce many false positives.
3. Could regression to the mean explain this change? Did I select based on extreme values?
4. Does this aggregate trend hold in subgroups? Simpson's paradox check.
5. Am I missing the failures? Survivorship bias check.
6. Am I claiming causation from correlation? Do I have an experiment or a rigorous causal method?
7. Is my sample large enough to detect the effect I care about? Power check.
8. How would I explain this finding to someone who wants to disprove it? Red team your conclusion.

Key Takeaways

• Variation is natural in all data. Distinguish common cause (random) from special cause (real signal) before acting.
• The base rate fallacy: even accurate models produce mostly false positives when the true event is rare. Always compute precision, not just accuracy.
• Regression to the mean explains why extreme performers naturally improve without intervention. Control groups are essential.
• Simpson's paradox: aggregate trends can be misleading. Always segment and check subgroups before concluding.
• Survivorship bias and correlation-causation confusion are among the most common reasoning errors in business analysis. Build the habit of asking what data you cannot see.

Practice Exercise

import numpy as np
import pandas as pd

# Scenario: You are analysing a new employee training programme.
# The 20 lowest-performing employees were enrolled and re-evaluated 3 months later.

np.random.seed(99)
true_performance = np.random.normal(70, 15, 200)  # True performance scores
month1 = true_performance + np.random.normal(0, 10, 200)  # Observed in month 1
month2 = true_performance + np.random.normal(0, 10, 200)  # Observed in month 2 (no real change)

# Select the 20 lowest performers from month 1
bottom_20 = np.argsort(month1)[:20]

print("=== Training Programme Analysis ===")
print(f"Month 1 avg score (bottom 20): {month1[bottom_20].mean():.1f}")
print(f"Month 2 avg score (same 20):   {month2[bottom_20].mean():.1f}")
print(f"Improvement: {month2[bottom_20].mean() - month1[bottom_20].mean():.1f} points")
print()
print("But for ALL 200 employees:")
print(f"Month 1 avg: {month1.mean():.1f}")
print(f"Month 2 avg: {month2.mean():.1f}")
print()
print("Conclusion: The apparent improvement is regression to the mean, not a training effect.")
print("To isolate the training effect, you need a control group of similar employees who did NOT receive training.")