Finding Patterns in Data

From Summaries to Relationships

Descriptive statistics describe one column at a time. The real insights come from relationships: does spend rise with age? Do USSD users churn more than app users? Which state and product combination drives the most revenue? This lesson covers three tools that answer these questions: correlation, groupby analysis and pivot tables.

We will use a realistic simulated dataset from a Nigerian fintech app with 5,000 users.

Open in Google Colab: Nigerian Fintech Dataset

Open a new notebook and paste the generator below into the first cell. It builds a 5,000-user fintech dataset with realistic patterns hidden inside. Your job in this lesson is to find them.

Launch a Colab notebook →

import numpy as np
import pandas as pd

rng = np.random.default_rng(2025)
n = 5000
states = rng.choice(["Lagos", "Abuja", "Rivers", "Kano", "Oyo", "Enugu"], n, p=[.35, .18, .12, .13, .12, .10])
channel = rng.choice(["App", "USSD", "Web"], n, p=[.6, .3, .1])
age = rng.integers(18, 60, n)
tenure_months = rng.integers(1, 48, n)
income = np.round(rng.lognormal(12.3, 0.55, n) * np.where(states == "Lagos", 1.3, 1.0), -3)
monthly_txns = np.clip(rng.poisson(8, n) + (channel == "App") * 6 + tenure_months // 8, 0, None)
monthly_volume = np.round(monthly_txns * income * rng.uniform(0.02, 0.05, n), -2)
support_tickets = rng.poisson(np.where(channel == "USSD", 2.2, 0.9))
churn_score = -0.04 * tenure_months + 0.45 * support_tickets - 0.05 * monthly_txns + rng.normal(0, 1, n)
churned = (churn_score > np.quantile(churn_score, 0.82)).astype(int)

df = pd.DataFrame({
    "state": states, "channel": channel, "age": age, "tenure_months": tenure_months,
    "income": income, "monthly_txns": monthly_txns, "monthly_volume": monthly_volume,
    "support_tickets": support_tickets, "churned": churned,
})
df.head()

1. Correlation

Correlation measures how strongly two numeric variables move together. The Pearson correlation coefficient r ranges from -1 to +1:

df["monthly_txns"].corr(df["monthly_volume"])        # a pair
corr = df.select_dtypes("number").corr()             # every pair

import seaborn as sns
import matplotlib.pyplot as plt
plt.figure(figsize=(9, 7))
sns.heatmap(corr, annot=True, fmt=".2f", cmap="RdBu_r", vmin=-1, vmax=1)
plt.show()

Look at the row for churned. Which features have the strongest positive and negative correlation with churn?

Three warnings about correlation

1. Correlation is not causation. Ice cream sales and drowning deaths both rise in summer, and neither causes the other. Heat causes both. Always ask whether a third variable (a confounder) could explain the relationship.
2. Pearson only measures linear relationships. A U-shaped relationship can have r close to 0. Plot a scatter chart before trusting a number.
3. Outliers distort r. Use Spearman correlation, which works on ranks, for skewed data: df.corr(method="spearman", numeric_only=True).

2. Groupby Analysis

Groupby compares a metric across segments. It is how most business insights are found.

# Churn rate by channel
df.groupby("channel")["churned"].mean().sort_values(ascending=False)

# Several metrics per state
df.groupby("state").agg(
    users=("churned", "size"),
    churn_rate=("churned", "mean"),
    avg_income=("income", "median"),
    avg_volume=("monthly_volume", "mean"),
).sort_values("avg_volume", ascending=False).round(2)

Binning a numeric variable

To compare across a numeric variable, cut it into groups first:

df["tenure_band"] = pd.cut(df["tenure_months"], bins=[0, 6, 12, 24, 48],
                           labels=["0-6m", "6-12m", "1-2y", "2-4y"])
df.groupby("tenure_band", observed=True)["churned"].mean()

You should see churn falling as tenure increases. That is a classic pattern: new users are the most likely to leave, so onboarding matters.

3. Pivot Tables

A pivot table is a groupby over two dimensions at once, laid out as a grid. It is the same idea as an Excel pivot table.

pivot = pd.pivot_table(
    df, values="churned", index="state", columns="channel", aggfunc="mean"
).round(3)
pivot

sns.heatmap(pivot, annot=True, fmt=".1%", cmap="Reds")
plt.title("Churn rate by state and channel")
plt.show()

pd.crosstab is a shortcut for counting combinations:

pd.crosstab(df["state"], df["channel"], normalize="index").round(2)   # channel mix per state

Turning Patterns Into Insights

A pattern is not an insight until you explain so what? Compare:

• Pattern: "USSD users have a higher churn rate."
• Insight: "USSD users churn at more than twice the rate of app users and raise twice as many support tickets. Improving the USSD experience, or moving USSD users to the app, could reduce churn."

For every pattern you find, write one sentence on what the business should consider doing. Then go back to the heatmap and check the insight holds within every state, not only on average.

Your Challenge

Using the generated dataset, find answers to these questions in your notebook:

1. Which single feature is most correlated with churned?
2. Which state has the highest median income, and why might that be the case in this data?
3. Is the channel effect on churn consistent across all six states?