10 minutes·Lesson 1 of 5

End-to-End Data Project

What is an End-to-End Data Project?

An end-to-end data project covers the complete lifecycle from business question to decision. It is the difference between isolated analysis tasks and work that actually changes how an organisation operates. Real data analyst work is rarely "run this query" -- it is "understand the problem, get the data, clean it, analyse it, communicate it, and help the business act on it."

In this lesson you will walk through the complete structure of a real-world project: a customer churn analysis for a subscription business.

Phase 1: Define the Business Question

Analysis without a clear question produces findings nobody uses. Start every project by writing the question in plain business language.

Bad question: "Analyse our churn data."

Good question: "Which customer segments are churning at the highest rate, and what are the key behavioural signals that precede churn in the 30 days before cancellation?"

A well-formed business question has:

A specific outcome (churn rate by segment)
A timeframe (30 days before cancellation)
An actionable angle (signals that can be acted on)

Write down the question and share it with your stakeholder before doing any analysis. Misaligned questions waste weeks.

Phase 2: Understand the Data Available

Before writing a single query, map out what data sources exist and their quality.

import pandas as pd
import numpy as np

# Load datasets
customers = pd.read_csv('customers.csv')
events = pd.read_csv('user_events.csv')
subscriptions = pd.read_csv('subscriptions.csv')

# Understand what you have
print("=== Customers ===")
print(customers.shape)
print(customers.dtypes)
print(customers.isnull().sum())

print("\n=== Events ===")
print(events.shape)
print(events['event_type'].value_counts())
print(f"Date range: {events['timestamp'].min()} to {events['timestamp'].max()}")

print("\n=== Subscriptions ===")
print(subscriptions['status'].value_counts())

Document what you find. Know your data limitations before you promise results.

Phase 3: Data Cleaning and Preparation

Real data is never clean. The typical breakdown: 60-70% of project time is data preparation.

# Convert types
events['timestamp'] = pd.to_datetime(events['timestamp'])
customers['signup_date'] = pd.to_datetime(customers['signup_date'])
subscriptions['cancel_date'] = pd.to_datetime(subscriptions['cancel_date'])

# Remove test accounts (a common issue in production databases)
customers = customers[~customers['email'].str.contains('test|internal|@company.com', na=False)]

# Deduplicate
customers = customers.drop_duplicates(subset='customer_id')

# Define churn: cancelled in the last 90 days
from datetime import datetime, timedelta
cutoff = datetime.now() - timedelta(days=90)
churned_ids = subscriptions[
    (subscriptions['status'] == 'cancelled') &
    (subscriptions['cancel_date'] >= cutoff)
]['customer_id'].unique()

customers['churned'] = customers['customer_id'].isin(churned_ids).astype(int)
print(f"Churn rate: {customers['churned'].mean():.1%}")

Phase 4: Exploratory Analysis

Explore before you model. Understand distributions, segment differences, and key patterns.

import matplotlib.pyplot as plt
import seaborn as sns

# Churn rate by plan type
churn_by_plan = customers.groupby('plan_type')['churned'].agg(['mean', 'count']).reset_index()
churn_by_plan.columns = ['plan_type', 'churn_rate', 'customers']
churn_by_plan['churn_rate_pct'] = (churn_by_plan['churn_rate'] * 100).round(1)
print(churn_by_plan)

# Churn rate by cohort (signup month)
customers['signup_month'] = customers['signup_date'].dt.to_period('M')
cohort_churn = customers.groupby('signup_month')['churned'].mean()

# Activity in last 30 days before potential churn window
last_30_events = events[events['timestamp'] >= (datetime.now() - timedelta(days=30))]
activity = last_30_events.groupby('customer_id').size().reset_index(name='events_last_30d')

customers = customers.merge(activity, on='customer_id', how='left')
customers['events_last_30d'] = customers['events_last_30d'].fillna(0)

# Compare activity between churned and retained
print("\nMean events (last 30d):")
print(customers.groupby('churned')['events_last_30d'].mean())

Phase 5: Statistical Validation

Do not report a finding without knowing whether it is statistically significant.

from scipy import stats

churned = customers[customers['churned'] == 1]['events_last_30d']
retained = customers[customers['churned'] == 0]['events_last_30d']

t_stat, p_value = stats.ttest_ind(churned, retained)
print(f"t-statistic: {t_stat:.2f}")
print(f"p-value: {p_value:.4f}")

if p_value < 0.05:
    print("Statistically significant difference in activity between churned and retained customers.")
else:
    print("No statistically significant difference found.")

# Effect size (Cohen's d)
pooled_std = np.sqrt((churned.std()**2 + retained.std()**2) / 2)
cohens_d = (retained.mean() - churned.mean()) / pooled_std
print(f"Cohen's d: {cohens_d:.2f} (effect size)")

Phase 6: Build the Narrative

Analysis is not a report. It is a story with a point of view. Structure your output around the business question.

Structure for a churn analysis:

Executive summary (3 bullets, no jargon): What are the most important findings?
Churn rate overview: Current rate, trend, and comparison to benchmark
Segment breakdown: Which segments churn most?
Leading indicators: What behaviours precede churn?
Recommended actions: Specific, prioritised, with estimated impact
Limitations and assumptions: What you could not measure

Every chart should have a title that states the conclusion ("Monthly active users down 23% among free-tier customers") not just a description ("Monthly active users by tier").

Phase 7: Deliver and Follow Up

Analysis without action is wasted. After presenting:

Send a written summary with key findings and recommendations within 24 hours
Offer to meet with the team who will implement changes
Schedule a follow-up in 30-60 days to measure whether the intervention worked
Document your methodology so others can reproduce or update it

Key Takeaways

Define the business question before touching any data. Misaligned questions waste weeks of work.
Data preparation typically takes 60-70% of total project time. Plan for it.
Explore before you model -- distributions, segment differences, and correlations should be understood before formal analysis.
Statistical significance tells you the finding is real; effect size tells you whether it matters in practice. Report both.
Analysis ends only when the business acts. Build your output around actionable recommendations, not just findings.

Practice Exercise

import pandas as pd
import numpy as np
from scipy import stats

# Generate synthetic churn dataset
np.random.seed(42)
n = 1000

df = pd.DataFrame({
    'customer_id': range(1, n+1),
    'plan_type': np.random.choice(['free', 'basic', 'premium'], n, p=[0.5, 0.3, 0.2]),
    'tenure_days': np.random.exponential(180, n).astype(int),
    'support_tickets': np.random.poisson(1.2, n),
    'events_last_30d': np.random.poisson(15, n),
})

# Simulate churn -- higher for free, lower tenure, low activity
churn_prob = (
    (df['plan_type'] == 'free') * 0.3 +
    (df['tenure_days'] < 30) * 0.4 +
    (df['events_last_30d'] < 5) * 0.35 +
    (df['support_tickets'] > 3) * 0.25
).clip(0, 1)
df['churned'] = (np.random.rand(n) < churn_prob * 0.4).astype(int)

# 1. Calculate overall churn rate
print(f"Overall churn rate: {df['churned'].mean():.1%}")

# 2. Churn rate by plan type
print("\nChurn rate by plan:")
print(df.groupby('plan_type')['churned'].mean().round(3))

# 3. Are low-activity users more likely to churn? (t-test)
low_activity = df[df['events_last_30d'] < 5]['churned']
high_activity = df[df['events_last_30d'] >= 5]['churned']
t, p = stats.ttest_ind(low_activity, high_activity)
print(f"\nLow vs high activity churn -- p-value: {p:.4f}")

# 4. Write 3 business recommendations based on your findings

Try it yourself

Preview

Key Takeaways

Define the business question before touching data. Share it with stakeholders and confirm alignment to avoid weeks of misaligned analysis.
Data preparation is 60-70% of project time. Document data sources, quality issues, and assumptions before analysis begins.
Explore before you model. Understand distributions, segment differences, and patterns through visualisation and summary statistics.
Report both statistical significance (is it real?) and effect size (does it matter?). Significant but tiny effects rarely justify action.
Analysis ends when the business acts. Deliver written summaries, support implementation, and measure outcomes to close the loop.

Quick Quiz

1.What is the most important first step in any data analysis project?

2.Approximately what proportion of a typical data project is spent on data cleaning and preparation?

3.You find that churned customers have an average of 3.2 events in the last 30 days while retained customers average 18.7 events (p < 0.001, Cohen's d = 1.8). How should you interpret this?

4.After presenting a churn analysis with clear recommendations, what should a data analyst do next?

← Statistical Thinking for Analysts Next: Presenting Data Insights →

Ready to go further?

CareerEx gives you structured 12-week training, live classes every Saturday and Sunday, real tutor feedback, and a certificate. Join the next cohort.

Join CareerEx

10 minutes·Lesson 1 of 5

End-to-End Data Project

What is an End-to-End Data Project?

In this lesson you will walk through the complete structure of a real-world project: a customer churn analysis for a subscription business.

Phase 1: Define the Business Question

Analysis without a clear question produces findings nobody uses. Start every project by writing the question in plain business language.

Bad question: "Analyse our churn data."

Good question: "Which customer segments are churning at the highest rate, and what are the key behavioural signals that precede churn in the 30 days before cancellation?"

A well-formed business question has:

A specific outcome (churn rate by segment)
A timeframe (30 days before cancellation)
An actionable angle (signals that can be acted on)

Write down the question and share it with your stakeholder before doing any analysis. Misaligned questions waste weeks.

Phase 2: Understand the Data Available

Before writing a single query, map out what data sources exist and their quality.

import pandas as pd
import numpy as np

# Load datasets
customers = pd.read_csv('customers.csv')
events = pd.read_csv('user_events.csv')
subscriptions = pd.read_csv('subscriptions.csv')

# Understand what you have
print("=== Customers ===")
print(customers.shape)
print(customers.dtypes)
print(customers.isnull().sum())

print("\n=== Events ===")
print(events.shape)
print(events['event_type'].value_counts())
print(f"Date range: {events['timestamp'].min()} to {events['timestamp'].max()}")

print("\n=== Subscriptions ===")
print(subscriptions['status'].value_counts())

Document what you find. Know your data limitations before you promise results.

Phase 3: Data Cleaning and Preparation

Real data is never clean. The typical breakdown: 60-70% of project time is data preparation.

# Convert types
events['timestamp'] = pd.to_datetime(events['timestamp'])
customers['signup_date'] = pd.to_datetime(customers['signup_date'])
subscriptions['cancel_date'] = pd.to_datetime(subscriptions['cancel_date'])

# Remove test accounts (a common issue in production databases)
customers = customers[~customers['email'].str.contains('test|internal|@company.com', na=False)]

# Deduplicate
customers = customers.drop_duplicates(subset='customer_id')

# Define churn: cancelled in the last 90 days
from datetime import datetime, timedelta
cutoff = datetime.now() - timedelta(days=90)
churned_ids = subscriptions[
    (subscriptions['status'] == 'cancelled') &
    (subscriptions['cancel_date'] >= cutoff)
]['customer_id'].unique()

customers['churned'] = customers['customer_id'].isin(churned_ids).astype(int)
print(f"Churn rate: {customers['churned'].mean():.1%}")

Phase 4: Exploratory Analysis

Explore before you model. Understand distributions, segment differences, and key patterns.

import matplotlib.pyplot as plt
import seaborn as sns

# Churn rate by plan type
churn_by_plan = customers.groupby('plan_type')['churned'].agg(['mean', 'count']).reset_index()
churn_by_plan.columns = ['plan_type', 'churn_rate', 'customers']
churn_by_plan['churn_rate_pct'] = (churn_by_plan['churn_rate'] * 100).round(1)
print(churn_by_plan)

# Churn rate by cohort (signup month)
customers['signup_month'] = customers['signup_date'].dt.to_period('M')
cohort_churn = customers.groupby('signup_month')['churned'].mean()

# Activity in last 30 days before potential churn window
last_30_events = events[events['timestamp'] >= (datetime.now() - timedelta(days=30))]
activity = last_30_events.groupby('customer_id').size().reset_index(name='events_last_30d')

customers = customers.merge(activity, on='customer_id', how='left')
customers['events_last_30d'] = customers['events_last_30d'].fillna(0)

# Compare activity between churned and retained
print("\nMean events (last 30d):")
print(customers.groupby('churned')['events_last_30d'].mean())

Phase 5: Statistical Validation

Do not report a finding without knowing whether it is statistically significant.

from scipy import stats

churned = customers[customers['churned'] == 1]['events_last_30d']
retained = customers[customers['churned'] == 0]['events_last_30d']

t_stat, p_value = stats.ttest_ind(churned, retained)
print(f"t-statistic: {t_stat:.2f}")
print(f"p-value: {p_value:.4f}")

if p_value < 0.05:
    print("Statistically significant difference in activity between churned and retained customers.")
else:
    print("No statistically significant difference found.")

# Effect size (Cohen's d)
pooled_std = np.sqrt((churned.std()**2 + retained.std()**2) / 2)
cohens_d = (retained.mean() - churned.mean()) / pooled_std
print(f"Cohen's d: {cohens_d:.2f} (effect size)")

Phase 6: Build the Narrative

Analysis is not a report. It is a story with a point of view. Structure your output around the business question.

Structure for a churn analysis:

Executive summary (3 bullets, no jargon): What are the most important findings?
Churn rate overview: Current rate, trend, and comparison to benchmark
Segment breakdown: Which segments churn most?
Leading indicators: What behaviours precede churn?
Recommended actions: Specific, prioritised, with estimated impact
Limitations and assumptions: What you could not measure

Every chart should have a title that states the conclusion ("Monthly active users down 23% among free-tier customers") not just a description ("Monthly active users by tier").

Phase 7: Deliver and Follow Up

Analysis without action is wasted. After presenting:

Send a written summary with key findings and recommendations within 24 hours
Offer to meet with the team who will implement changes
Schedule a follow-up in 30-60 days to measure whether the intervention worked
Document your methodology so others can reproduce or update it

Key Takeaways

Define the business question before touching any data. Misaligned questions waste weeks of work.
Data preparation typically takes 60-70% of total project time. Plan for it.
Explore before you model -- distributions, segment differences, and correlations should be understood before formal analysis.
Statistical significance tells you the finding is real; effect size tells you whether it matters in practice. Report both.
Analysis ends only when the business acts. Build your output around actionable recommendations, not just findings.

Practice Exercise

import pandas as pd
import numpy as np
from scipy import stats

# Generate synthetic churn dataset
np.random.seed(42)
n = 1000

df = pd.DataFrame({
    'customer_id': range(1, n+1),
    'plan_type': np.random.choice(['free', 'basic', 'premium'], n, p=[0.5, 0.3, 0.2]),
    'tenure_days': np.random.exponential(180, n).astype(int),
    'support_tickets': np.random.poisson(1.2, n),
    'events_last_30d': np.random.poisson(15, n),
})

# Simulate churn -- higher for free, lower tenure, low activity
churn_prob = (
    (df['plan_type'] == 'free') * 0.3 +
    (df['tenure_days'] < 30) * 0.4 +
    (df['events_last_30d'] < 5) * 0.35 +
    (df['support_tickets'] > 3) * 0.25
).clip(0, 1)
df['churned'] = (np.random.rand(n) < churn_prob * 0.4).astype(int)

# 1. Calculate overall churn rate
print(f"Overall churn rate: {df['churned'].mean():.1%}")

# 2. Churn rate by plan type
print("\nChurn rate by plan:")
print(df.groupby('plan_type')['churned'].mean().round(3))

# 3. Are low-activity users more likely to churn? (t-test)
low_activity = df[df['events_last_30d'] < 5]['churned']
high_activity = df[df['events_last_30d'] >= 5]['churned']
t, p = stats.ttest_ind(low_activity, high_activity)
print(f"\nLow vs high activity churn -- p-value: {p:.4f}")

# 4. Write 3 business recommendations based on your findings

Try it yourself

Preview

Key Takeaways

Define the business question before touching data. Share it with stakeholders and confirm alignment to avoid weeks of misaligned analysis.
Data preparation is 60-70% of project time. Document data sources, quality issues, and assumptions before analysis begins.
Explore before you model. Understand distributions, segment differences, and patterns through visualisation and summary statistics.
Report both statistical significance (is it real?) and effect size (does it matter?). Significant but tiny effects rarely justify action.
Analysis ends when the business acts. Deliver written summaries, support implementation, and measure outcomes to close the loop.

Quick Quiz

1.What is the most important first step in any data analysis project?

2.Approximately what proportion of a typical data project is spent on data cleaning and preparation?

3.You find that churned customers have an average of 3.2 events in the last 30 days while retained customers average 18.7 events (p < 0.001, Cohen's d = 1.8). How should you interpret this?

4.After presenting a churn analysis with clear recommendations, what should a data analyst do next?

← Statistical Thinking for Analysts Next: Presenting Data Insights →

Ready to go further?

CareerEx gives you structured 12-week training, live classes every Saturday and Sunday, real tutor feedback, and a certificate. Join the next cohort.

Join CareerEx