PadhaikoAll subjects

Data Visualization and Storytelling

Advanced Python Programming for Data Science (ENCT 325)

Open in the Padhaiko app

5.1 Principles of effective visualization and dashboard design

A visualization succeeds when a viewer grasps the pattern, comparison, or anomaly accurately in seconds without misreading scale or drowning in clutter.

Core Principles of Effective Visualization

  1. Match Chart Type to Variable Types:
    • Trend over time: Line chart.
    • Category comparison: Horizontal or vertical bar chart (zero baseline required!).
    • Single distribution: Histogram, KDE, or box/violin plot.
    • Relationship between two continuous variables: Scatter plot.
    • Part-to-whole: Stacked bar or 100% stacked bar (avoid 3D pie charts or pies with more than 5 slices).
  2. Maximize Data-Ink Ratio (Edward Tufte): Remove redundant 3D effects, heavy dark gridlines, chartjunk, and decorative borders so visual ink encodes data values.
  3. Honest Scales and Integrity:
    • Bar charts must start at zero (0) because bar length encodes magnitude; truncating a bar axis exaggerates small differences.
    • Line charts may zoom to a narrower y-range when small fluctuations are meaningful, provided the axis range is clearly labeled.
  4. Color and Accessibility: Use sequential palettes (viridis, Blues) for ordered magnitudes, diverging palettes (coolwarm, RdBu) for values centered on zero/target, and qualitative palettes (tab10) for distinct categories—preferring colorblind-safe palettes and direct labels.

Dashboard Design Principles

Definition

Effective Dashboard Architecture


5.2 Visualization with Matplotlib: Line, bar, histogram, scatter, subplots

Matplotlib is Python's foundational plotting library. Using its object-oriented API (fig, ax = plt.subplots()) instead of stateful global plt.plot() calls gives fine-grained control over multi-panel figures, styling, and automated script exports.

Core Matplotlib Plot Types and Methods
Plot TypeOOP MethodData Purpose & Key Arguments
Line Plotax.plot(x, y, marker="o", linestyle="-")Continuous trends or time-series trajectories
Bar Chartax.bar(cats, vals) / ax.barh(...)Comparing discrete category magnitudes (barh fits long labels cleanly)
Histogramax.hist(x, bins=20, edgecolor="black")Frequency distribution of one continuous variable (density=True normalizes area to 1)
Scatter Plotax.scatter(x, y, c=z, s=sizes, alpha=0.7)Joint relationship, clusters, and outliers between two numeric features
Subplots Gridfig, axes = plt.subplots(2, 2, figsize=(10, 6))Multi-panel layouts; call fig.tight_layout() or constrained_layout=True to avoid label overlap
Python
import matplotlib.pyplot as plt

months = ["Jan", "Feb", "Mar", "Apr"]
revenue = [120, 145, 138, 172]
orders = [40, 48, 45, 58]

fig, axes = plt.subplots(1, 2, figsize=(10, 4), constrained_layout=True)

axes[0].plot(months, revenue, marker="o", linewidth=2)
axes[0].set_title("Monthly Revenue")
axes[0].set_ylabel("Revenue (k NPR)")

axes[1].bar(months, orders, color="steelblue")
axes[1].set_title("Monthly Orders")
axes[1].set_ylabel("Order Count")

fig.savefig("monthly_summary.png", dpi=150)

5.3 Seaborn for statistical visualization: Box plot, pair plot, heat map

Seaborn builds on Matplotlib with a tidy-DataFrame-first API (data=df, x=..., y=..., hue=...) and built-in statistical aggregations (confidence intervals, kernel densities, quantiles).

Definition

Key Statistical Plots in Seaborn

AspectMatplotlibSeaborn
Abstraction LevelLow-level, imperative control over every artist, tick, spine, and annotationHigh-level statistical interface working directly on Pandas DataFrames
Statistical DefaultsRaw arrays; manual binning, grouping, and confidence intervalsAutomatic grouping (hue, col), bootstrap CIs, regression fits, correlation heatmaps
InteroperabilityFoundation layerReturns/accepts Matplotlib Axes (ax=axes[0]), so both mix seamlessly
Python
import seaborn as sns
import matplotlib.pyplot as plt

corr = df[["age", "income", "spend_score"]].corr()
fig, ax = plt.subplots(figsize=(6, 5))
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm", vmin=-1, vmax=1, ax=ax)
ax.set_title("Feature Correlation Matrix")

5.4 Interactive visualization using Plotly

Static PNG charts cannot zoom into dense time series, hover over outlier metadata, or filter categories live. Plotly (plotly.express and plotly.graph_objects) renders interactive HTML/JavaScript charts supporting tooltips, pan/zoom, legend toggling, and animated frames.

plotly.express vs. plotly.graph_objects

  1. plotly.express (px): Concise one-line API analogous to Seaborn for rapid interactive exploration (px.scatter, px.line, px.bar, px.box, px.histogram), supporting hover_data=, color=, facet_col=, and animation_frame=.
  2. plotly.graph_objects (go): Low-level composable figure builder (fig.add_trace(go.Scatter(...)), make_subplots()) for custom dual-axis dashboards, annotations, and fine-grained layout control.
  3. Exporting & Embedding: Save standalone interactive HTML files via fig.write_html("report.html") (zero backend server needed) or integrate directly inside Dash or Streamlit web apps.
Python
import plotly.express as px

fig = px.scatter(
    df,
    x="marketing_spend",
    y="revenue",
    color="region",
    size="customers",
    hover_data=["month"],
    title="Marketing Spend vs. Revenue by Region",
)
fig.update_layout(template="plotly_white")
fig.write_html("interactive_revenue.html")

5.5 Visualization driven insight generation

Data storytelling transforms raw chart outputs into clear, actionable decisions for technical and non-technical stakeholders.

Definition

Three Pillars of Data Storytelling

  1. Data (Evidence): Verified metrics and fair visual encodings without cherry-picked ranges or distorted axes.
  2. Visuals (Focus): Deliberate visual hierarchy—muting background series in light gray while highlighting the critical trend in accent color, plus direct callout annotations (ax.annotate(...)) at inflection points.
  3. Narrative (Actionable Context): Follow the Context → Complication (Finding) → Resolution (Recommendation) arc: replace generic titles like "Sales vs. Month" with insight-driven headlines like "Q3 Mobile Sales Dropped 18% Following Checkout Redesign".

Avoiding Misleading Visualizations


5.6 Case study: End-to-end visualization and reporting project

An end-to-end visualization project connects ingestion, cleaning, statistical summaries, multi-library charts, and automated report delivery into a repeatable script.

End-to-End Reporting Pipeline FlowText
Raw CSV / SQL ---> Clean & Aggregate ---> Matplotlib / Seaborn (PNG) ---> HTML / PDF Report
  (pandas)          (groupby/KPIs)       Plotly Interactive (HTML)       (Jinja2 / Dash)

Case Study: Retail Churn & Sales Diagnostic Report

Suppose a telecom analytics team delivers a weekly executive report from orders.csv:

  1. Load & Clean: Parse dates (pd.to_datetime), filter valid transactions, compute derived KPIs (net_revenue, churn_rate, avg_order_value).
  2. Overview Panel (Matplotlib Subplots): Export a 2x1 high-DPI summary figure comparing weekly revenue trend (with annotated campaign launch date) against category bar totals.
  3. Statistical Diagnostic (Seaborn): Plot a boxplot of order size by customer tier (highlighting high-value outliers) and a heatmap of numeric feature correlations.
  4. Interactive Drill-Down (Plotly): Generate scatter_drilldown.html with hover_data=["customer_id", "plan"] so managers inspect individual outlier accounts directly in a browser.
  5. Insight-First Summary: Lead the final report with three quantified takeaways (e.g., "Month-to-month contracts show 3.2x higher churn than annual plans; targeted renewal offers in Week 3 recover 14% of at-risk revenue").

Worked Examples

Example 5.1

5.1

A Seaborn boxplot summarizes response latencies (in ms) with min_raw = 5, Q1 = 20, median = 31, Q3 = 45, and max_raw = 92. Compute the box height (IQR), the theoretical upper and lower whisker fences (1.5 * IQR), and state whether 92 and 5 are drawn inside the whiskers or plotted as outlier points.

Computing Box Height (IQR) and Whisker Fences:

  • IQR = Q3 - Q1 = 45 - 20 = 25 ms.
  • Upper fence = Q3 + 1.5 * IQR = 45 + 1.5 * 25 = 45 + 37.5 = 82.5 ms.
  • Lower fence = Q1 - 1.5 * IQR = 20 - 1.5 * 25 = 20 - 37.5 = -17.5 ms.

Classifying min_raw = 5 and max_raw = 92:

  • Since 5 >= -17.5, 5 ms lies well within the lower fence, so the lower whisker extends down to 5 ms (no lower outliers).
  • Since 92 > 82.5, 92 ms exceeds the upper fence, so 92 ms is plotted as an outlier dot, and the upper whisker stops at the largest observed point <= 82.5 ms.
Answer:

IQR = 25 ms, lower fence = -17.5 ms, upper fence = 82.5 ms. Thus 5 ms is the lower whisker endpoint (not an outlier), whereas 92 ms exceeds 82.5 ms and is plotted as an upper outlier point.

Example 5.2

5.2

A manager creates a bar chart comparing Plan A (100 users) and Plan B (105 users) with the y-axis starting at 98 instead of 0.

  1. Compute the true percentage increase from Plan A to Plan B.
  2. Compute the visual bar-height ratio (height_B / height_A) when y_min = 98, and write the Matplotlib OOP fix enforcing a zero baseline with direct bar labels.

Quantifying the Visual Distortion:

  • True percentage increase = ((105 - 100) / 100) * 100 = 5.0% (true ratio 105 / 100 = 1.05x).
  • When truncated at y_min = 98:
    • Plan A bar height = 100 - 98 = 2 units.
    • Plan B bar height = 105 - 98 = 7 units.
    • Visual height ratio = 7 / 2 = 3.5x—making a modest 5% gain look 3.5 times larger!

Corrected Matplotlib OOP Code:

Python
import matplotlib.pyplot as plt

plans = ["Plan A", "Plan B"]
users = [100, 105]

fig, ax = plt.subplots(figsize=(5, 4))
bars = ax.bar(plans, users, color=["#8c8c8c", "#1f77b4"])
ax.set_ylim(0, 120)
ax.bar_label(bars, padding=3)
ax.set_title("Active Users: Plan B Leads Plan A by 5%")
ax.set_ylabel("Users")
Answer:

The true increase is 5.0% (1.05x), but starting the bar y-axis at 98 inflates the visual height ratio to 3.5x (7 / 2). Setting ax.set_ylim(0, 120) and adding ax.bar_label(bars) restores honest visual comparison.

Example 5.3

5.3

How should a student answer an exam question asking: "Design an end-to-end Python visualization and reporting workflow using Matplotlib, Seaborn, and Plotly, specifying what each library contributes"?

Assigning Clear Roles to Each Library:

  1. Pandas Aggregation: Clean data and compute KPI summary tables (groupby, .resample).
  2. Matplotlib (fig, ax = plt.subplots): Produce crisp, publication-grade multi-panel overview figures (png/svg) with custom annotations, consistent axes, and high DPI.
  3. Seaborn (boxplot, heatmap, pairplot): Reveal statistical distributions, outliers, and correlation structure across subgroups with minimal code.
  4. Plotly (plotly.express + write_html): Deliver interactive stakeholder drill-down charts with hover tooltips, legend filtering, and zoom saved as self-contained HTML.

Adding Storytelling and Delivery Structure: Conclude with the inverted-pyramid layout (KPIs → visual insights with action-oriented titles → drill-down links) and automated export into an HTML/PDF weekly report.

Answer:

Describe a 4-stage workflow: Pandas aggregation → Matplotlib OOP multi-panel KPI charts → Seaborn statistical diagnostics (boxplot, heatmap) → Plotly interactive HTML drill-downs, unified under an inverted-pyramid narrative with insight-driven titles.


Exercises

Practice Exercises — Data Visualization and Storytelling

Question 1

Explain Edward Tufte's data-ink ratio and state which chart type is most appropriate for: (a) tracking monthly electricity consumption over 3 years, (b) comparing median salary and outliers across 5 departments, and (c) inspecting pairwise correlations among 8 numeric features.

Data-Ink Ratio: The data-ink ratio is the proportion of a graphic's ink devoted to displaying non-redundant data information (data-ink / total-ink). Removing 3D bars, heavy gridlines, decorative frames, and redundant legends increases clarity and comprehension speed.

Choosing Appropriate Charts:

  • (a) Monthly consumption over 3 years: Line chart (emphasizes temporal continuity and trend).
  • (b) Median salary and outliers across 5 departments: Box plot (or violin + box overlay; displays median, IQR spread, skew, and outliers simultaneously).
  • (c) Pairwise correlations among 8 numeric features: Annotated Heatmap (sns.heatmap(df.corr(), annot=True, center=0)).
Answer:

The data-ink ratio measures the share of visual ink representing actual data rather than clutter. Use a line chart for (a), a box plot for (b), and an annotated correlation heatmap for (c).

Question 2

Why is Matplotlib's object-oriented API (fig, ax = plt.subplots(...)) preferred over the stateful plt.plot(...) API for data science projects? Write a 2x1 subplot example plotting a line chart on top and a histogram below with sharex=False.

Why OOP (fig, ax) Beats Stateful plt: Global plt functions operate implicitly on whichever figure/axes happens to be "current," causing bugs in multi-panel layouts, functions, and automated batch scripts. Explicit Figure and Axes objects let you target axes[0], axes[1] deterministically and pass ax= into Seaborn or helper functions.

2x1 Matplotlib OOP Subplot Example:

Python
import matplotlib.pyplot as plt

fig, axes = plt.subplots(2, 1, figsize=(7, 6), constrained_layout=True)
axes[0].plot([1, 2, 3, 4], [10, 14, 13, 18], marker="o")
axes[0].set_title("Weekly Trend")

axes[1].hist([12, 15, 15, 18, 20, 22, 22, 25], bins=5, edgecolor="black")
axes[1].set_title("Score Distribution")
Answer:

The OOP API binds every plot call to explicit fig and ax objects, preventing state collisions in multi-panel and automated scripts. Use fig, axes = plt.subplots(2, 1) and call axes[0].plot(...) and axes[1].hist(...).

Question 3

Compare Matplotlib, Seaborn, and Plotly across abstraction level, interactivity, and ideal use cases. Give one code snippet using plotly.express to build an interactive scatter plot saved to HTML.

Library Comparison:

  • Matplotlib: Low-level static engine; total control over publication figures, layouts, and custom annotations.
  • Seaborn: High-level statistical wrapper around Matplotlib working directly on tidy Pandas DataFrames (hue, boxplot, pairplot, heatmap).
  • Plotly: Interactive browser-ready charts (plotly.express / graph_objects) with hover tooltips, zoom/pan, legend toggling, and standalone HTML export.

Plotly Express Snippet:

Python
import plotly.express as px

fig = px.scatter(df, x="hours_studied", y="exam_score", color="section",
                 hover_data=["student_id"], title="Study Hours vs. Exam Score")
fig.write_html("study_scatter.html")
Answer:

Matplotlib offers low-level static customization; Seaborn offers concise DataFrame-based statistical plots on Matplotlib; Plotly offers interactive browser charts (zoom, hover, HTML export via fig.write_html()).

Question 4

What insights can you read from a Seaborn pairplot and a correlation heatmap? For a DataFrame with 8 numeric columns, how many total subplots appear in a full sns.pairplot(df) grid, and how many unique off-diagonal pairs exist?

Reading Pair Plots and Heatmaps:

  • A pair plot reveals univariate distributions on the diagonal (skewness, multimodality) and pairwise non-linear shapes, clusters, or class separation on off-diagonal scatter plots.
  • A correlation heatmap summarizes linear (or Spearman) strength and sign (-1 to +1) at a glance, quickly flagging multicollinear feature pairs (|r| > 0.85) and strong predictors of the target.

Grid Size Calculation for 8 Columns:

  • Full pairplot grid dimensions = 8 x 8 = 64 subplots (8 diagonal distribution plots + 56 off-diagonal scatter plots).
  • Number of unique off-diagonal pairs = 8 * (8 - 1) / 2 = 28 pairs (which corner=True displays without mirroring duplicates).
Answer:

Pair plots reveal diagonal distributions, non-linear relationships, and cluster separation, while heatmaps quantify pairwise correlation strength and multicollinearity. For 8 numeric columns, a full pairplot renders 64 subplots (8 x 8) representing 28 unique pairs.

Question 5

Explain the three pillars of data storytelling and the inverted-pyramid dashboard layout. If weekly support tickets dropped from 600 to 480 after deploying a self-service FAQ, calculate the percentage change and rewrite the generic chart title "Support Tickets by Week" into an insight-driven headline.

Data Storytelling Pillars and Dashboard Layout:

  • Three Pillars: (1) Verified Data, (2) Focused Visuals (minimal clutter, strategic color/annotations), and (3) Narrative (Context → Finding → Action).
  • Inverted Pyramid: Top = KPI summary cards; Middle = trend and breakdown charts; Bottom = granular filterable tables.

Calculating Percentage Change and Rewriting the Title:

  • Percentage change = ((480 - 600) / 600) * 100 = (-120 / 600) * 100 = -20.0% (a 20% reduction).
  • Insight-driven title: "Self-Service FAQ Cut Weekly Support Tickets by 20% (600 to 480)".
Answer:

Data storytelling combines verified data, focused visuals, and an actionable narrative, while dashboards follow an inverted pyramid (KPIs → trends/breakdowns → tables). Tickets fell by 20.0%, yielding the headline: "Self-Service FAQ Cut Weekly Support Tickets by 20% (600 to 480)".

← Applied Statistics and Exploratory AnalysisData Engineering and Automation →