PadhaikoAll subjects

Advanced Python Concepts and Best Practices

Advanced Python Programming for Data Science (ENCT 325)

Open in the Padhaiko app

1.1 Review of Python essentials and coding conventions

Writing maintainable Python code for data science requires mastering core language semantics alongside clean formatting and documentation standards. Data pipelines are often read, debugged, and extended by multiple engineers, so consistency directly reduces bugs and onboarding time.

Core Python Essentials

  1. Dynamic Typing and Object References: Variables in Python are names bound to objects in memory, not fixed-size storage boxes. Every object carries a type, an identity (id()), and a value.
  2. Mutability vs. Immutability:
    • Immutable types (int, float, str, tuple, frozenset, bytes) cannot be altered after creation; operations return new objects.
    • Mutable types (list, dict, set, bytearray) can be modified in place. Passing a mutable object to a function allows the function to mutate the caller's object unless a copy is made.
  3. Truthiness and Identity: Empty containers ([], {}, set(), ""), 0, 0.0, and None evaluate to False in boolean contexts. Use is / is not only for identity checks (especially x is None), and == / != for value equality.
  4. Comprehensions and Unpacking: List, dictionary, and set comprehensions replace verbose loops with concise expressions, while sequence unpacking (first, *middle, last = seq) cleanly extracts elements.
AspectMutable ObjectsImmutable Objects
In-place ModificationAllowed (append, item assignment)Raises TypeError; creates a new object instead
HashabilityUnhashable by default (cannot be dict keys)Hashable if all nested elements are immutable
Default Argument TrapUnsafe as function defaults (def f(x=[]):)Safe as function defaults (def f(x=None):)
Thread / Pipeline SafetyRequires defensive copying (.copy() / deepcopy)Inherently safe to share across functions

Coding Conventions (PEP 8, Type Hints, and Docstrings)

PEP 8 is the official style guide for Python code. Adhering to PEP 8 keeps scientific notebooks, reusable modules, and production packages uniform and readable.

Definition

PEP 8, Type Hints, and Docstring Standards

Python
from typing import Iterable


def mean_center(values: Iterable[float]) -> list[float]:
    """Subtract the arithmetic mean from each numeric value.

    Args:
        values: An iterable of numeric observations.

    Returns:
        A list of centered floating-point values.

    Raises:
        ValueError: If values is empty.
    """
    data = list(values)
    if not data:
        raise ValueError("Cannot mean-center an empty sequence.")
    avg = sum(data) / len(data)
    return [x - avg for x in data]

1.2 Advanced data structures: Collections, iterators, generators, and decorators

Standard lists and dictionaries handle general tasks, but large-scale data processing demands specialized containers, lazy iteration, and clean cross-cutting wrappers.

The collections Module

Python's built-in collections module supplies specialized container datatypes tuned for frequency counting, grouping, sliding windows, and lightweight records:

Key Container Classes in collections
ClassUnderlying IdeaPrimary Data Science Use Case
Counterdict subclass mapping hashable items to countsWord frequencies, class label counts, .most_common(k)
defaultdictdict with automatic factory for missing keysGrouping records by category without manual key checks
dequeDoubly-linked list with O(1) endsSliding-window moving averages (deque(maxlen=k)), BFS queues
namedtupleImmutable tuple with named attribute fieldsLightweight read-only data rows without class boilerplate
ChainMapLayered view over multiple dictionariesMerging CLI flags, environment settings, and default configs
Python
from collections import Counter, defaultdict, deque, namedtuple

# 1. Frequency counting
labels = ["spam", "ham", "ham", "spam", "ham", "ham"]
counts = Counter(labels)               # Counter({'ham': 4, 'spam': 2})

# 2. Grouping by key
groups = defaultdict(list)
for dept, score in [("CS", 88), ("ECE", 76), ("CS", 92)]:
    groups[dept].append(score)         # {'CS': [88, 92], 'ECE': [76]}

# 3. Fixed-size sliding window
window = deque(maxlen=3)
for val in [10, 20, 30, 40]:
    window.append(val)                 # ends as deque([20, 30, 40], maxlen=3)

# 4. Lightweight record
Record = namedtuple("Record", ["id", "feature", "label"])
row = Record(101, 0.85, 1)

Iterators and the Iterator Protocol

Definition

Iterable vs. Iterator Protocol

Generators (yield and Generator Expressions)

A generator is a concise way to build an iterator without writing __iter__ and __next__ manually.

AspectList ComprehensionGenerator Expression
Evaluation StrategyEager (computes all items immediately)Lazy (computes one item on demand)
Memory UsageO(N) — stores entire list in RAMO(1) — holds only current state frame
ReusabilityReusable; supports indexing lst[i] and len()Single-pass exhaustion; no indexing or len()
Best Used ForSmall/medium data needing random access or sortingLarge streams, file pipelines, sum() / any() reductions

Decorators

A decorator is a higher-order callable that takes a function (or class), wraps it with additional behavior, and returns a new callable—without modifying the wrapped function's source code. Decorators rely on Python's first-class functions and closures, and use @functools.wraps(func) so the wrapper preserves __name__, __doc__, and type metadata.

Common data science uses include execution timing, result memoization (@functools.lru_cache), input validation, and retry logic for flaky API requests.

Python
import functools
import time


def timer(func):
    """Measure and print execution time of the decorated function."""
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        t0 = time.perf_counter()
        result = func(*args, **kwargs)
        elapsed_ms = (time.perf_counter() - t0) * 1000.0
        print(f"{func.__name__} took {elapsed_ms:.2f} ms")
        return result
    return wrapper


@timer
def sum_squares(n: int) -> int:
    return sum(i * i for i in range(1, n + 1))

1.3 Functions and lambda expressions

Functions decompose complex analytical workflows into testable, reusable units. Python supports rich parameter-passing semantics alongside compact anonymous functions for functional transformations.

Function Arguments and Scope (LEGB Rule)

  1. Argument Kinds:
    • Positional and Keyword arguments: func(10, scale=2.0).
    • Default arguments: def clip(x, low=0.0, high=1.0):.
    • Arbitrary positional (*args): Collects extra positional arguments into a tuple.
    • Arbitrary keyword (**kwargs): Collects extra keyword arguments into a dict.
    • Positional-only (/) and Keyword-only (*): Enforced in APIs like def train(X, y, /, *, epochs=10, lr=0.01): so callers cannot pass hyperparameters positionally by accident.
  2. Variable Scope (LEGB): Python resolves unqualified names in order: Local → Enclosing (nested functions / closures) → Global (module) → Built-in (len, sum). Use nonlocal to rebind an enclosing variable and avoid global in data pipelines.

Lambda Expressions and Functional Primitives

A lambda expression defines a small, single-expression anonymous function:

lambda arguments: single_expression

Lambdas cannot contain statements (return, for, try, assignments) and return the value of single_expression automatically. They are most idiomatic as short inline callbacks (key= functions) and in functional pipelines (map, filter, functools.reduce):

Functional Utilities with Lambda Expressions
FunctionSignature / BehaviorTypical Example
sorted(seq, key=...)Returns new sorted list using key functionsorted(rows, key=lambda r: r["score"], reverse=True)
map(fn, iter)Lazy iterator applying fn to each itemlist(map(lambda x: x / 100.0, [25, 50, 75]))
filter(pred, iter)Lazy iterator keeping items where pred(x) is truthylist(filter(lambda x: x > 0, [-3, 0, 4, 9]))
functools.reduce(fn, iter)Folds items left-to-right into one cumulative valuereduce(lambda a, b: a * b, [2, 3, 4]) → 24

1.4 Object-Oriented Programming for data science applications

Modern Python ML and data libraries (scikit-learn, PyTorch, pandas) organize datasets, preprocessors, and estimators as classes. Understanding OOP enables you to write custom transformers, models, and modular data containers.

Classes, Instances, and Special (Dunder) Methods

Definition

Core OOP Concepts in Data Science

Python
from abc import ABC, abstractmethod
from dataclasses import dataclass


@dataclass(frozen=True)
class ScalerConfig:
    with_mean: bool = True
    eps: float = 1e-8


class BaseTransformer(ABC):
    @abstractmethod
    def fit(self, data: list[float]) -> "BaseTransformer":
        pass

    @abstractmethod
    def transform(self, data: list[float]) -> list[float]:
        pass

    def fit_transform(self, data: list[float]) -> list[float]:
        return self.fit(data).transform(data)


class ZScoreScaler(BaseTransformer):
    def __init__(self, config: ScalerConfig | None = None):
        self.config = config or ScalerConfig()
        self.mean_: float = 0.0
        self.std_: float = 1.0

    def fit(self, data: list[float]) -> "ZScoreScaler":
        if not data:
            raise ValueError("Cannot fit ZScoreScaler on empty data.")
        self.mean_ = sum(data) / len(data)
        var = sum((x - self.mean_) ** 2 for x in data) / len(data)
        self.std_ = (var ** 0.5) or self.config.eps
        return self

    def transform(self, data: list[float]) -> list[float]:
        shift = self.mean_ if self.config.with_mean else 0.0
        return [(x - shift) / self.std_ for x in data]

    def __repr__(self) -> str:
        return f"ZScoreScaler(mean_={self.mean_:.3f}, std_={self.std_:.3f})"

Notice the scikit-learn convention: attributes learned during .fit() carry a trailing underscore (mean_, std_), and .fit() returns self to allow method chaining (scaler.fit(X_train).transform(X_test)).


1.5 Exception handling, debugging, and logging

Unattended data ingestion and training jobs encounter corrupt rows, network timeouts, missing keys, and division by zero. Graceful error handling, structured logging, and systematic debugging separate brittle scripts from production pipelines.

Exception Handling (try / except / else / finally)

Definition

Exception Handling Flow and Custom Exceptions

Python
class SchemaValidationError(ValueError):
    """Raised when an input record violates expected column schema."""


def parse_positive_float(raw: str) -> float:
    try:
        val = float(raw)
    except ValueError as exc:
        raise SchemaValidationError(f"Non-numeric value: {raw!r}") from exc
    if val <= 0:
        raise SchemaValidationError(f"Expected positive value, got {val}")
    return val

Debugging Techniques

  1. Traceback Reading: Read Python tracebacks bottom-up: the last line gives the exception type and message, while the stack frame just above it points to the failing line in your code.
  2. Interactive Debugger (pdb / breakpoint()): Inserting breakpoint() (Python 3.7+) pauses execution at that line and opens pdb, where commands n (next line), s (step into), c (continue), p expr (print variable), and l (list source) inspect live state.
  3. Assertions (assert): Use assert len(X) == len(y), "Mismatched samples" for internal sanity checks during development—never for validating untrusted user input, because python -O strips assert statements.

Structured Logging with logging

Replacing print() calls with Python's standard logging module provides severity filtering, timestamps, module names, and simultaneous output to console and rotating log files:

Standard Python Logging Levels
LevelNumeric ValueWhen to Use in Data Science Pipelines
DEBUG10Detailed shapes, intermediate tensors, row-level diagnostics
INFO20Pipeline milestones (rows loaded, epoch accuracy, job complete)
WARNING30Recoverable anomalies (missing optional column, fallback used)
ERROR40Failed record batch, skipped file, API call failure
CRITICAL50Fatal configuration or storage failure halting the entire pipeline
Python
import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
)
logger = logging.getLogger("etl.cleaner")
logger.info("Loaded %d valid rows from batch %s", 1250, "2026-11-18")

1.6 Working with modules and packages

Modularizing code into modules and packages prevents monolithic notebooks, enables unit testing, and supports reproducible environments across machines.

Definition

Modules, Packages, and Import Resolution

Standard Data Science Package LayoutText
my_ds_project/
|-- pyproject.toml          # Build metadata and pinned dependencies
|-- README.md
|-- src/
|   `-- pipeline/
|       |-- __init__.py     # Re-exports public classes/functions
|       |-- ingest.py       # Data loading module
|       |-- transform.py    # Feature scaling & cleaning
|       `-- models/
|           |-- __init__.py
|           `-- trainer.py  # Subpackage module
`-- tests/
    `-- test_transform.py   # Unit tests for transform module

Use absolute imports (from pipeline.transform import ZScoreScaler) across packages for clarity, and keep relative imports (from .ingest import load_csv) strictly within the same package.


Worked Examples

Example 1.1

1.1

Write a Python pipeline using a generator expression and a custom decorator to compute the sum of squares of even numbers in [1, 2, 3, 4, 5, 6], and trace the exact output and memory behavior. Also trace the output of a deque(maxlen=2) sliding average on [5, 10, 15].

Building the Decorator and Generator Pipeline: Define a @log_call decorator using functools.wraps to preserve metadata, and pass a lazy generator expression (x ** 2 for x in nums if x % 2 == 0) to sum():

Python
import functools


def log_call(fn):
    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        res = fn(*args, **kwargs)
        print(f"{fn.__name__} -> {res}")
        return res
    return wrapper


@log_call
def even_square_sum(nums: list[int]) -> int:
    return sum(x ** 2 for x in nums if x % 2 == 0)

Tracing Execution and Sliding Window:

  • Filtered even numbers from [1, 2, 3, 4, 5, 6] are 2, 4, 6.
  • Generator yields 2**2 = 4, then 4**2 = 16, then 6**2 = 36 one at a time in O(1) auxiliary space.
  • sum(...) accumulates 4 + 16 + 36 = 56, and @log_call prints even_square_sum -> 56.
  • For window = deque(maxlen=2) over [5, 10, 15]:
    • After 5: [5] (avg = 5.0)
    • After 10: [5, 10] (avg = (5 + 10) / 2 = 7.5)
    • After 15: 5 is evicted automatically → [10, 15] (avg = (10 + 15) / 2 = 12.5)
Answer:

even_square_sum([1, 2, 3, 4, 5, 6]) prints even_square_sum -> 56 and returns 56 in O(1) extra memory. The deque(maxlen=2) sliding averages across [5, 10, 15] are 5.0, 7.5, and 12.5.

Example 1.2

1.2

Implement a MinMaxNormalizer class with fit() and transform() methods following scikit-learn OOP conventions, and compute the normalized output for training data [10, 20, 30] and test value [25].

Implementing the OOP Transformer: Store min_ and max_ as trailing-underscore attributes fitted only on training data, guard against zero range (max_ == min_), and handle unseen test data with the stored bounds:

Python
class MinMaxNormalizer:
    def __init__(self) -> None:
        self.min_: float = 0.0
        self.max_: float = 1.0

    def fit(self, data: list[float]) -> "MinMaxNormalizer":
        if not data:
            raise ValueError("Training data cannot be empty.")
        self.min_ = min(data)
        self.max_ = max(data)
        return self

    def transform(self, data: list[float]) -> list[float]:
        span = self.max_ - self.min_
        if span == 0:
            return [0.0 for _ in data]
        return [(x - self.min_) / span for x in data]

Evaluating on [10, 20, 30] and Test [25]:

  • Fitting on [10, 20, 30] sets min_ = 10, max_ = 30, so span = 30 - 10 = 20.
  • Transforming training data:
    • (10 - 10) / 20 = 0.0
    • (20 - 10) / 20 = 0.5
    • (30 - 10) / 20 = 1.0
  • Transforming test value [25] using the training min_ and max_ prevents data leakage: (25 - 10) / 20 = 0.75.
Answer:

fit([10, 20, 30]) sets min_ = 10 and max_ = 30. transform([10, 20, 30]) returns [0.0, 0.5, 1.0], and transform([25]) returns [0.75].

Example 1.3

1.3

How should a student answer the exam question: "Explain Python's iterator protocol, contrast iterators with generators, and show how try / except / else / finally and logging work together in a resilient data loader"?

Structuring the Protocol & Comparison:

  1. State the iterator protocol: __iter__() returns the iterator object (self), and __next__() returns the next value or raises StopIteration.
  2. Contrast class-based iterators (explicit state class with __iter__ + __next__) with generators (functions using yield or (expr for x in seq) that compile the state machine automatically and pause between yields).
  3. Include a 3-line code snippet showing next() and StopIteration or yield.

Integrating Exception Handling & Logging: Write a concise loader pattern showing all four try / except / else / finally clauses and logging levels (INFO on success, ERROR on missing/corrupt file, finally for guaranteed cleanup), and explain why bare except: must be avoided.

Answer:

A full-mark answer defines __iter__() and __next__() (StopIteration), contrasts manual iterator classes against yield generators in memory and syntax, and illustrates a try / except FileNotFoundError / else / finally file loader paired with logging.getLogger() at INFO/ERROR levels.


Exercises

Practice Exercises — Advanced Python Concepts and Best Practices

Question 1

Explain PEP 8 naming conventions, type hints, and the mutable default argument trap in Python. Show how to fix a buggy function def add_item(item, bucket=[]):.

PEP 8 Conventions and Type Hints:

  • Functions, methods, and variables use snake_case; classes use PascalCase; constants use UPPER_SNAKE_CASE; indentation uses 4 spaces per level.
  • Type hints (PEP 484) annotate argument and return types (e.g., def add_item(item: str, bucket: list[str] | None = None) -> list[str]:) for static verification and clearer APIs.

Fixing the Mutable Default Argument Trap: Default expressions are evaluated once when def executes, so bucket=[] creates a single shared list across every call omitting bucket. Replace [] with None and allocate inside the body:

Python
def add_item(item: str, bucket: list[str] | None = None) -> list[str]:
    if bucket is None:
        bucket = []
    bucket.append(item)
    return bucket
Answer:

PEP 8 standardizes snake_case functions/variables, PascalCase classes, UPPER_SNAKE_CASE constants, 4-space indentation, and grouped imports; type hints specify expected types statically. Because default arguments are evaluated once at function definition time, replace bucket=[] with bucket=None and initialize bucket = [] inside the function body.

Question 2

Given a list of product records [("pen", 20), ("book", 100), ("bag", 80)], write Python code using lambda with sorted() and filter() to select items costing at least 50, sort them by price descending, and compute a 25% discount on the highest-priced item.

Filtering and Sorting with Lambda:

Python
items = [("pen", 20), ("book", 100), ("bag", 80)]
expensive = list(filter(lambda item: item[1] >= 50, items))
ranked = sorted(expensive, key=lambda item: item[1], reverse=True)
  • filter keeps ("book", 100) and ("bag", 80) ("pen" at 20 is dropped).
  • sorted(..., reverse=True) orders them as [("book", 100), ("bag", 80)].

Computing the 25% Discount on the Top Item: The highest-priced item is ranked[0] = ("book", 100).

  • Discount amount = 100 * 0.25 = 25.0.
  • Discounted price = 100 - 25.0 = 75.0.
Answer:

Filtered and sorted list: [("book", 100), ("bag", 80)]. A 25% discount on "book" (100) subtracts 25.0, giving a discounted price of 75.0.

Question 3

Differentiate between an iterator and a generator in Python. Write a generator function squares_up_to(n) that yields 1^2, 2^2, ..., n^2, and calculate the sum yielded for n = 5.

Iterator vs. Generator:

  • An iterator is any object implementing __iter__() and __next__(), often requiring boilerplate state attributes in a custom class.
  • A generator is a special iterator created either by a function using yield or by a generator expression (expr for x in seq). Python automatically generates __iter__(), __next__(), state preservation, and StopIteration on return.

Generator Implementation and Trace for n = 5:

Python
def squares_up_to(n: int):
    for i in range(1, n + 1):
        yield i * i

Yielded values for n = 5: 1, 4, 9, 16, 25. Total sum: 1 + 4 + 9 + 16 + 25 = 55.

Answer:

Every generator is an iterator, but generators automate __iter__/__next__ state management via yield (or (expr for ...)). For n = 5, squares_up_to(5) yields 1, 4, 9, 16, 25, whose sum is 55.

Question 4

What is a decorator in Python, and why is @functools.wraps(func) important when writing one? Write a @require_non_empty decorator that raises ValueError if the first positional argument is empty.

Decorator Definition and functools.wraps: A decorator is a callable that accepts a function, wraps it in an inner closure adding pre/post behavior, and returns the wrapper. Without @functools.wraps(func), the wrapper overwrites func.__name__, func.__doc__, and __wrapped__, breaking introspection, logging, and documentation tools.

Implementing @require_non_empty:

Python
import functools


def require_non_empty(func):
    @functools.wraps(func)
    def wrapper(seq, *args, **kwargs):
        if len(seq) == 0:
            raise ValueError(f"{func.__name__} requires a non-empty sequence.")
        return func(seq, *args, **kwargs)
    return wrapper
Answer:

A decorator wraps a callable to extend its behavior without modifying its source code, and @functools.wraps(func) preserves the original function's __name__, __doc__, and signature metadata. The @require_non_empty wrapper checks len(seq) == 0, raises ValueError, and otherwise forwards arguments to func.

Question 5

Explain how custom exceptions, try / except / else / finally, logging, and Python packages (__init__.py, __name__ == "__main__") work together to make a data science project production-ready.

Resilient Execution with Exceptions and Logging:

  • Custom exceptions (inheriting from Exception or ValueError) give callers meaningful domain errors (e.g., MissingColumnError) instead of generic failures.
  • try / except / else / finally isolates risky I/O, catches specific exceptions, runs normal post-load logic in else, and guarantees resource release in finally.
  • logging records timestamped INFO, WARNING, and ERROR events to files and consoles without relying on unfilterable print() calls.

Modular Structure with Packages and Guards:

  • Grouping code into a package (__init__.py + focused submodules like ingest.py, features.py) enables clean imports and isolated unit testing.
  • Placing script entrypoints behind if __name__ == "__main__": allows modules to be imported safely by test suites and downstream pipelines without triggering execution side effects.
Answer:

Custom exceptions provide domain-specific error semantics; try / except / else / finally guarantees safe handling and cleanup; logging records severity-tagged audit trails; and packages (__init__.py) combined with if __name__ == "__main__": separate reusable library code from runnable scripts.

Data Sources and APIs →