PadhaikoAll subjects

Probability, Random Variables & Distributions

Foundation of Data Science

Open in the Padhaiko app

2.4 Introduction to Probability and Random Variable

Data science investigates real-world phenomena governed by randomness, observational noise, and incomplete information. Whether predicting customer churn, classifying medical diagnoses, filtering spam emails, or modeling sensor errors, deterministic models fail to capture underlying variability. Probability theory provides the formal mathematical framework for quantifying uncertainty, reasoning about random outcomes, and updating beliefs in the presence of empirical evidence.

Foundational Terminologies

Before establishing formal mathematical probability, several foundational concepts must be clearly defined:

  1. Random Experiment: A process or procedure whose exact outcome cannot be predicted with certainty beforehand, but whose set of all possible outcomes is known (e.g., tossing a coin, rolling a die, observing the daily click-through rate of an online advertisement).
  2. Outcome (): A single, specific result generated by one execution of a random experiment (e.g., getting "Heads", rolling a , or observing a fraud transaction).
  3. Sample Space ( or ): The set containing all possible mutually exclusive outcomes of a random experiment.
    • For a single coin flip: .
    • For rolling a standard six-sided die: .
    • For flipping two fair coins simultaneously: .
  4. Event (): Any subset of the sample space (). An event occurs if the actual outcome of the experiment belongs to .
    • A simple (elementary) event consists of a single outcome (e.g., rolling a , ).
    • A compound event combines multiple outcomes (e.g., rolling an even number on a die, ).

Definitions of Probability

Historically and operationally, probability is interpreted in three complementary ways:

where is the number of times event occurred in experimental repetitions.

AspectFrequentist ProbabilityBayesian Probability
InterpretationLong-run relative frequency in repeated experimentsDegree of belief or certainty given available evidence
Parameters ()Fixed, unknown constant valuesRandom variables governed by probability distributions
Data ()Random sample drawn from populationObserved, fixed evidence
Prior KnowledgeExplicitly excluded; relies solely on observed dataFormally incorporated as a prior distribution
OutputPoint estimates, confidence intervals, -valuesFull posterior probability distribution

Axiomatic Probability (Kolmogorov Foundations)

Modern mathematical probability is grounded in the three axioms established by Andrey Kolmogorov in 1933:

Definition

Axiomatic Probability and Kolmogorov Foundations

Let be the sample space of a random experiment, and let be the family of all observable events. A probability measure is a real-valued function satisfying three foundational axioms:

  1. Axiom 1 (Non-Negativity): For every event :

Probabilities are never negative.

  1. Axiom 2 (Normalization / Certainty): The probability of the entire sample space is unity:

The certain event occurs with probability .

  1. Axiom 3 (Countable Additivity): For any sequence of mutually exclusive (pairwise disjoint) events (where for all ):

Fundamental Corollaries Derived from Kolmogorov Axioms

From the three axioms, all standard rules of probability follow directly:

  1. Probability of the Impossible Event:
  1. The Complement Rule: For any event , let (or ) denote its complementary event (the event that does not occur). Since and :
  1. Monotonicity: If , then . Consequently, for any event :
  1. General Addition Rule (Inclusion-Exclusion): For any two arbitrary events and (not necessarily mutually exclusive):

If and are mutually exclusive, , reducing to Axiom 3: .

Relationships Between Events

Mutually Exclusive vs. Independent Events

A frequent point of confusion in probabilistic modeling is the distinction between mutually exclusive events and statistically independent events:

AspectMutually Exclusive EventsIndependent Events
Mathematical Definition
Physical MeaningOccurrence of completely prevents occurrence of Occurrence of provides zero information about
Can they occur together?Never; impossible simultaneouslyYes; can occur together with joint probability
Conditional Probability (assuming )
Classic ExampleRolling a and rolling a on a single dieGetting Heads on Coin 1 and getting Heads on Coin 2

[!IMPORTANT] Two non-empty events with non-zero probabilities () can never be simultaneously mutually exclusive and independent. If they are mutually exclusive, . But if they are independent, , an obvious mathematical contradiction. In fact, mutually exclusive events are extremely dependent: knowing that occurred guarantees that did not occur!

Conditional Probability and Joint Probability

In real-world data science, events rarely occur in isolation. We constantly evaluate the probability of an outcome given partial observations or contextual features.

Conditional Probability

Let and be two events in sample space with . The conditional probability of event given that event has already occurred is:

Similarly, the conditional probability of given (with ) is:

Joint Probability and the Multiplication Rule

Rearranging the conditional probability formulas yields the General Multiplication Rule for the joint probability :

For three events , the Chain Rule of Probability expands joint probabilities into a sequential cascade of conditional factors:

This decomposition is the foundational architectural pillar of probabilistic graphical models, Markov chains, and autoregressive language models (such as modern transformers predicting the next token conditioned on all preceding tokens).

The Law of Total Probability and Bayes' Theorem

Let the collection of events form a partition of the sample space , meaning:

  1. They are mutually exclusive: for all .
  2. They are collectively exhaustive: .
  3. Each partition has non-zero probability: .

Law of Total Probability

For any arbitrary event , the total probability of can be decomposed across the partitions:

Theorem

Bayes' Theorem and Total Probability

Let be a partition of the sample space , and let be an observed event with . The conditional (posterior) probability of hypothesis given evidence is:

Terminology in Bayesian Machine Learning

Random Variables

In data science, we rarely work directly with textual outcome names like "Heads" or "Patient survived". Instead, we map experimental outcomes to real numbers so they can be processed by mathematical algorithms.

Definition

Discrete and Continuous Random Variables

A random variable is a measurable mathematical function that maps each outcome in the sample space to a unique real number:

Random variables are categorized based on the cardinality of their range:

  1. Discrete Random Variable: Takes on a finite or countably infinite set of distinct numerical values (e.g., ).
    • Characterized by a Probability Mass Function (PMF): .
    • Conditions: for all , and .
    • The probability of a specific point value is generally non-zero ().
  2. Continuous Random Variable: Takes on any value within an uncountable real interval (e.g., height, temperature, transaction latency).
    • Characterized by a Probability Density Function (PDF): .
    • Conditions: for all , and .
    • Crucial distinction: The probability of a continuous random variable taking on any single exact point value is identically zero:
Discrete vs. Continuous Random Variables
CharacteristicDiscrete Random VariableContinuous Random Variable
Possible ValuesFinite or countably infinite ()Uncountable continuum / real intervals ()
Probability FunctionProbability Mass Function (PMF): Probability Density Function (PDF):
NormalizationSummation: Integration:
Point ProbabilityCan be non-zero ()Strictly zero ()
Interval Probability
Cumulative Dist. (CDF)
Expected Value
Real-World ExamplesNumber of website clicks, defective items, word countCustomer wait time, student height, transaction amount

Expectation and Variance of a Random Variable


2.5 Probability Distributions: Normal, Bernoulli, Binomial, Poisson

A probability distribution is a mathematical formula, rule, or table that describes how probabilities are distributed across the potential values of a random variable. In data science, standard parametric distributions serve as generative models for observed data, theoretical baselines for hypothesis testing, and likelihood functions in statistical learning.

Bernoulli Distribution

Definition

Bernoulli and Binomial Distributions

The Bernoulli Distribution is the discrete probability distribution of a random variable representing a single trial with exactly two mutually exclusive outcomes:

Probability Mass Function (PMF):

Moments:

Role in Data Science: The Bernoulli distribution forms the probabilistic foundation of binary classification. In logistic regression, the model outputs , and the binary cross-entropy loss function is derived directly from the negative log-likelihood of Bernoulli-distributed labels:

Binomial Distribution

The Binomial Distribution models the total number of successes observed in a sequence of independent and identically distributed (i.i.d.) Bernoulli trials, each having a constant probability of success .

Structural Assumptions

  1. The number of trials is fixed in advance.
  2. Each trial has exactly two outcomes: success () or failure ().
  3. The probability of success remains constant across all trials.
  4. Trials are mutually independent: the outcome of any trial does not influence any other.

Probability Mass Function (PMF):

where the binomial coefficient counts the number of distinct combinations of choosing successes out of trials:

Moments: Since a Binomial random variable is the sum of independent Bernoulli random variables where :

Role in Data Science: Used in digital marketing A/B testing (measuring the number of conversions out of site visitors), manufacturing quality assurance (counting defective circuit boards out of a batch of ), and clinical drug trial efficacy testing.

Poisson Distribution

Definition

Poisson Distribution

The Poisson Distribution is a discrete probability distribution that models the count of rare events occurring within a fixed continuous interval of time, distance, area, or volume, given that these events occur at a known constant average rate and independently of the time since the last event.

Probability Mass Function (PMF):

where:

Moments and Equidispersion:

[!NOTE] The defining mathematical hallmark of the Poisson distribution is equidispersion: its theoretical variance is exactly equal to its theoretical mean (). In applied exploratory data analysis, if empirical sample variance significantly exceeds the sample mean, the data exhibits overdispersion, suggesting a Negative Binomial model rather than a standard Poisson.

Poisson as a Limiting Form of the Binomial Distribution

When the number of trials and the probability of success such that the expected value remains finite and constant, the Binomial distribution converges asymptotically to the Poisson distribution:

Role in Data Science: Used in server traffic engineering (modeling the number of HTTP requests hitting an API server per second), cybersecurity (detecting DDoS attacks and anomalous login attempts), telecom queueing systems, and insurance catastrophe claim frequency modeling.

Normal (Gaussian) Distribution

Definition

Normal (Gaussian) Distribution

The Normal (or Gaussian) Distribution is a continuous probability distribution whose probability density function is symmetric and bell-shaped around its central mean. It is completely parameterized by two moments: its mean and its variance (written ).

Probability Density Function (PDF):

where:

Key Geometric and Mathematical Properties

  1. Unimodal and Symmetrical: The bell curve is perfectly symmetric around . The left and right halves are mirror images.
  2. Central Tendency Alignment: For any normal distribution:
  1. Inflection Points: The curve transitions from concave downward to concave upward at exactly one standard deviation on either side of the mean ( and ).
  2. Asymptotic Tails: The tails extend infinitely in both directions along the horizontal axis (), approaching zero asymptotically but never touching it.
  3. Total Probability Area: The total area enclosed under the probability density curve is exactly :

The Standard Normal Distribution ()

Any normal random variable can be transformed into the Standard Normal Distribution with mean and variance via the standardizing -score transformation:

Its probability density function simplifies to:

The Empirical Rule ( Rule)

For any normally distributed population:

Values lying more than standard deviations from the mean () are exceptionally rare () and are frequently flagged as statistical outliers in data preprocessing pipelines.

Comprehensive Reference Table of Fundamental Probability Distributions
DistributionTypeParametersPMF / PDF FormulaMean ()Variance ()Primary Data Science Application
BernoulliDiscreteBinary classification targets, logistic regression
BinomialDiscreteA/B testing conversion counts, batch defects
PoissonDiscreteServer request arrival rates, fraud call frequency
NormalContinuousCLT asymptotics, OLS residual noise, -score scaling

Worked Examples

Example 2.5

2.5

A market research firm models customer habits using Bayes' Theorem.

Calculate the posterior probability that an individual who is observed buying a pack of cigarettes is a smoker ().

Identifying given probabilistic components:

  • Prior probability of hypothesis:
  • Likelihood of evidence given hypothesis:
  • Marginal probability of evidence:

Applying Bayes' Theorem: Substitute the values into Bayes' formula:

Answer:

Observing the purchase of cigarettes updates the probability that the individual is a smoker from a baseline prior of to a posterior probability of .

Example 2.6

2.6

The height of students in a university engineering class is modeled as a normally distributed continuous random variable . Calculate the probability that a randomly selected student has a height between and .

Standardizing the interval endpoints via -scores: The parameters are mean and standard deviation . For lower limit :

For upper limit :

Evaluating the probability over the standardized range: The interval corresponds to :

Using standard normal distribution tables where and :

This confirms the first tier of the Empirical Rule ( within ).

Answer:

Approximately of students in the class have heights between and .

Example 2.7

2.7

An electronics manufacturer produces microchips where the probability of a chip passing quality control (defect-free) is . In a random sample of microchips, calculate:

  1. The probability that exactly chips pass quality control ().
  2. The probability that all chips pass quality control ().

Formulating the Binomial PMF: With trials and success probability (failure probability ):

Part 1: Exactly defect-free chips (): Compute the binomial coefficient:

Compute probability:

Part 2: All defect-free chips ():

Answer:
Example 2.8

2.8

A customer support call center receives incoming calls at an average rate of calls per hour. Assuming call arrivals follow a Poisson distribution, determine:

  1. The probability that the call center receives exactly calls in a given hour.
  2. The probability that the call center receives zero calls in a given hour.

Setting up the Poisson PMF: For rate parameter :

Using .

Part 1: Probability of receiving exactly calls ():

Part 2: Probability of receiving zero calls ():

Answer:

Exercises

Exercises — Probability, Random Variables & Distributions

Question 1

Define mathematical probability and state its three foundational Kolmogorov axioms. From these axioms, derive the complement rule and the addition rule for non-mutually exclusive events.

Formal definition of probability: Probability is a mathematical function that assigns to each event in sample space a real number quantifying its likelihood of occurrence.

The three Kolmogorov axioms:

  1. Non-negativity: For any event , .
  2. Normalization (Certainty): For the entire sample space , .
  3. Countable Additivity: For pairwise mutually exclusive events ( for ):

Derivation of the complement rule: Any event and its complement partition the sample space :

By Axiom 3 (Additivity for disjoint events):

By Axiom 2 (Normalization):

Derivation of the general addition rule: Express as the union of two mutually disjoint sets:

where . Applying Axiom 3:

Similarly, can be partitioned into two disjoint subsets:

Rearranging gives . Substituting back:

Answer:

Probability is a normalized measure on sample spaces satisfying: (1) , (2) , and (3) for disjoint events. These axioms yield the complement rule and the general addition rule .

Question 2

Differentiate between discrete and continuous random variables with mathematical definitions and practical engineering examples. Present a comprehensive comparative table.

Discrete random variables: A random variable is discrete if its range of possible values is finite or countably infinite:

Its distribution is specified by a Probability Mass Function (PMF) satisfying and . The probability of an exact individual value can be non-zero. Example: counting the number of defective packets in an IoT transmission batch.

Continuous random variables: A random variable is continuous if it can take on any real value within an uncountable interval . Its distribution is specified by a Probability Density Function (PDF) satisfying and .

Because the real continuum contains uncountably infinite points, the probability of obtaining any exact point value is zero: . Probabilities are non-zero only over finite intervals: . Example: measuring battery discharge life or server response latency.

Comparative analysis table:

Comparison DimensionDiscrete Random VariableContinuous Random Variable
Domain RangeCountable set of isolated numbersContinuous real interval
Probability FunctionProbability Mass Function (PMF), Probability Density Function (PDF),
NormalizationSummation: Integration:
Probability at PointMay be strictly positive: Always identically zero:
Interval Probability
Engineering ExamplesDaily website signups, bit transmission errorsVehicle speed, temperature, sensor output voltage
Answer:

Discrete random variables have countable outcome values characterized by PMFs with non-zero point probabilities; continuous random variables span uncountable real intervals characterized by PDFs where single-point probability is zero and probabilities exist only over intervals.

Question 3

Derive the mathematical formulation of conditional probability and the joint multiplication rule. From these principles, derive Bayes' Theorem for updating probabilistic beliefs.

Definition of conditional probability: The conditional probability of event given that event has occurred () is defined as the ratio of the joint probability to the conditioning probability:

Similarly, the conditional probability of given () is:

The joint multiplication rule: Rearranging equations (i) and (ii) yields two equivalent expressions for the joint probability :

Equating the right-hand sides:

Deriving Bayes' Formula: Dividing both sides by yields the fundamental Bayes' Formula:

Expansion via Law of Total Probability: If the sample space is partitioned into mutually exclusive, exhaustive hypotheses , then by the Law of Total Probability:

Substituting into the denominator gives Bayes' Theorem in its complete form:

Answer:

Conditional probability implies , which yields Bayes' Theorem: .

Question 4

State the mathematical difference between independent events and mutually exclusive events. Use algebraic equations and concrete examples to explain why two events with positive probabilities cannot be both mutually exclusive and independent.

Definitions and mathematical criteria:

  • Mutually Exclusive Events: Events and cannot happen simultaneously. Their intersection is empty:

Example: Drawing a single card from a deck that is simultaneously an Ace and a King ().

  • Independent Events: Occurrence of event does not alter the likelihood of event :

Example: Flipping a coin twice where the first toss is Heads and the second toss is Tails ().

Rigorous proof of incompatibility: Let and be two events with strictly positive probabilities ( and ).

  1. Suppose and are independent. Then:

Since their joint probability is strictly positive, , so they cannot be mutually exclusive.

  1. Conversely, suppose and are mutually exclusive. Then:

For independence to hold, we would require:

which contradicts the premise that both and .

Answer:

Mutually exclusive events cannot occur together (), whereas independent events satisfy . Therefore, two events with non-zero probabilities can never be both mutually exclusive and statistically independent.

Question 5

Write down the Probability Mass Function (PMF), expected value, and variance for both the Bernoulli distribution and the Binomial distribution. Explain how the Binomial distribution is related to the Bernoulli distribution.

Bernoulli Distribution: Models a single trial with binary outcomes: success () with probability and failure () with probability .

  • PMF:
  • Expected Value:
  • Variance:

Binomial Distribution: Models the total count of successes in independent and identically distributed Bernoulli trials:

  • PMF:

where .

  • Expected Value:
  • Variance:

Relationship between the two distributions: The Binomial distribution is the exact sum of independent, identically distributed Bernoulli trials:

Conversely, the Bernoulli distribution is a special case of the Binomial distribution with .

Answer:

Bernoulli PMF is with mean and variance . Binomial PMF is with mean and variance . The Binomial distribution represents the sum of independent, identically distributed Bernoulli trials.

Question 6

Explain the Normal distribution. State its Probability Density Function (PDF), specify its parameters, and indicate its mean, variance, and standard deviation. State and explain the Empirical Rule.

Mathematical description and PDF: The Normal (Gaussian) distribution is a continuous, unimodal, bell-shaped probability distribution parameterized by mean and variance :

Distribution parameters and moments:

  • Location parameter (Mean): (also equals median and mode due to symmetry).
  • Scale parameter (Variance): .
  • Standard Deviation: .
  • Total area under the density curve is .

The Empirical Rule ( Rule): For any normal distribution :

  1. of observations lie within standard deviation of the mean: .
  2. of observations lie within standard deviations of the mean: .
  3. of observations lie within standard deviations of the mean: .
Answer:

The Normal distribution has PDF , with mean , variance , and standard deviation . The Empirical Rule states that approximately , , and of data lie within , , and standard deviations of the mean, respectively.

Question 7

A marble is drawn at random from an opaque bag containing red marbles, blue marbles, and green marbles. What is the probability that the drawn marble is not green?

Determining the sample space and favorable outcomes: Total number of marbles in the bag:

Computing the probability of drawing a green marble: Let denote the event of selecting a green marble:

Applying the complement rule: Let denote the event of drawing a marble that is not green:

Alternatively, directly summing non-green favorable outcomes:

Answer:

The probability that the drawn marble is not green is ().

Question 8

A public health symposium is attended by medical professionals classified by gender and profession:

GenderDoctorsNursesTotal
Female
Male
Total

If one attendee is selected at random:

  1. Find the probability that a doctor is chosen ().
  2. Find the probability that a female attendee is chosen ().
  3. Find the probability that a female nurse is chosen ().
  4. Given that a female attendee is chosen, find the conditional probability that she is a nurse ().
  5. Determine whether the events "a nurse is chosen" and "a female attendee is chosen" are statistically independent.

Evaluating marginal and joint probabilities: Total attendees .

  1. Probability of selecting a doctor:
  1. Probability of selecting a female:
  1. Joint probability of selecting a female nurse:

Evaluating conditional probability : Using the conditional probability formula:

Testing statistical independence: For events and to be statistically independent, we must have , or equivalently . Calculate the marginal probability of being a nurse:

Comparing:

Because , the events are not independent (they are statistically dependent). Female attendees are more likely to be nurses () than attendees in the general population ().

Answer:

Events and are not independent because .

Question 9

A naive Bayesian email filter is designed to classify incoming emails into spam () and valid (). Statistical analysis indicates:

Determine:

  1. The total probability that an incoming email contains the keyword "free" ().
  2. The posterior probability that an email is spam given that it contains "free" ().
  3. The posterior probability that an email is valid given that it does not contain "free" ().

Formal definitions and complementary priors: Let:

  • : Event that an email is spam ().
  • : Event that an email is valid ().
  • : Event that an email contains the keyword "free".
  • : Event that an email does not contain "free".

Conditional probabilities:

Part 1: Total probability that an email contains "free": By the Law of Total Probability:

As a simplified fraction: .

Part 2: Probability of spam given that "free" is present (): Applying Bayes' Theorem:

The presence of "free" elevates the probability of spam from to .

Part 3: Probability of valid given that "free" is absent (): First compute the marginal probability of not containing "free":

Applying Bayes' Theorem:

Answer:
Question 10

Define a complementary event. A fair six-sided die is rolled. Let be the event "a number less than is obtained" and let be the event "a number of or more is obtained". Find and , and verify that and are complementary events. [Assignment, notebook p052]

Definition of complementary events: Two events and in a sample space are complementary events if they are mutually exclusive (, so they cannot occur at the same time) and collectively exhaustive (, so they make up the entire sample space). Consequently, their probabilities satisfy:

Listing the sample space and event outcomes: When rolling a standard six-sided die, the sample space has equally likely outcomes:

  • Event ("a number less than is obtained"):
  • Event ("a number of or more is obtained"):

Computing probabilities and verifying complementarity:

Notice that and . Summing their probabilities:

Thus, and are complementary events.

Answer:

Two events are complementary if they cannot occur simultaneously () and their union is the entire sample space (). Here , , and .

Question 11

Consider the following training dataset of observations with two categorical features and and a binary target class :

Obs123456789101112131415
AAAAABBBBBCCCCC
SMMSSSMMLLLMMLL
001100011111110

Using Bayes' theorem under the Naive Bayes conditional independence assumption, compute the prior probabilities and , the class-conditional likelihoods, and predict the class label for the test observation by maximizing . [Assignment, notebook p243]

Computing class prior probabilities : Out of the total observations:

  • Class occurs in observations ( times):
  • Class occurs in observations ( times):

Computing class-conditional feature likelihoods: Counting feature occurrences within each class subset:

  • For :
  • For :
  • For :

Evaluating unnormalized posterior scores for : By the Naive Bayes conditional independence assumption, :

  1. For class :
  1. For class :

Decision by maximum a posteriori (MAP) rule: Comparing the two posterior numerators:

Therefore, the classifier assigns the observation to class .

Answer:

Since , the predicted class label is .

Past exam questions for Foundation of Data Science

← Linear Algebra, Matrix Factorization & Gradient DescentDescriptive & Inferential Statistics, CLT & Sampling →