Probability, Random Variables & Distributions
Open in the Padhaiko app2.4 Introduction to Probability and Random Variable
Data science investigates real-world phenomena governed by randomness, observational noise, and incomplete information. Whether predicting customer churn, classifying medical diagnoses, filtering spam emails, or modeling sensor errors, deterministic models fail to capture underlying variability. Probability theory provides the formal mathematical framework for quantifying uncertainty, reasoning about random outcomes, and updating beliefs in the presence of empirical evidence.
Foundational Terminologies
Before establishing formal mathematical probability, several foundational concepts must be clearly defined:
- Random Experiment: A process or procedure whose exact outcome cannot be predicted with certainty beforehand, but whose set of all possible outcomes is known (e.g., tossing a coin, rolling a die, observing the daily click-through rate of an online advertisement).
- Outcome (): A single, specific result generated by one execution of a random experiment (e.g., getting "Heads", rolling a , or observing a fraud transaction).
- Sample Space ( or ): The set containing all possible mutually exclusive outcomes of a random experiment.
- For a single coin flip: .
- For rolling a standard six-sided die: .
- For flipping two fair coins simultaneously: .
- Event (): Any subset of the sample space (). An event occurs if the actual outcome of the experiment belongs to .
- A simple (elementary) event consists of a single outcome (e.g., rolling a , ).
- A compound event combines multiple outcomes (e.g., rolling an even number on a die, ).
Definitions of Probability
Historically and operationally, probability is interpreted in three complementary ways:
- Classical (Laplacian) Probability: When all outcomes in a finite sample space are equally likely, the probability of event containing favorable outcomes is:
- Frequentist (Empirical) Probability: Probability is defined as the long-run relative frequency of an event over an infinite sequence of identical, independent trials:
where is the number of times event occurred in experimental repetitions.
- Bayesian (Subjective) Probability: Probability quantifies an agent's degree of rational belief in a hypothesis or proposition, which is dynamically updated as new data and evidence are observed using Bayes' Theorem.
| Aspect | Frequentist Probability | Bayesian Probability |
|---|---|---|
| Interpretation | Long-run relative frequency in repeated experiments | Degree of belief or certainty given available evidence |
| Parameters () | Fixed, unknown constant values | Random variables governed by probability distributions |
| Data () | Random sample drawn from population | Observed, fixed evidence |
| Prior Knowledge | Explicitly excluded; relies solely on observed data | Formally incorporated as a prior distribution |
| Output | Point estimates, confidence intervals, -values | Full posterior probability distribution |
Axiomatic Probability (Kolmogorov Foundations)
Modern mathematical probability is grounded in the three axioms established by Andrey Kolmogorov in 1933:
Axiomatic Probability and Kolmogorov Foundations
Let be the sample space of a random experiment, and let be the family of all observable events. A probability measure is a real-valued function satisfying three foundational axioms:
- Axiom 1 (Non-Negativity): For every event :
Probabilities are never negative.
- Axiom 2 (Normalization / Certainty): The probability of the entire sample space is unity:
The certain event occurs with probability .
- Axiom 3 (Countable Additivity): For any sequence of mutually exclusive (pairwise disjoint) events (where for all ):
Fundamental Corollaries Derived from Kolmogorov Axioms
From the three axioms, all standard rules of probability follow directly:
- Probability of the Impossible Event:
- The Complement Rule: For any event , let (or ) denote its complementary event (the event that does not occur). Since and :
- Monotonicity: If , then . Consequently, for any event :
- General Addition Rule (Inclusion-Exclusion): For any two arbitrary events and (not necessarily mutually exclusive):
If and are mutually exclusive, , reducing to Axiom 3: .
Relationships Between Events
Mutually Exclusive vs. Independent Events
A frequent point of confusion in probabilistic modeling is the distinction between mutually exclusive events and statistically independent events:
| Aspect | Mutually Exclusive Events | Independent Events |
|---|---|---|
| Mathematical Definition | ||
| Physical Meaning | Occurrence of completely prevents occurrence of | Occurrence of provides zero information about |
| Can they occur together? | Never; impossible simultaneously | Yes; can occur together with joint probability |
| Conditional Probability | (assuming ) | |
| Classic Example | Rolling a and rolling a on a single die | Getting Heads on Coin 1 and getting Heads on Coin 2 |
[!IMPORTANT] Two non-empty events with non-zero probabilities () can never be simultaneously mutually exclusive and independent. If they are mutually exclusive, . But if they are independent, , an obvious mathematical contradiction. In fact, mutually exclusive events are extremely dependent: knowing that occurred guarantees that did not occur!
Conditional Probability and Joint Probability
In real-world data science, events rarely occur in isolation. We constantly evaluate the probability of an outcome given partial observations or contextual features.
Conditional Probability
Let and be two events in sample space with . The conditional probability of event given that event has already occurred is:
Similarly, the conditional probability of given (with ) is:
Joint Probability and the Multiplication Rule
Rearranging the conditional probability formulas yields the General Multiplication Rule for the joint probability :
For three events , the Chain Rule of Probability expands joint probabilities into a sequential cascade of conditional factors:
This decomposition is the foundational architectural pillar of probabilistic graphical models, Markov chains, and autoregressive language models (such as modern transformers predicting the next token conditioned on all preceding tokens).
The Law of Total Probability and Bayes' Theorem
Let the collection of events form a partition of the sample space , meaning:
- They are mutually exclusive: for all .
- They are collectively exhaustive: .
- Each partition has non-zero probability: .
Law of Total Probability
For any arbitrary event , the total probability of can be decomposed across the partitions:
Bayes' Theorem and Total Probability
Let be a partition of the sample space , and let be an observed event with . The conditional (posterior) probability of hypothesis given evidence is:
Terminology in Bayesian Machine Learning
- Prior Probability : The baseline probability of hypothesis before observing new evidence.
- Likelihood : The probability that evidence would be observed if hypothesis were true.
- Marginal Evidence : The total probability of observing evidence summed across all possible mutually exclusive hypotheses.
- Posterior Probability : The updated probability of hypothesis after observing and conditioning on evidence .
Random Variables
In data science, we rarely work directly with textual outcome names like "Heads" or "Patient survived". Instead, we map experimental outcomes to real numbers so they can be processed by mathematical algorithms.
Discrete and Continuous Random Variables
A random variable is a measurable mathematical function that maps each outcome in the sample space to a unique real number:
Random variables are categorized based on the cardinality of their range:
- Discrete Random Variable: Takes on a finite or countably infinite set of distinct numerical values (e.g., ).
- Characterized by a Probability Mass Function (PMF): .
- Conditions: for all , and .
- The probability of a specific point value is generally non-zero ().
- Continuous Random Variable: Takes on any value within an uncountable real interval (e.g., height, temperature, transaction latency).
- Characterized by a Probability Density Function (PDF): .
- Conditions: for all , and .
- Crucial distinction: The probability of a continuous random variable taking on any single exact point value is identically zero:
- Probabilities are defined exclusively over intervals:
| Characteristic | Discrete Random Variable | Continuous Random Variable |
|---|---|---|
| Possible Values | Finite or countably infinite () | Uncountable continuum / real intervals () |
| Probability Function | Probability Mass Function (PMF): | Probability Density Function (PDF): |
| Normalization | Summation: | Integration: |
| Point Probability | Can be non-zero () | Strictly zero () |
| Interval Probability | ||
| Cumulative Dist. (CDF) | ||
| Expected Value | ||
| Real-World Examples | Number of website clicks, defective items, word count | Customer wait time, student height, transaction amount |
Expectation and Variance of a Random Variable
- Expected Value (Mean ): Measures the center of mass or theoretical average of the distribution:
- Variance ( or ): Measures the dispersion or spread of values around the mean:
- Standard Deviation (): The square root of the variance, expressed in the same physical units as :
2.5 Probability Distributions: Normal, Bernoulli, Binomial, Poisson
A probability distribution is a mathematical formula, rule, or table that describes how probabilities are distributed across the potential values of a random variable. In data science, standard parametric distributions serve as generative models for observed data, theoretical baselines for hypothesis testing, and likelihood functions in statistical learning.
Bernoulli Distribution
Bernoulli and Binomial Distributions
The Bernoulli Distribution is the discrete probability distribution of a random variable representing a single trial with exactly two mutually exclusive outcomes:
- Success (): with probability ().
- Failure (): with probability .
Probability Mass Function (PMF):
Moments:
- Mean:
- Variance:
- Standard Deviation:
Role in Data Science: The Bernoulli distribution forms the probabilistic foundation of binary classification. In logistic regression, the model outputs , and the binary cross-entropy loss function is derived directly from the negative log-likelihood of Bernoulli-distributed labels:
Binomial Distribution
The Binomial Distribution models the total number of successes observed in a sequence of independent and identically distributed (i.i.d.) Bernoulli trials, each having a constant probability of success .
Structural Assumptions
- The number of trials is fixed in advance.
- Each trial has exactly two outcomes: success () or failure ().
- The probability of success remains constant across all trials.
- Trials are mutually independent: the outcome of any trial does not influence any other.
Probability Mass Function (PMF):
where the binomial coefficient counts the number of distinct combinations of choosing successes out of trials:
Moments: Since a Binomial random variable is the sum of independent Bernoulli random variables where :
- Mean:
- Variance:
- Standard Deviation:
Role in Data Science: Used in digital marketing A/B testing (measuring the number of conversions out of site visitors), manufacturing quality assurance (counting defective circuit boards out of a batch of ), and clinical drug trial efficacy testing.
Poisson Distribution
Poisson Distribution
The Poisson Distribution is a discrete probability distribution that models the count of rare events occurring within a fixed continuous interval of time, distance, area, or volume, given that these events occur at a known constant average rate and independently of the time since the last event.
Probability Mass Function (PMF):
where:
- is the expected number of event occurrences in the interval (rate parameter).
- is Euler's constant.
- is the factorial of .
Moments and Equidispersion:
- Mean:
- Variance:
- Standard Deviation:
[!NOTE] The defining mathematical hallmark of the Poisson distribution is equidispersion: its theoretical variance is exactly equal to its theoretical mean (). In applied exploratory data analysis, if empirical sample variance significantly exceeds the sample mean, the data exhibits overdispersion, suggesting a Negative Binomial model rather than a standard Poisson.
Poisson as a Limiting Form of the Binomial Distribution
When the number of trials and the probability of success such that the expected value remains finite and constant, the Binomial distribution converges asymptotically to the Poisson distribution:
Role in Data Science: Used in server traffic engineering (modeling the number of HTTP requests hitting an API server per second), cybersecurity (detecting DDoS attacks and anomalous login attempts), telecom queueing systems, and insurance catastrophe claim frequency modeling.
Normal (Gaussian) Distribution
Normal (Gaussian) Distribution
The Normal (or Gaussian) Distribution is a continuous probability distribution whose probability density function is symmetric and bell-shaped around its central mean. It is completely parameterized by two moments: its mean and its variance (written ).
Probability Density Function (PDF):
where:
- is the location parameter (mean, median, and mode).
- is the scale parameter (standard deviation).
- and .
Key Geometric and Mathematical Properties
- Unimodal and Symmetrical: The bell curve is perfectly symmetric around . The left and right halves are mirror images.
- Central Tendency Alignment: For any normal distribution:
- Inflection Points: The curve transitions from concave downward to concave upward at exactly one standard deviation on either side of the mean ( and ).
- Asymptotic Tails: The tails extend infinitely in both directions along the horizontal axis (), approaching zero asymptotically but never touching it.
- Total Probability Area: The total area enclosed under the probability density curve is exactly :
The Standard Normal Distribution ()
Any normal random variable can be transformed into the Standard Normal Distribution with mean and variance via the standardizing -score transformation:
Its probability density function simplifies to:
The Empirical Rule ( Rule)
For any normally distributed population:
- of all values fall within standard deviation of the mean:
- of all values fall within standard deviations of the mean:
- of all values fall within standard deviations of the mean:
Values lying more than standard deviations from the mean () are exceptionally rare () and are frequently flagged as statistical outliers in data preprocessing pipelines.
| Distribution | Type | Parameters | PMF / PDF Formula | Mean () | Variance () | Primary Data Science Application |
|---|---|---|---|---|---|---|
| Bernoulli | Discrete | Binary classification targets, logistic regression | ||||
| Binomial | Discrete | A/B testing conversion counts, batch defects | ||||
| Poisson | Discrete | Server request arrival rates, fraud call frequency | ||||
| Normal | Continuous | CLT asymptotics, OLS residual noise, -score scaling |
Worked Examples
2.5
A market research firm models customer habits using Bayes' Theorem.
- Prior probability that a randomly chosen individual is a smoker: .
- Probability that a smoker buys a pack of cigarettes: .
- Total probability that any random individual buys a pack of cigarettes: .
Calculate the posterior probability that an individual who is observed buying a pack of cigarettes is a smoker ().
Identifying given probabilistic components:
- Prior probability of hypothesis:
- Likelihood of evidence given hypothesis:
- Marginal probability of evidence:
Applying Bayes' Theorem: Substitute the values into Bayes' formula:
Observing the purchase of cigarettes updates the probability that the individual is a smoker from a baseline prior of to a posterior probability of .
2.6
The height of students in a university engineering class is modeled as a normally distributed continuous random variable . Calculate the probability that a randomly selected student has a height between and .
Standardizing the interval endpoints via -scores: The parameters are mean and standard deviation . For lower limit :
For upper limit :
Evaluating the probability over the standardized range: The interval corresponds to :
Using standard normal distribution tables where and :
This confirms the first tier of the Empirical Rule ( within ).
Approximately of students in the class have heights between and .
2.7
An electronics manufacturer produces microchips where the probability of a chip passing quality control (defect-free) is . In a random sample of microchips, calculate:
- The probability that exactly chips pass quality control ().
- The probability that all chips pass quality control ().
Formulating the Binomial PMF: With trials and success probability (failure probability ):
Part 1: Exactly defect-free chips (): Compute the binomial coefficient:
Compute probability:
Part 2: All defect-free chips ():
2.8
A customer support call center receives incoming calls at an average rate of calls per hour. Assuming call arrivals follow a Poisson distribution, determine:
- The probability that the call center receives exactly calls in a given hour.
- The probability that the call center receives zero calls in a given hour.
Setting up the Poisson PMF: For rate parameter :
Using .
Part 1: Probability of receiving exactly calls ():
Part 2: Probability of receiving zero calls ():
Exercises
Exercises — Probability, Random Variables & Distributions
Define mathematical probability and state its three foundational Kolmogorov axioms. From these axioms, derive the complement rule and the addition rule for non-mutually exclusive events.
Formal definition of probability: Probability is a mathematical function that assigns to each event in sample space a real number quantifying its likelihood of occurrence.
The three Kolmogorov axioms:
- Non-negativity: For any event , .
- Normalization (Certainty): For the entire sample space , .
- Countable Additivity: For pairwise mutually exclusive events ( for ):
Derivation of the complement rule: Any event and its complement partition the sample space :
By Axiom 3 (Additivity for disjoint events):
By Axiom 2 (Normalization):
Derivation of the general addition rule: Express as the union of two mutually disjoint sets:
where . Applying Axiom 3:
Similarly, can be partitioned into two disjoint subsets:
Rearranging gives . Substituting back:
Probability is a normalized measure on sample spaces satisfying: (1) , (2) , and (3) for disjoint events. These axioms yield the complement rule and the general addition rule .
Differentiate between discrete and continuous random variables with mathematical definitions and practical engineering examples. Present a comprehensive comparative table.
Discrete random variables: A random variable is discrete if its range of possible values is finite or countably infinite:
Its distribution is specified by a Probability Mass Function (PMF) satisfying and . The probability of an exact individual value can be non-zero. Example: counting the number of defective packets in an IoT transmission batch.
Continuous random variables: A random variable is continuous if it can take on any real value within an uncountable interval . Its distribution is specified by a Probability Density Function (PDF) satisfying and .
Because the real continuum contains uncountably infinite points, the probability of obtaining any exact point value is zero: . Probabilities are non-zero only over finite intervals: . Example: measuring battery discharge life or server response latency.
Comparative analysis table:
| Comparison Dimension | Discrete Random Variable | Continuous Random Variable |
|---|---|---|
| Domain Range | Countable set of isolated numbers | Continuous real interval |
| Probability Function | Probability Mass Function (PMF), | Probability Density Function (PDF), |
| Normalization | Summation: | Integration: |
| Probability at Point | May be strictly positive: | Always identically zero: |
| Interval Probability | ||
| Engineering Examples | Daily website signups, bit transmission errors | Vehicle speed, temperature, sensor output voltage |
Discrete random variables have countable outcome values characterized by PMFs with non-zero point probabilities; continuous random variables span uncountable real intervals characterized by PDFs where single-point probability is zero and probabilities exist only over intervals.
Derive the mathematical formulation of conditional probability and the joint multiplication rule. From these principles, derive Bayes' Theorem for updating probabilistic beliefs.
Definition of conditional probability: The conditional probability of event given that event has occurred () is defined as the ratio of the joint probability to the conditioning probability:
Similarly, the conditional probability of given () is:
The joint multiplication rule: Rearranging equations (i) and (ii) yields two equivalent expressions for the joint probability :
Equating the right-hand sides:
Deriving Bayes' Formula: Dividing both sides by yields the fundamental Bayes' Formula:
Expansion via Law of Total Probability: If the sample space is partitioned into mutually exclusive, exhaustive hypotheses , then by the Law of Total Probability:
Substituting into the denominator gives Bayes' Theorem in its complete form:
Conditional probability implies , which yields Bayes' Theorem: .
State the mathematical difference between independent events and mutually exclusive events. Use algebraic equations and concrete examples to explain why two events with positive probabilities cannot be both mutually exclusive and independent.
Definitions and mathematical criteria:
- Mutually Exclusive Events: Events and cannot happen simultaneously. Their intersection is empty:
Example: Drawing a single card from a deck that is simultaneously an Ace and a King ().
- Independent Events: Occurrence of event does not alter the likelihood of event :
Example: Flipping a coin twice where the first toss is Heads and the second toss is Tails ().
Rigorous proof of incompatibility: Let and be two events with strictly positive probabilities ( and ).
- Suppose and are independent. Then:
Since their joint probability is strictly positive, , so they cannot be mutually exclusive.
- Conversely, suppose and are mutually exclusive. Then:
For independence to hold, we would require:
which contradicts the premise that both and .
Mutually exclusive events cannot occur together (), whereas independent events satisfy . Therefore, two events with non-zero probabilities can never be both mutually exclusive and statistically independent.
Write down the Probability Mass Function (PMF), expected value, and variance for both the Bernoulli distribution and the Binomial distribution. Explain how the Binomial distribution is related to the Bernoulli distribution.
Bernoulli Distribution: Models a single trial with binary outcomes: success () with probability and failure () with probability .
- PMF:
- Expected Value:
- Variance:
Binomial Distribution: Models the total count of successes in independent and identically distributed Bernoulli trials:
- PMF:
where .
- Expected Value:
- Variance:
Relationship between the two distributions: The Binomial distribution is the exact sum of independent, identically distributed Bernoulli trials:
Conversely, the Bernoulli distribution is a special case of the Binomial distribution with .
Bernoulli PMF is with mean and variance . Binomial PMF is with mean and variance . The Binomial distribution represents the sum of independent, identically distributed Bernoulli trials.
Explain the Normal distribution. State its Probability Density Function (PDF), specify its parameters, and indicate its mean, variance, and standard deviation. State and explain the Empirical Rule.
Mathematical description and PDF: The Normal (Gaussian) distribution is a continuous, unimodal, bell-shaped probability distribution parameterized by mean and variance :
Distribution parameters and moments:
- Location parameter (Mean): (also equals median and mode due to symmetry).
- Scale parameter (Variance): .
- Standard Deviation: .
- Total area under the density curve is .
The Empirical Rule ( Rule): For any normal distribution :
- of observations lie within standard deviation of the mean: .
- of observations lie within standard deviations of the mean: .
- of observations lie within standard deviations of the mean: .
The Normal distribution has PDF , with mean , variance , and standard deviation . The Empirical Rule states that approximately , , and of data lie within , , and standard deviations of the mean, respectively.
A marble is drawn at random from an opaque bag containing red marbles, blue marbles, and green marbles. What is the probability that the drawn marble is not green?
Determining the sample space and favorable outcomes: Total number of marbles in the bag:
Computing the probability of drawing a green marble: Let denote the event of selecting a green marble:
Applying the complement rule: Let denote the event of drawing a marble that is not green:
Alternatively, directly summing non-green favorable outcomes:
The probability that the drawn marble is not green is ().
A public health symposium is attended by medical professionals classified by gender and profession:
| Gender | Doctors | Nurses | Total |
|---|---|---|---|
| Female | |||
| Male | |||
| Total |
If one attendee is selected at random:
- Find the probability that a doctor is chosen ().
- Find the probability that a female attendee is chosen ().
- Find the probability that a female nurse is chosen ().
- Given that a female attendee is chosen, find the conditional probability that she is a nurse ().
- Determine whether the events "a nurse is chosen" and "a female attendee is chosen" are statistically independent.
Evaluating marginal and joint probabilities: Total attendees .
- Probability of selecting a doctor:
- Probability of selecting a female:
- Joint probability of selecting a female nurse:
Evaluating conditional probability : Using the conditional probability formula:
Testing statistical independence: For events and to be statistically independent, we must have , or equivalently . Calculate the marginal probability of being a nurse:
Comparing:
Because , the events are not independent (they are statistically dependent). Female attendees are more likely to be nurses () than attendees in the general population ().
Events and are not independent because .
A naive Bayesian email filter is designed to classify incoming emails into spam () and valid (). Statistical analysis indicates:
- of incoming emails are spam ().
- The keyword "free" appears in of spam messages ().
- The keyword "free" appears in only of valid messages ().
Determine:
- The total probability that an incoming email contains the keyword "free" ().
- The posterior probability that an email is spam given that it contains "free" ().
- The posterior probability that an email is valid given that it does not contain "free" ().
Formal definitions and complementary priors: Let:
- : Event that an email is spam ().
- : Event that an email is valid ().
- : Event that an email contains the keyword "free".
- : Event that an email does not contain "free".
Conditional probabilities:
Part 1: Total probability that an email contains "free": By the Law of Total Probability:
As a simplified fraction: .
Part 2: Probability of spam given that "free" is present (): Applying Bayes' Theorem:
The presence of "free" elevates the probability of spam from to .
Part 3: Probability of valid given that "free" is absent (): First compute the marginal probability of not containing "free":
Applying Bayes' Theorem:
Define a complementary event. A fair six-sided die is rolled. Let be the event "a number less than is obtained" and let be the event "a number of or more is obtained". Find and , and verify that and are complementary events. [Assignment, notebook p052]
Definition of complementary events: Two events and in a sample space are complementary events if they are mutually exclusive (, so they cannot occur at the same time) and collectively exhaustive (, so they make up the entire sample space). Consequently, their probabilities satisfy:
Listing the sample space and event outcomes: When rolling a standard six-sided die, the sample space has equally likely outcomes:
- Event ("a number less than is obtained"):
- Event ("a number of or more is obtained"):
Computing probabilities and verifying complementarity:
Notice that and . Summing their probabilities:
Thus, and are complementary events.
Two events are complementary if they cannot occur simultaneously () and their union is the entire sample space (). Here , , and .
Consider the following training dataset of observations with two categorical features and and a binary target class :
| Obs | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| A | A | A | A | A | B | B | B | B | B | C | C | C | C | C | |
| S | M | M | S | S | S | M | M | L | L | L | M | M | L | L | |
| 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 |
Using Bayes' theorem under the Naive Bayes conditional independence assumption, compute the prior probabilities and , the class-conditional likelihoods, and predict the class label for the test observation by maximizing . [Assignment, notebook p243]
Computing class prior probabilities : Out of the total observations:
- Class occurs in observations ( times):
- Class occurs in observations ( times):
Computing class-conditional feature likelihoods: Counting feature occurrences within each class subset:
- For :
- For :
- For :
Evaluating unnormalized posterior scores for : By the Naive Bayes conditional independence assumption, :
- For class :
- For class :
Decision by maximum a posteriori (MAP) rule: Comparing the two posterior numerators:
Therefore, the classifier assigns the observation to class .
Since , the predicted class label is .