PadhaikoAll subjects

Logistic Regression

Foundation of Data Science

Open in the Padhaiko app

5.3 Logistic Regression

While linear regression predicts unbounded continuous quantities, many data science tasks require predicting categorical outcomes—most commonly binary classification, where the target variable takes two values (e.g., spam vs. non-spam, default vs. no default, disease vs. healthy). Logistic regression solves this by passing a linear combination of predictors through the sigmoid (logistic) function so that the output is always a valid probability in .

Binary Classification & The Sigmoid Function

Definition

Logistic Regression Model and Sigmoid Function

Logistic regression is a supervised statistical classification model that estimates the conditional probability of a binary outcome from predictors .

First compute the linear predictor :

and then apply the logistic (sigmoid) function :

and .

To convert probabilities into a hard class prediction , we apply a decision threshold (commonly ):

When , the decision boundary corresponds to .

AspectLinear RegressionLogistic Regression
Target TypeContinuous numerical response ().Binary categorical outcome ().
Output RangeUnbounded: .Bounded probability: via the sigmoid curve.
Model Equation
Estimation MethodOrdinary Least Squares (closed-form ).Maximum Likelihood Estimation (iterative optimization; no closed form).
Error / Outcome DistributionTypically assumes Gaussian residuals .Assumes Bernoulli outcome .
Coefficient MeaningExpected additive change in per unit increase in .Additive change in log-odds (multiplicative factor on odds).

Odds, Log-Odds (Logit) & Coefficient Interpretation

To interpret logistic regression coefficients cleanly, we invert the sigmoid function to express a linear relationship on the log-odds scale.

Definition

Odds, Odds Ratio, and Log-Odds (Logit)

Let be the probability of the positive event:

  1. Odds: The ratio of the probability that the event occurs to the probability that it does not occur:
  1. Log-Odds (Logit): The natural logarithm of the odds:
  1. Coefficient Interpretation and Odds Ratio: Holding all other predictors constant, a one-unit increase in predictor :
    • Changes the log-odds by (positive increases ; negative decreases ).
    • Multiplies the odds by the odds ratio . If , (odds increase); if , (odds decrease); if , (no effect).

Maximum Likelihood Estimation (MLE) & Log-Loss

Unlike linear regression, logistic regression does not use ordinary least squares. Instead, parameters are estimated by Maximum Likelihood Estimation (MLE).

Theorem

Maximum Likelihood Estimation and Binary Cross-Entropy

Given independent observations with and , each response follows with PMF . The likelihood function is:

Taking the natural logarithm gives the log-likelihood:

Maximizing is equivalent to minimizing the negative average log-likelihood (Binary Cross-Entropy / Log-Loss):

Differentiating with respect to using gives the gradient:

Because is nonlinear, has no closed-form algebraic solution; however, is convex, so iterative optimization methods such as gradient descent () or Newton–Raphson (IRLS) converge reliably to the global optimum.

Regularization ( Lasso and Ridge)

When a logistic regression model has many correlated predictors or nearly linearly separable classes, unregularized MLE weights can grow arbitrarily large and overfit the training data. Regularization adds a penalty term on the slope coefficients (leaving the intercept unpenalized) controlled by a hyperparameter .

Definition

L1 (Lasso) and L2 (Ridge) Regularization

  1. Regularization (Lasso): Adds a penalty proportional to the sum of absolute coefficient magnitudes:
  1. Regularization (Ridge): Adds a penalty proportional to the sum of squared coefficient magnitudes:
AspectL1 Regularization (Lasso)L2 Regularization (Ridge)
Penalty Term
Effect on CoefficientsDrives weak coefficients to exact zero (sparse solution).Shrinks coefficients toward zero, keeping them small and non-zero.
Feature SelectionYes—eliminates irrelevant predictors automatically.No—retains all predictors with damped weights.
Multicollinearity HandlingArbitrary selection among correlated predictors.Distributes weight evenly across correlated features; very stable.

Real-World Applications of Logistic Regression

Because logistic regression outputs calibrated probabilities and interpretable feature weights, it is widely deployed across industries.

Real-World Applications of Logistic Regression
DomainTypical Predictors ()Binary Outcome ()Operational Decision
Healthcare (Medical Diagnosis)Age, BMI, blood pressure, glucose, lab biomarkersDisease present () vs. absent/benign ()Flag at-risk patients for confirmatory screening or treatment.
Finance (Credit Scoring)Credit score, annual income, debt-to-income ratio, payment historyLoan default () vs. non-default ()Approve/reject loan applications and set risk-adjusted interest rates.
Marketing (Customer Churn)Usage frequency, tenure, support tickets, billing tierSubscription cancelled () vs. retained ()Trigger targeted retention offers for high-churn-probability customers.
Cybersecurity (Spam Detection)Keyword frequencies, link count, sender domain reputationSpam () vs. legitimate mail ()Quarantine or filter incoming messages exceeding the probability threshold.

Worked Examples

Example 1

Computing Probability, Odds, Odds Ratio, and Class Prediction

A bank's credit-risk logistic regression model predicts the probability of loan default () using debt-to-income ratio (scaled) and prior late payments :

For an applicant with and :

  1. Compute the linear predictor , predicted probability , and odds of default.
  2. Classify the applicant using threshold .
  3. Compute the odds ratios for and and interpret them.

Evaluate linear predictor and probability :

Applying the sigmoid function:

Compute the odds and classify at :

(equivalently, ). Since (equivalently ), the model predicts (Default).

Compute and interpret the odds ratios:

  • A one-unit increase in multiplies the odds of default by (an increase in odds), holding constant.
  • Each additional late payment ( increases by ) multiplies the odds of default by (more than tripling the odds), holding constant.
Answer:

For : , , , and predicted class . The odds ratios are for and for .

Example 2

Comparing Candidate Parameters via Likelihood and Binary Cross-Entropy

Consider a binary dataset of observations : . Two candidate logistic models are proposed:

Compute the likelihood , log-likelihood , and average binary cross-entropy for both candidates, and state which has higher likelihood.

Evaluate Candidate A ():

  • (): , contribution
  • (): , contribution
  • (): , contribution
  • (): , contribution

Evaluate Candidate B ():

  • (): ,
  • (): ,
  • ():
  • ():

Compare likelihood and log-loss: Candidate A yields more than four times the likelihood of Candidate B () and a substantially lower binary cross-entropy (), so MLE strongly favors Candidate A.

Answer:

Candidate A achieves , , ; Candidate B achieves , , . Candidate A is selected under MLE.

Example 3

One Gradient Descent Iteration for Logistic Regression MLE

Using the binary points and starting from with learning rate :

  1. Compute the initial log-loss and gradients .
  2. Perform one gradient descent update to obtain .
  3. Verify that .

Evaluate initial predictions and log-loss at : At , and for every . Every term in the log-loss equals :

Compute the gradient of at : The prediction errors are:

  • :
  • :
  • :
  • :
  • :

Perform the gradient descent update and verify log-loss reduction: With :

At :

  • : ,
  • : ,
  • : ,
  • : ,
  • : ,
Answer:

Initial log-loss , gradient , updated weights , and new log-loss .

Exercises

Exercises on Logistic Regression

Question 1

Define logistic regression and explain how it differs from linear regression. Write the mathematical form of the logistic (sigmoid) function. [Assignment-5, Q12]

Definition and mathematical form: Logistic regression is a statistical classification algorithm used for binary outcomes (). Rather than predicting a raw continuous value directly, it maps a linear combination of predictors through the sigmoid (logistic) function to produce a probability in :

Key differences from linear regression:

  1. Target and Output: Linear regression models continuous responses over ; logistic regression models binary categories by outputting probabilities strictly between and .
  2. Functional Mapping: Linear regression uses the identity link ; logistic regression wraps in the nonlinear S-shaped sigmoid curve.
  3. Parameter Estimation: Linear regression minimizes sum of squared errors (closed form); logistic regression maximizes Bernoulli likelihood (iterative optimization).
Answer:

Logistic regression models binary classification outcomes by predicting . Unlike linear regression—which outputs unbounded continuous values and fits parameters via least squares—logistic regression bounds outputs to using the sigmoid function and estimates coefficients via Maximum Likelihood Estimation.

Question 2

Explain the concepts of odds and log-odds (logit) in logistic regression, and show mathematically how the logit maps probabilities from to . [Assignment-5, Q13]

Defining odds and log-odds (logit): If an event occurs with probability , the odds are the ratio of event probability to non-event probability:

The log-odds (logit) is the natural logarithm of the odds:

Logistic regression assumes that the logit is a linear function of the predictors: .

Mapping :

  • As , the ratio , so .
  • At , , so .
  • As , , so and .

Hence the logit maps bijectively onto , permitting unbounded linear modeling without violating probability bounds.

Answer:

Odds are and log-odds (logit) are . Taking the ratio maps to and taking the natural logarithm maps to , enabling linear combinations of predictors to determine valid probabilities.

Question 3

Consider the simple logistic regression model with and :

  1. Compute the predicted probability manually at and at .
  2. Describe how the coefficient is interpreted in terms of log-odds, odds ratio, and event probability. [Assignment-5, Q14]

Compute predicted probability at :

Compute predicted probability at :

Interpret :

  • Log-odds interpretation: Each one-unit increase in increases by .
  • Odds ratio interpretation: Each one-unit increase in multiplies the odds by (a increase in odds).
  • Direction of probability: Because , increasing raises the probability of (as seen from ), whereas a negative coefficient would decrease it.
Answer:

At , and ; at , and . The slope means each one-unit increase in increases the log-odds by and multiplies the odds by , raising the probability of .

Question 4

Explain how Maximum Likelihood Estimation (MLE) is used to fit a logistic regression model, write the log-likelihood function, and state why ordinary least squares is not used instead. [Assignment-5, Q15]

Bernoulli likelihood and log-likelihood: For binary targets with predicted probabilities , each observation follows . MLE selects to maximize the joint probability of observing the sample labels:

Taking the logarithm produces the concave log-likelihood:

Iterative optimization and why least squares is avoided: Setting yields a nonlinear system with no closed-form solution, so MLE is maximized iteratively via gradient ascent/descent on or Newton–Raphson (IRLS). Ordinary least squares is avoided because passing the sigmoid through a squared loss produces a non-convex objective with local minima and ignores the Bernoulli variance .

Answer:

MLE fits logistic regression by choosing to maximize the Bernoulli log-likelihood via iterative optimization (e.g., gradient descent or Newton–Raphson). Least squares is not used because the squared-error sigmoid loss is non-convex and unsuited to Bernoulli binary outcomes.

Question 5

Explain real-world applications of logistic regression in each of the following domains: (a) Healthcare, (b) Finance, (c) Marketing, and (d) Cybersecurity. [Assignment-5, Q16]

Healthcare (Medical Diagnosis): Used to estimate the probability of disease presence vs. absence (e.g., malignant vs. benign tumor, or diabetes risk). Predictors such as age, BMI, and blood biomarkers quantify patient risk so clinicians can prioritize diagnostic follow-up.

Finance (Credit Scoring): Banks predict whether a loan applicant will Default () or Not Default () using credit score, income, debt-to-income ratio, and repayment history. The resulting default probability informs loan approval and pricing.

Marketing (Customer Churn): Subscription businesses predict whether a customer will Cancel () or Stay () based on engagement, tenure, and billing complaints, allowing targeted retention incentives to be sent to high-risk users.

Cybersecurity (Spam Detection): Email filters classify incoming messages as Spam () or Not Spam () using keyword frequencies, URL counts, and sender metadata, blocking malicious or unwanted mail automatically.

Answer:

Logistic regression supports probability-driven binary decisions across: (a) Healthcare (disease diagnosis from patient vitals and lab markers), (b) Finance (loan default prediction from credit and income features), (c) Marketing (customer churn prediction to guide retention offers), and (d) Cybersecurity (spam vs. legitimate email classification from text and sender features).

Question 6

Describe the impact of (Lasso) and (Ridge) regularization on logistic regression models, comparing their effects on coefficients, feature selection, and multicollinearity. [Assignment-5, Q17]

Regularization (Lasso): Adds the penalty to the negative log-likelihood.

  • Impact: Shrinks coefficients and can force weak or redundant feature weights to exact zero.
  • Benefit: Automatically performs feature selection, yielding sparse, compact, and more interpretable models.

Regularization (Ridge): Adds the penalty to the negative log-likelihood.

  • Impact: Shrinks all coefficients proportionally toward zero, but never drives them strictly to zero.
  • Benefit: Effectively stabilizes estimates under multicollinearity (strongly correlated predictors) and prevents weights from diverging when classes are nearly separable, improving out-of-sample stability.
Answer:

(Lasso) regularization adds and can shrink coefficients to exact zero, performing embedded feature selection. (Ridge) regularization adds , shrinking coefficients smoothly without zeroing them, which handles multicollinearity effectively and improves model stability.

Question 7

A fitted logistic regression model has . Evaluate it on the test observations :

  1. Compute the predicted probability and predicted class (with threshold ) for each observation.
  2. Compute the log-likelihood , the average binary cross-entropy , and the classification accuracy.

Evaluate , , and for all four points:

  • : (correct).
  • : (correct).
  • : (correct).
  • : (correct).

Compute log-likelihood , average log-loss , and accuracy:

All out of predictions match the true labels, so classification accuracy is .

Answer:

Predicted probabilities are , giving predicted classes ( accuracy), with and .

Past exam questions for Foundation of Data Science

← Multiple & Polynomial Regression, Categorical Regressors & Model BuildingMachine Learning: Supervised, Unsupervised & Reinforcement →