Past exam question: 2071 Shrawan, Q9(b)
Open in the Padhaiko appQuestion
Define Entropy with example.
Solution
Definition of Entropy: Entropy, denoted , is the statistical average (expected value) of information conveyed by a discrete random source alphabet :
- Entropy represents the average uncertainty of the source and defines the theoretical lower bound on average codeword length for lossless data compression.
Example: Consider a source emitting 4 equiprobable symbols with :
Answer:
Entropy is the average information per symbol emitted by a source (e.g., 4 equiprobable symbols with have ).