PadhaikoAll subjects

Past exam question: 2075 Chaitra, Q8

Data Communication · 8 marks

Open in the Padhaiko app

Topic: Wireless propagation (Introduction to groundwave propagation, sky…

Question

Define entropy in information theory. Find the efficient code word and efficiency using Hoffman algorithm using probabilities .

Solution

Definition of Entropy in Information Theory: In information theory (formulated by Claude Shannon in 1948), entropy is the mathematical measure of the average information content, uncertainty, or surprise produced by a discrete memoryless information source :

where is the probability of occurrence of source symbol , and is the self-information of symbol . Entropy establishes the fundamental lower bound on the average codeword length required for lossless source compression (Shannon's Source Coding Theorem: ).

Huffman Tree Construction for the Given Probabilities: Source symbols arranged in descending order:

Merging stages:

  1. Merge lowest two: . New sorted list: .
  2. Merge lowest two: . New sorted list: .
  3. Merge lowest two: . New sorted list: .
  4. Merge final two: .

Assigning binary digits (0 to top/higher, 1 to bottom/lower branch):

Codeword Table and Performance Calculations:

SymbolProbability CodewordLength
01
102
1103
11104
11114
Total1.00——
  1. Average Codeword Length:
  1. Entropy:
  1. Coding Efficiency ():
Answer:

All past exam questions for Data Communication