Sound/Audio System
Open in the Padhaiko app2.1 Overview of sound system
Sound and Acoustic Parameters
Sound is a physical phenomenon produced by the vibration of matter that propagates through an elastic medium (such as air) as longitudinal pressure waves. A complete sound system captures, processes, stores, transmits, and reproduces these acoustic pressure variations.
Key Physical Properties of Sound
- Frequency (): Number of complete pressure cycles per second, measured in Hertz (); determines perceived pitch. The human audible range spans approximately to . Frequencies below are infrasound; frequencies above are ultrasound.
- Amplitude (): Maximum pressure deviation from equilibrium; determines perceived loudness (sound intensity).
- Decibel Scale (): Because human hearing responds logarithmically to sound intensity, relative loudness/amplitude is expressed in decibels:
- Waveform & Timbre: A pure single-frequency tone is a sine wave; real musical instruments and speech are complex sums of a fundamental frequency () and harmonics (integer multiples ) that give a sound its characteristic timbre.
Basic Audio Signal Chain
2.2 Producing digital audio
Continuous analog sound waves must be converted into discrete binary numbers through Pulse Code Modulation (PCM) inside an Analog-to-Digital Converter (ADC) in three consecutive stages:
Three Stages of Audio Digitization (PCM)
- Sampling (Time Discretization): Measuring the analog signal's amplitude at uniform time intervals , where is the sampling frequency (sampling rate) in .
- Quantization (Amplitude Discretization): Mapping each continuous sampled voltage to the nearest discrete level out of levels, where is the bit depth (bits per sample). The rounding difference between the true sample and the quantized level is quantization error (quantization noise).
- Encoding (Binary Representation): Representing each quantized level index as an -bit binary code word (e.g., 8-bit unsigned or 16-bit signed two's complement).
Nyquist–Shannon Sampling Theorem & Quantization SNR
- Nyquist–Shannon Theorem: A band-limited analog signal with maximum frequency can be reconstructed without aliasing distortion if and only if the sampling rate is strictly at least twice the maximum frequency:
- Signal-to-Quantization-Noise Ratio (SQNR): For a full-scale sinusoidal input quantized uniformly with bits per sample:
Each additional bit of resolution improves dynamic range by roughly .
Common Digital Audio Standards, Bit Rate, and File Size
For uncompressed PCM audio with sampling rate (), bit depth (), number of channels ( for mono, for stereo), and duration ():
| Standard / Medium | Sampling Rate () | Bit Depth () | Channels () | Uncompressed Bit Rate |
|---|---|---|---|---|
| PSTN Telephony | (-law / A-law) | (Mono) | ||
| Wideband Speech / VoIP | (Mono) | |||
| Audio CD (Red Book) | (Stereo) | |||
| Studio / Digital Video | (Stereo) |
2.3 Music and speech
Although both music and speech occupy the audible acoustic spectrum, they differ markedly in bandwidth, harmonic complexity, temporal structure, and coding requirements:
| Aspect | Speech | Music |
|---|---|---|
| Primary Frequency Bandwidth | Narrow: intelligible in (PSTN); wideband | Full audible spectrum: |
| Signal Source | Single human vocal tract (glottis + oral/nasal cavity resonators) | Multiple simultaneous instruments and voices (polyphony) |
| Harmonic & Spectral Structure | Quasi-periodic voiced vowels + noise-like unvoiced consonants; characterized by formant peaks | Rich overtones, complex chords, transients, and wide spectral variation |
| Dynamic Range | Moderate () | Very wide ( in orchestral music) |
| Coding Strategy | Parametric vocal-tract models (LPC, CELP) achieving | Perceptual psychoacoustic coding (MP3, AAC) at |
2.4 Speech Generation
Speech generation (speech synthesis) is the artificial production of human speech by a computer, most commonly realized as Text-to-Speech (TTS).
Basic Mechanism of Human Speech (Source–Filter Model)
- Voiced Sounds (e.g., vowels
/a/,/e/): Vocal folds vibrate periodically at a fundamental pitch frequency , producing a pulse train shaped by the resonant frequencies (formants ) of the vocal tract. - Unvoiced Sounds (e.g.,
/s/,/f/,/sh/): Vocal folds stay open; turbulent air forced through a constriction generates broadband noise shaped by the vocal tract.
Major Speech Generation Techniques
- Concatenative Synthesis: Joins pre-recorded segments of real human speech (words, syllables, diphones, or phonemes) stored in a database. Produces very natural voice quality (especially unit-selection TTS), though transitions require smoothing (PSOLA — Pitch-Synchronous Overlap and Add) and large voice databases.
- Formant (Rule-Based) Synthesis: Generates speech entirely mathematically without human recordings by driving parallel/cascade resonant filters (modeling formants ) with a synthetic glottal pulse (voiced) or white noise (unvoiced). Very small memory footprint and intelligible even at high speeds, though slightly robotic.
- Articulatory Synthesis: Models the physical 3D geometry and biomechanics of the tongue, lips, jaw, and vocal tract; accurate in theory but computationally expensive.
Standard Text-to-Speech (TTS) Pipeline
2.5 Speech Analysis
Speech analysis extracts acoustic, phonetic, and linguistic parameters from a spoken audio signal for Automatic Speech Recognition (ASR), speaker identification, and low-bit-rate speech coding.
Key Speech Analysis Steps and Techniques
- Pre-emphasis & Framing (Windowing): Speech is non-stationary over long intervals, but approximately stationary over short frames (typically shifted by overlap) multiplied by a Hamming/Hann window to reduce spectral leakage.
- Time-Domain Features:
- Short-Time Energy: Distinguishes speech from silence and voiced vowels (high energy) from unvoiced fricatives (lower energy).
- Zero-Crossing Rate (ZCR): Counts sign changes per frame; high ZCR indicates unvoiced consonants (
/s/,/f/), while low ZCR indicates voiced sounds.
- Frequency-Domain & Spectral Analysis:
- Short-Time Fourier Transform (STFT) & Spectrogram: Displays frequency content over time, revealing pitch harmonics and formant bands.
- Linear Predictive Coding (LPC): Models each speech sample as a linear combination of the past samples to compactly estimate the vocal-tract spectral envelope (formants) and separate the excitation source from the filter.
- MFCC (Mel-Frequency Cepstral Coefficients): Maps the power spectrum onto the perceptual Mel scale and applies Discrete Cosine Transform (DCT) to produce compact feature vectors widely used in ASR.
- Speech Recognition Pipeline: Acoustic frames (MFCCs) Acoustic Model (phoneme likelihoods via HMM/DNN) Pronunciation Dictionary + Language Model Recognized Word Sequence.
2.6 Speech Transmission
Real-time speech transmission (over PSTN cellular networks or IP-based VoIP) must minimize bit rate while maintaining intelligibility and conversational latency ( one-way).
| Technique | Coding Principle | Typical Bit Rate | Application |
|---|---|---|---|
| PCM (G.711) | Direct waveform coding with -law / A-law companding (, ) | PSTN digital trunks, LAN VoIP | |
| ADPCM (G.726) | Differential encoding of quantized prediction error () | DECT cordless phones, legacy VoIP | |
| LPC (LPC-10) | Transmits vocal-tract filter coefficients + pitch + voiced/unvoiced flag only | Military / ultra-low-bandwidth links | |
| CELP / AMR / Opus | Analysis-by-synthesis hybrid coding matching perceptual excitation codebook | GSM/LTE cellular voice, WebRTC VoIP |
Network Speech Transmission Mechanisms
- Voice Activity Detection (VAD) & DTX: Detects silence pauses (roughly of conversation) and suppresses packet transmission (Discontinuous Transmission), inserting Comfort Noise Generation (CNG) at the receiver.
- Echo Cancellation: Removes acoustic feedback and electrical hybrid reflections so speakers do not hear their own delayed voice.
- RTP/UDP Transport & Jitter Buffer: Speech packets travel over UDP + RTP (Real-time Transport Protocol) rather than TCP (avoiding retransmission stalls); a small playout (jitter) buffer at the receiver smooths network delay variation.
2.7 Representation of audio files
Digital audio is stored inside audio file formats (containers) that pair a header (metadata describing sample rate, channel count, bit depth, and encoding) with a data payload of samples or compressed frames.
| Format | Compression Type | Key Characteristics | Typical Use |
|---|---|---|---|
| WAV (RIFF) | Uncompressed PCM (usually) | Chunk-based RIFF header (fmt + data); standard header; interleaved L/R stereo samples | Editing masters, Windows audio, sound effects |
| AIFF | Uncompressed PCM | Apple IFF chunk format (big-endian PCM samples) | macOS studio production |
| AU | Uncompressed / -law | Simple Sun/UNIX header + big-endian PCM or -law | Legacy UNIX & Java audio |
| FLAC | Lossless Compression | Bit-exact reconstruction at of WAV size | Archival music, hi-res audio |
| MP3 (MPEG-1 Layer III) | Lossy (Psychoacoustic) | Removes masked/inaudible frequencies via MDCT + Huffman coding; compression () | Portable music, web streaming |
| AAC (MPEG-4 Audio) | Lossy (Psychoacoustic) | Higher coding efficiency than MP3 at identical bit rates; supports multichannel surround | Streaming video (MP4), mobile audio |
Sample Interleaving in Multi-Channel PCM
In a stereo WAV file, each time instant forms one audio frame of stored sequentially in interleaved order:
2.8 Computer Music –MIDI
MIDI (Musical Instrument Digital Interface)
MIDI is an industry-standard communication protocol and event representation (not an audio waveform!) that enables electronic musical instruments, synthesizers, sequencers, and computers to exchange performance control instructions in real time.
Instead of recording sound pressure samples, a MIDI file stores symbolic musical instructions—which note was pressed, on which channel, at what velocity (loudness), for how long, and using which instrument patch.
MIDI Architecture & Specifications
- Physical Serial Link: Asynchronous serial interface running at (, i.e., ), using MIDI IN, MIDI OUT, and MIDI THRU ports (or USB-MIDI).
- 16 Logical Channels: Supports up to independent instrument channels ( in binary, numbered for musicians). Under General MIDI (GM), standardized instrument programs (patches) are defined, and Channel 10 is reserved for percussion.
- MIDI Message Format: Messages consist of a Status Byte (, values
0x80–0xFF, encoding command type + 4-bit channel) followed by , , or Data Bytes (, values0x00–0x7F, i.e., ).- Note On (
0x9n):[Status: 0x90+ch] [Pitch: 0–127] [Velocity: 0–127](). Note is Middle C (). - Note Off (
0x8n):[Status: 0x80+ch] [Pitch: 0–127] [Release Velocity: 0–127](). - Program Change (
0xCn):[Status: 0xC0+ch] [Patch #: 0–127]() selects the instrument sound. - Pitch Bend (
0xEn) / Control Change (0xBn): Continuous expression (vibrato, sustain pedal, volume).
- Note On (
- Standard MIDI File (
.mid/ SMF): Composed of a Header Chunk (MThd, specifying format , track count, and ticks per quarter note) and one or more Track Chunks (MTrk) storing delta-time + MIDI events. - Sound Synthesis Methods: A MIDI sound card/synthesizer turns MIDI events into sound using FM Synthesis (Frequency Modulation of carrier/modulator sine oscillators) or Wavetable Synthesis (playing back short sampled instrument waveforms stored in ROM/RAM).
2.9 MIDI versus Digital Audio
| Aspect | MIDI | Digital Audio (PCM / WAV / MP3) |
|---|---|---|
| Underlying Representation | Symbolic performance events (note pitch, velocity, timing, patch) | Digitized acoustic waveform samples (amplitude over time) |
| File Size | Extremely small (a few KB for minutes of music) | Very large uncompressed ( for CD stereo); moderate when compressed |
| Speech & Natural Sound | Cannot represent spoken dialogue, singing lyrics, or arbitrary real-world noise | Accurately records speech, singing, ambient noise, and any acoustic sound |
| Playback Consistency | Device-dependent: sound quality varies across synthesizers/soundfonts | Hardware-independent: reproduces the exact recorded waveform everywhere |
| Editing Flexibility | Effortless to transpose pitch, change tempo without pitch shift, or swap instruments | Changing individual notes or separating instruments inside a mix is difficult |
| Typical Use Cases | Background game music, notation software, electronic composition, ringtones | Narration, podcasts, recorded songs, movie soundtracks, sound effects |
Worked Examples
1
An audio signal contains frequency components up to .
- What is the minimum Nyquist sampling rate required to digitize this signal without aliasing?
- If the audio is recorded in stereo at CD standard (, ) for , compute the theoretical quantization SQNR (in ) and the uncompressed PCM data size in ().
Determine the Nyquist sampling rate: By the Nyquist–Shannon sampling theorem, the minimum sampling rate is twice the highest frequency component:
(Using exceeds and leaves a guard band for practical anti-aliasing filters.)
Compute the theoretical SQNR for :
Compute the uncompressed stereo PCM size for :
- Minimum Nyquist sampling rate .
- Theoretical ; uncompressed stereo PCM size .
2
A speech analysis system samples voice at and segments a utterance into overlapping frames of duration with a frame shift ( overlap).
- Calculate the number of audio samples per frame () and the frame shift in samples ().
- How many complete frames are extracted from the recording?
- Briefly explain how LPC and MFCC features differ in speech analysis.
Compute frame length and frame shift in samples:
Count total complete frames in ():
LPC vs. MFCC:
- LPC (Linear Predictive Coding) models the vocal tract as an all-pole filter predicting each sample from previous samples, directly yielding formant frequencies and compact filter coefficients ideal for speech compression.
- MFCC (Mel-Frequency Cepstral Coefficients) transforms the short-time spectrum onto the human auditory Mel frequency scale and applies DCT, capturing perceptually weighted spectral envelopes ideal for speech and speaker recognition.
- , frame shift .
- Total complete frames .
- LPC models vocal-tract resonance filters for compression/formant tracking, whereas MFCC computes perceptually Mel-warped cepstral vectors for speech recognition.
3
A MIDI keyboard communicates over a standard serial link using ( start, data, stop bit).
- What is the maximum number of Note On messages that can be transmitted per second?
- A song uses note pairs (each note requiring one Note On and one Note Off message, plus of header/meta overhead, ignoring delta-time expansion). Calculate the MIDI file size in () and find the size ratio compared to a stereo CD-quality WAV payload (, ).
Maximum Note On message rate on the MIDI link:
Compare MIDI size against CD WAV payload:
Because MIDI stores symbolic events rather than sampled waveforms, it is over four orders of magnitude smaller than uncompressed CD audio!
- Maximum rate ().
- MIDI size (), which is roughly smaller than the CD WAV payload.
Exercises
Practice Exercises — Sound and Audio Systems
Explain the three stages involved in producing digital audio (PCM) from an analog sound wave. State the Nyquist–Shannon sampling theorem, define aliasing, and explain why telephony uses -law / A-law companding.
Three Stages of PCM Digitization:
- Sampling: Measuring continuous analog voltage at regular intervals .
- Quantization: Rounding each continuous sample amplitude to the nearest of discrete levels ( bits/sample), introducing quantization error.
- Encoding: Mapping each quantized level to an -bit binary code word.
Nyquist Theorem, Aliasing, and Companding:
- Nyquist–Shannon Theorem: To reconstruct a signal with highest frequency without loss, the sampling rate must satisfy .
- Aliasing: If , high frequencies fold back into lower frequencies during reconstruction, creating false audible tones; an analog anti-aliasing low-pass filter precedes sampling.
- Companding (-law / A-law): Logarithmic non-uniform quantization assigns finer quantization steps to quiet amplitudes and coarser steps to loud amplitudes, matching human hearing and delivering dynamic quality using only .
Digital audio (PCM) is produced via Sampling, Quantization, and Encoding. The Nyquist theorem requires to avoid aliasing (high frequencies folding into low frequencies). Telephony uses -law / A-law non-uniform companding to allocate more quantization levels to low-amplitude speech signals, achieving high perceptual quality at ().
Compare speech generation techniques (concatenative vs. formant synthesis) and describe the key stages of speech analysis and speech transmission in modern telecommunication systems.
Speech Generation (TTS):
- Concatenative Synthesis stitches recorded speech units (phonemes, diphones, words) for natural timbre, smoothed via PSOLA.
- Formant (Rule-Based) Synthesis generates sound mathematically using a source-filter model (glottal pulse / noise exciting resonant formant filters), requiring minimal storage.
Speech Analysis & Transmission:
- Speech Analysis segments speech into windowed frames and extracts short-time energy, zero-crossing rate, STFT spectrograms, LPC vocal-tract parameters, or MFCC features for recognition and coding.
- Speech Transmission compresses voice using PCM (), ADPCM (), or CELP/AMR codecs, and applies VAD/DTX (silence suppression), echo cancellation, and RTP/UDP with a jitter buffer to keep conversational latency low.
Concatenative synthesis joins recorded speech segments for naturalness, whereas formant synthesis models vocal-tract resonances mathematically. Speech analysis extracts frame-based time/spectral features (Energy, ZCR, STFT, LPC, MFCC), and speech transmission combines low-rate codecs (G.711, ADPCM, CELP), VAD/DTX, echo cancellation, and RTP/UDP jitter buffering.
What is MIDI? Explain the structure of MIDI messages (status vs. data bytes) and compare MIDI with Digital Audio across representation, file size, speech capability, and editing flexibility.
MIDI Definition and Message Structure: MIDI (Musical Instrument Digital Interface) is a digital protocol transmitting symbolic musical performance commands rather than audio waveforms over a serial link across channels. Each message begins with a Status Byte (, specifying command and channel, e.g., Note On 0x9n, Note Off 0x8n, Program Change 0xCn) followed by or Data Bytes (, carrying values for pitch, velocity, or patch number).
MIDI vs. Digital Audio:
- Representation: Symbolic control events vs. sampled acoustic amplitudes.
- File Size: Tiny (KB) vs. very large (MB for PCM, moderate for MP3/AAC).
- Speech / Natural Sound: MIDI cannot reproduce vocals or real-world sounds; Digital Audio captures any sound.
- Editing & Playback: MIDI notes/tempo/instruments are trivially editable, though playback depends on the synthesizer; Digital Audio is hardware-consistent everywhere but hard to edit note-by-note.
MIDI transmits symbolic musical instructions using a Status Byte () and Data Bytes (). Compared to Digital Audio, MIDI files are thousands of times smaller and trivial to transpose or re-orchestrate, but they cannot record speech or arbitrary sounds and depend on the local synthesizer for sound quality.
A recording studio captures of uncompressed stereo audio at sampling rate and .
- Calculate the uncompressed bit rate in .
- Calculate the total audio payload size in ().
- By how many does the theoretical of audio exceed that of CD audio?
Bit rate and payload size:
SQNR gain from to :
- Uncompressed bit rate .
- payload size .
- Theoretical increases by .
A mono telephone speech call is sampled at with -law PCM.
- Compute the uncompressed PCM file size in ().
- If the same recording is compressed into an audio file of using a speech codec, what is the compression ratio and the compressed bit rate in ?
Uncompressed -law PCM size:
Compression ratio and compressed bit rate:
- Uncompressed -law PCM size ().
- Compression ratio ; compressed bit rate .