PadhaikoAll subjects

Spectral Representation

Audio Processing (ENCT 389)

Open in the Padhaiko app

3.1 DFT and FFT: Derivation and computational complexity

Time-domain waveforms reveal amplitude variations over time but conceal which constituent frequencies are present. Spectral analysis transforms a finite block of time-domain samples into frequency-domain coefficients.

Definition

Discrete Fourier Transform (DFT) and Inverse DFT

Given a finite discrete-time audio frame of length (), the -point Discrete Fourier Transform (DFT) and its inverse (IDFT) are:

for , where is the principal -th root of unity (twiddle factor).

Derivation of the Radix-2 Decimation-in-Time (DIT) FFT

Computing all DFT bins directly requires complex multiplications and complex additions per bin, totaling complex multiplications (). The Fast Fourier Transform (FFT) exploits the periodicity () and symmetry () of to factor the DFT recursively when .

Splitting the DFT sum into even-indexed () and odd-indexed () samples for :

using the identity . Defining and as the -point DFTs of the even and odd subsequences (each periodic with period ), and applying , yields the Radix-2 FFT Butterfly equations for :

Theorem

Computational Complexity of Direct DFT vs. Radix-2 FFT

Recursively halving the -point DFTs down to -point butterflies produces stages, each executing butterflies ( complex multiplication and complex additions per butterfly):


3.2 Windowing function in audio

Taking an -point DFT implicitly treats the finite -sample slice as one period of an infinitely repeating periodic signal. Whenever a sinusoid does not complete an exact integer number of cycles inside the frame, discontinuities appear at the frame boundaries, smearing energy from that single tone across all DFT bins—an artifact called spectral leakage.

To suppress boundary discontinuities, the audio frame is multiplied element-wise by a tapered window function before applying the FFT:

By the convolution theorem, multiplying in time convolves the true spectrum with the window's spectrum . Every window faces an unavoidable trade-off between main-lobe width (frequency resolution) and sidelobe attenuation (leakage suppression).

Common Audio Window Functions and Spectral Characteristics
WindowFormula ()Main-Lobe WidthHighest SidelobeBest Audio Use
Rectangular (narrowest)Transient capture or bin-centered periodic test tones
Hann (Hanning)General-purpose STFT, spectrograms, overlap-add synthesis
HammingSpeech analysis and MFCC extraction (nonzero endpoints)
Blackman (widest)High-dynamic-range THD and harmonic distortion testing

3.3 Short time Fourier transform: Frame size, hope size and overlap

Because real-world music and speech evolve continuously over time, a single global DFT averages away when notes or phonemes occurred. The Short-Time Fourier Transform (STFT) analyzes time-varying audio by sliding a short window of length along the signal in fixed steps called the hop size (printed as "hope size" in the syllabus).

Definition

Short-Time Fourier Transform (STFT)

The discrete STFT of signal using analysis window of frame size and hop size is:

where is the time frame index and is the frequency bin index.

Time–Frequency Uncertainty Trade-off and COLA

By the Gabor–Heisenberg uncertainty principle, a window cannot be arbitrarily narrow in both time () and frequency () simultaneously ():

When modifying and resynthesizing audio via Inverse STFT (ISTFT), the shifted windows must satisfy the Constant Overlap-Add (COLA) condition (for example, a periodic Hann window with overlap or overlap ).


3.4 Spectrograms: Reading and interpreting time-frequency representation,

A spectrogram visualizes the STFT as a 2D heat map with time on the horizontal axis, frequency on the vertical axis, and log magnitude or power encoded by color or brightness:

Logarithmic (dB) scaling is essential because human hearing perceives amplitude logarithmically and weak upper harmonics would otherwise vanish next to strong fundamentals.

Visual Signatures in Audio Spectrograms
Visual Pattern on SpectrogramAcoustic / Musical PhenomenonPhysical Explanation
Horizontal parallel linesHarmonic pitched note (vowels, strings, brass)Energy concentrated at ; vertical line spacing
Broad horizontal dark/bright bandsVocal tract formants ()Resonances of the mouth/throat shaping vowel identity regardless of pitch
Sharp vertical linesPercussive transients (snare, click, plosives /p/, /t/)Short impulse-like energy distributed broadly across all frequencies at one instant
Granular high-frequency cloudUnvoiced fricatives (/s/, /sh/), cymbals, white noiseAperiodic turbulent energy lacking discrete harmonic striations
Wavy or sloping horizontal tracksVibrato, pitch bend, or glissandoContinuous modulation or slide in fundamental and harmonic frequencies over time

3.5 Mel spectrogram: Reading and interpreting; Mel frequency representation

A linear-frequency spectrogram assigns equal vertical pixel space to (a full musical octave) and (barely distinguishable to the ear). To align spectral analysis with human auditory perception, frequencies are warped onto the Mel scale, which models the logarithmic frequency selectivity of the cochlea's critical bands.

Definition

Mel Frequency Scale and Mel Filterbank

Using O'Shaughnessy's standard conversion between physical frequency () and Mel frequency :

Constructing and Interpreting a Mel Spectrogram

  1. Compute the STFT power spectrum .
  2. Choose lower and upper bounds (), convert them to Mels, space center frequencies () uniformly on the Mel axis, and convert those centers back to Hz to build overlapping triangular bandpass filters .
  3. Multiply and sum power across each triangular filter: , then take log energy (in dB).

Reading a Mel Spectrogram: Lower pitches and formants receive high visual resolution at the bottom of the plot, while high frequencies are smoothly compressed into fewer, wider bands. Consequently, Mel spectrograms provide compact, perceptually balanced representations widely used as input features for speech recognition, speaker identification, and audio classification CNNs.


3.6 Constant Q transform and its relevance to music analysis

While Mel spectrograms warp frequency bins after a fixed-window STFT, Western musical pitch follows an exact geometric (logarithmic) progression: in 12-tone equal temperament, adjacent semitones differ by a constant ratio (). A linear STFT has fixed , which blurs closely spaced bass notes together while wasting excessive bins at high octaves.

Definition

Constant-Q Transform (CQT)

Proposed by Judith Brown (1991), the Constant-Q Transform evaluates spectral bins at geometrically spaced center frequencies with bins per octave (typically or ):

Each bin uses its own frequency-dependent window length so that the quality factor (ratio of center frequency to bandwidth) remains constant across the entire spectrum:

Relevance to Music Analysis and Comparison with STFT / Mel

  1. Exact Musical Pitch Alignment: Every bin maps directly to a musical note or microtonal subdivision (e.g., yields one bin per piano key; yields bins per semitone to track vibrato/tuning).
  2. Transposition Invariance: Raising a chord or melody by semitones simply shifts its CQT magnitude pattern upward by bins without distorting harmonic spacing, making CQT ideal for chord recognition, chroma extraction, cover song detection, and polyphonic music transcription.
  3. Time–Frequency Trade-off in CQT: Because , low bass notes automatically receive long windows (high frequency precision to separate low notes), while high treble notes receive short windows (sharp time localization for transients and hi-hats).
Comparison of STFT, Mel Spectrogram, and Constant-Q Transform (CQT)
PropertySTFTMel SpectrogramConstant-Q Transform (CQT)
Frequency AxisLinear ()Mel-warped triangular bandsGeometric ()
Bandwidth Constant across all binsGrows roughly logarithmically Proportional to ()
Window LengthFixed for all binsFixed (inherited from STFT)Variable (long in bass, short in treble)
Primary ApplicationGeneral DSP, ISTFT editing, denoisingSpeech recognition, deep learning audio classificationMusic pitch tracking, chord/key estimation, chroma features

Worked Examples

Example 3.1

3.1

  1. Compute the -point DFT () of the audio sequence using the Radix-2 DIT FFT butterfly equations.
  2. Compare the number of complex multiplications required by a direct -point DFT versus a Radix-2 -point FFT.

Stage 1 of Radix-2 DIT FFT (2-point DFTs of even and odd samples): Split into even () and odd () subsequences:

Stage 2 butterfly combination (, twiddle factors , ):

Note the conjugate symmetry .

Complexity comparison for ():

  • Direct DFT: complex multiplications.
  • Radix-2 FFT: complex multiplications.
  • Speedup: .
Answer:
  1. .
  2. Direct -point DFT requires complex multiplications vs. for Radix-2 FFT ( fewer).
Example 3.2

3.2

An STFT analyzer processes a -second ( samples) audio clip sampled at using a Hann window of frame size samples and hop size samples (with no zero-padding). Calculate:

  1. The overlap in samples and percentage overlap.
  2. The frequency bin spacing () and time step () between consecutive frames.
  3. The total number of complete unpadded frames produced.

Overlap in samples and percentage:

(A overlapping Hann window satisfies the COLA condition for clean ISTFT reconstruction.)

Frequency spacing and frame time step :

Total number of unpadded frames :

Answer:
  1. Overlap samples ().
  2. , .
  3. complete frames.
Example 3.3

3.3

  1. Convert into Mel frequency using O'Shaughnessy's formula.
  2. A Constant-Q Transform (CQT) operates at with bins per octave (semitone resolution). Compute the quality factor , the bandwidth , and the required window length for the note A2 ().

Converting to Mels:

CQT quality factor for bins per octave:

Bandwidth and window length at :

Answer:
  1. .
  2. For : , , and samples at .

Exercises

Practice Exercises — Spectral Representation

Question 1
  1. Using the Radix-2 DIT FFT butterfly equations, find the -point DFT of .
  2. How many complex multiplications are saved when using a Radix-2 FFT instead of a direct DFT for ?

Stage 1 (2-point even/odd DFTs): Even samples , odd samples :

Stage 2 butterflies (, ):

Complexity comparison for ():

  • Direct DFT: complex multiplications.
  • Radix-2 FFT: complex multiplications.
  • Savings: multiplications ( speedup).
Answer:
  1. .
  2. Radix-2 FFT uses complex multiplications vs. for direct DFT, saving multiplications ( speedup).
Question 2

What causes spectral leakage when computing the DFT of an audio signal? Compare the Rectangular, Hann, and Blackman windows in terms of main-lobe width and sidelobe attenuation, and explain why zero-padding before an FFT does not improve physical frequency resolution.

Cause of spectral leakage: A finite -sample DFT assumes periodic repetition of the frame. If the signal's period does not divide evenly, the frame edges fail to match, creating sharp jump discontinuities. Equivalently, multiplying by a finite window convolves the true line spectrum with the window's sinc-like spectrum , spilling energy from sidelobes into adjacent frequency bins.

Window comparison:

  • Rectangular: Narrowest main lobe () for best two-tone separation, but worst sidelobe level ().
  • Hann: Moderate main lobe () with sidelobes and zero endpoints; ideal for COLA overlap-add STFT.
  • Blackman: Widest main lobe () among the three, trading resolution for deep sidelobe suppression.

Why zero-padding does not improve true resolution: Appending zeros extends the FFT length and places more interpolated bins () along the same underlying spectrum, but it adds zero new signal information. The main-lobe width—and thus the ability to separate two close sinusoids—is fixed solely by the non-zero window length and window shape.

Answer:

Spectral leakage arises from boundary discontinuities (implicit rectangular truncation) smearing energy into neighboring bins via the window's sidelobes. Rectangular offers the narrowest main lobe (, sidelobes), Hann balances width with sidelobes, and Blackman achieves sidelobes with a main lobe. Zero-padding only interpolates the spectrum visually and cannot narrow the window's main lobe.

Question 3

At , compare two STFT configurations: (A) narrowband with frame size and hop , versus (B) wideband with frame size and hop .

  1. Compute the frequency bin spacing and window duration for both configurations.
  2. For configuration (A), calculate the percentage overlap and frame rate (frames/sec).
  3. State which configuration is better suited for resolving low bass harmonics and which is better for locating drum transients.

Frequency spacing and window duration:

  • Config A (): , .
  • Config B (): , .

Overlap and frame step for Configuration A:

Application matching: Config A ( spacing) resolves closely spaced low-frequency bass harmonics accurately, whereas Config B ( window) pinpoints sharp drum transients in time without smearing.

Answer:
  1. Config A: , . Config B: , .
  2. Config A overlap , frame step ().
  3. Config A suits low-frequency harmonic resolution; Config B suits percussive transient localization.
Question 4

Explain how a Mel spectrogram differs from a linear STFT spectrogram and describe the steps to compute it. Also convert back to and convert to .

Linear vs. Mel spectrogram and pipeline: A linear spectrogram spaces bins uniformly in Hz (), over-representing high frequencies and compressing low octaves. A Mel spectrogram passes the STFT power spectrum through a bank of overlapping triangular bandpass filters uniformly spaced on the perceptual Mel scale, then applies logarithmic (dB) compression to each band's summed energy.

Mel-to-Hz conversion for :

Hz-to-Mel conversion for :

Answer:

A Mel spectrogram warps the STFT power spectrum through Mel-spaced triangular filters and log compression to mimic cochlear frequency resolution. Numerically, and .

Question 5

Explain why the Constant-Q Transform (CQT) is preferred over the STFT for musical pitch and chord analysis. For a CQT at with bins per octave (quarter-tone resolution), calculate and compare the window lengths required at middle C (C4, ) and three octaves higher (C7, ).

Why CQT excels for music: Musical pitches follow a geometric scale (). Because CQT centers bins geometrically and scales window length to hold constant, every octave receives the exact same number of bins, bass notes get narrow bandwidths so semitones don't blur, and musical transpositions become rigid vertical shifts.

Quality factor for bins/octave:

Window lengths at C4 () and C7 ():

Notice that raising pitch by octaves ( frequency) shrinks the window length by .

Answer:

CQT provides geometrically spaced bins and constant across all octaves, yielding uniform semitone spacing and transposition invariance. For at : , samples, and samples.

← Time Domain Audio ProcessingFrequency Domain and Non-Linear Effects (part 1: 4.1–4.3) →