PadhaikoAll subjects

Music Information Retrieval

Audio Processing (ENCT 389)

Open in the Padhaiko app

5.1 MIR overview: Types of music data, challenges, application

Music Information Retrieval (MIR) is the interdisciplinary field that extracts musically meaningful structure—such as melody, chords, tempo, instrumentation, mood, and identity—from music recordings and scores using digital signal processing and machine learning.

Types of Music Data

  1. Audio Signals (Acoustic / Waveform Data): Raw PCM waveforms (WAV, FLAC) and their time-frequency transforms (STFT spectrograms, Mel spectrograms, CQT). Rich in expressive nuance and timbre, but unstructured—notes and chords are overlaid in a single mix.
  2. Symbolic Music Data: Structured event representations such as MIDI (note-on/off, pitch number, velocity, timing), MusicXML, or ABC notation. Notes, voices, and tempo are explicit, though audio performance timbre is absent.
  3. Editorial, Cultural, and Multimodal Metadata: Artist/album tags, lyrics, chord sheets, listening histories, playlists, and social tags.

Key Challenges and Applications

Major Challenges and Real-World Applications of MIR
AspectCore Details
Polyphony & Source OverlapReal tracks layer vocals, drums, guitars, and synths simultaneously, causing overlapping harmonics that hinder single-note pitch tracking
Acoustic & Expressive VariabilityDifferent tempos, keys, live room acoustics, vibrato, and instrument timbres make two performances of the same song look completely different in raw waveforms
Subjectivity & Semantic GapLow-level DSP numbers (energy, bins) do not trivially map to high-level concepts like "melancholic," "jazz," or "catchy"
Core ApplicationsAudio fingerprinting (Shazam), music recommendation & mood/genre tagging, chord & key estimation, beat tracking, cover song identification, source separation (vocals/drums/bass), and query-by-humming

5.2 Spectral centroid and roll off

5.2.1 Basics of spectral centroid and spectral roll

Both Spectral Centroid and Spectral Roll-off summarize the shape and high-frequency balance of a single STFT frame (, bin frequencies ).

Definition

Spectral Centroid and Spectral Roll-Off

  1. Spectral Centroid (): The weighted mean frequency ("center of mass") of the magnitude spectrum, measured in :

Perceptually, spectral centroid correlates strongly with brightness (sharpness): instruments with strong upper harmonics (trumpet, cymbal, violin) have high centroids, whereas bass or mellow flute tones have low centroids.

  1. Spectral Roll-off (): The cutoff frequency below which a chosen proportion (commonly or ) of the frame's total spectral energy (or magnitude sum) is concentrated:

where is the smallest bin index satisfying the inequality, giving .

5.2.2 Comparison between centroid and roll off

AspectSpectral CentroidSpectral Roll-off
Mathematical MeaningFirst moment (mean / center of gravity) of the spectrumQuantile cutoff frequency (-th percentile of cumulative spectral energy)
SensitivityWeighted linearly by frequency ; sensitive to isolated high-frequency peaksMeasures upper tail bandwidth; robust to exact shape below cutoff
Physical InterpretationPerceptual brightness of timbreSpectral skewness / effective upper bandwidth of the sound
Voiced vs. Unvoiced BehaviorLow for voiced vowels, jumps moderately for bright harmonicsLow for voiced speech, jumps sharply upward for unvoiced fricatives and cymbals

5.2.3 Application (Timbre analysis, music/ speech classification)

  1. Timbre Analysis: Because brightness separates instruments playing identical pitch and loudness (e.g., muted vs. open brass, acoustic vs. distorted guitar), frame-wise mean and standard deviation of and serve as primary timbre descriptors in instrument recognition.
  2. Music / Speech Classification: Human speech alternates rapidly between low-centroid voiced vowels and high-centroid unvoiced consonants (/s/, /sh/) separated by brief silences, producing high frame-to-frame variance in both spectral centroid and roll-off. Music typically exhibits steadier, sustained harmonic energy with lower variance in centroid and roll-off over windows.

5.3 Mel frequency cepstral coefficient (MFCC)

5.3.1 Motivation for ceptral features

Under the Source–Filter Model of speech and acoustic instruments, an excitation signal (glottal pulses or violin bow slip-stick) passes through a resonant body/vocal-tract filter :

Taking the logarithm converts multiplication into addition:

Because the vocal tract/instrument resonance envelope varies slowly across frequency while harmonic excitation ripples rapidly every , applying a second frequency transform (DCT or IDFT) moves into the quefrency domain and neatly separates the smooth timbre/phoneme envelope (low-order cepstral coefficients) from pitch fine structure (high quefrency).

5.3.2 Steps in MFCC

Definition

Standard MFCC Extraction Pipeline

Computing Mel-Frequency Cepstral Coefficients (MFCCs) proceeds through seven sequential stages:

  1. Pre-Emphasis: Apply a first-order high-pass filter () to flatten natural spectral roll-off and boost high-frequency formants.
  2. Framing & Windowing: Split into frames (hop ) and multiply by a Hamming window to minimize spectral leakage.
  3. FFT Power Spectrum: Compute (or ).
  4. Mel Filterbank: Pass through () overlapping triangular bandpass filters uniformly spaced on the Mel scale , obtaining band energies .
  5. Log Compression: Take (mimics logarithmic human loudness perception and converts source-filter product into a sum).
  6. Discrete Cosine Transform (DCT-II): Decorrelate the overlapping Mel log-energies into cepstral coefficients:
  1. Coefficient Selection & Deltas: Keep or (often with ), optionally append (first temporal derivative) and (acceleration) features, and apply sinusoidal liftering.

5.3.3 Interpretation of coefficient

5.3.4 Application (Speech recognition, music classification)

  1. Automatic Speech Recognition (ASR) & Speaker Identification: Because MFCCs isolate vocal-tract phoneme shape () independent of pitch and loudness, they have long served as baseline acoustic features for HMM/GMM and neural speech recognizers and voice biometrics.
  2. Music Classification: Used widely in music genre classification, instrument identification, mood tagging, and audio similarity clustering because low-order MFCCs compactly summarize timbral texture.

5.4 Chroma

While MFCCs deliberately discard pitch to capture timbre, many musical tasks—such as chord recognition and cover-song matching—require features that capture harmonic pitch content while remaining invariant to timbre and octave register.

5.4.1 Concept of pitch classes (12- tone system) mapping

In human music perception, notes separated by an integer number of octaves () share a strong perceptual equivalence called octave equivalence (e.g., C2, C3, C4, and C5 all sound like the note "C"). Folding the 12-tone equal-tempered chromatic scale across all octaves produces 12 pitch classes (chroma):

Definition

Frequency-to-Pitch-Class (Chroma) Mapping

Any frequency () maps to its continuous MIDI note number relative to concert A4 ():

Rounding to the nearest semitone yields the 12-tone pitch class index :

Every spectral bin or Constant-Q bin belonging to pitch class (across all octaves) sums its energy into the -th chroma bin of a -element vector .

5.4.2 Chromagram representation,

A chromagram (or Pitch Class Profile, PCP, matrix) stacks the normalized -dimensional chroma vectors across time frames , forming a time–chroma representation:


5.5 Onset and tempo and their applications

Beyond pitch and timbre, music is organized in time by note onsets, beats, and tempo.

Definition

Onset Detection Function (Spectral Flux) and Tempo (BPM)

  1. Onset: The exact start instant of a musical note or percussive hit (attack transient). An Onset Detection Function (ODF) measures sudden upward changes in signal energy or spectral content—most commonly Half-Wave Rectified Spectral Flux:

Using triggers only on rising spectral energy (attacks), ignoring note decays. Local peaks above an adaptive moving threshold identify onset timestamps.

  1. Tempo: The perceived rate of musical beats (pulse), expressed in Beats Per Minute (BPM). If the dominant inter-beat period is seconds (or lag frames at frame rate ):

Tempo is commonly estimated via autocorrelation of the ODF, comb-filterbank resonance, or Fourier tempograms.

Applications of Onset and Tempo Analysis


Worked Examples

Example 5.1

5.1

A simplified -bin spectrum of an audio frame has frequencies with corresponding FFT magnitudes . Compute:

  1. The Spectral Centroid in .
  2. The Spectral Roll-off using the standard energy threshold.

Spectral Centroid :

Spectral Roll-off ( of total spectral energy): Compute bin energies :

Threshold energy . Cumulative energy by bin:

  • Up to :
  • Up to : (threshold reached!)
Answer:
  1. Spectral Centroid .
  2. Spectral Roll-off () is .
Example 5.2

5.2

  1. Map the two spectral peak frequencies and to their MIDI note numbers and 12-tone pitch classes (with , , ).
  2. If a frame's unnormalized -bin chroma vector has non-zero entries and (all others ), compute the -normalized chroma vector.

MIDI note and pitch class for :

MIDI note and pitch class for :

normalization of the chroma vector:

Hence the normalized chroma vector already has unit norm: , , all others .

Answer:
  1. (); ().
  2. -normalized entries: , .
Example 5.3

5.3

An onset and tempo tracker runs STFT frames at with hop size samples.

  1. Two consecutive -bin magnitude frames are and . Compute the Half-Wave Rectified Spectral Flux .
  2. If the ODF autocorrelation shows a strong beat periodicity peak at lag frames, estimate the track's tempo in BPM.

Half-Wave Rectified Spectral Flux :

Estimating tempo in BPM: Frame rate is . Lag frames gives beat period :

Answer:
  1. .
  2. Estimated tempo .

Exercises

Practice Exercises — Music Information Retrieval

Question 1

List the three major types of music data used in MIR, state three core technical challenges in MIR, and describe four practical MIR applications.

Three types of music data:

  1. Audio Waveforms & Time-Frequency Representations (PCM, STFT, Mel, CQT).
  2. Symbolic Representations (MIDI, MusicXML, scores).
  3. Metadata & Contextual Data (artist/genre tags, lyrics, user listening histories).

Three core MIR challenges:

  1. Polyphonic source overlap (multiple instruments share overlapping harmonics in one mix).
  2. Expressive & acoustic variability (tempo rubato, key transpositions, room reverb, timbre differences).
  3. Semantic gap & subjectivity (bridging low-level DSP numbers to subjective mood, genre, and similarity).

Four applications: Audio fingerprinting (song ID), music recommendation, automatic chord/beat tracking, and query-by-humming / cover song detection.

Answer:

MIR uses audio, symbolic (MIDI/scores), and metadata representations; key challenges include polyphonic overlap, acoustic/performance variability, and the semantic gap; applications span fingerprinting, recommendation, chord/beat extraction, and cover song identification.

Question 2

An audio frame has spectral bins at frequencies with magnitudes . Calculate:

  1. The Spectral Centroid .
  2. The Spectral Roll-off at energy threshold, and briefly state how centroid and roll-off help separate music from speech.

Spectral Centroid:

Spectral Roll-off (): Total energy . threshold . Cumulative energy: (), (), (). Thus .

Music vs. speech separation: Speech alternates rapidly between low-centroid voiced vowels and high-centroid unvoiced fricatives/pauses, producing much higher frame-to-frame variance in centroid and roll-off than sustained musical passages.

Answer:
  1. .
  2. ; higher temporal variance in centroid and roll-off distinguishes speech from steadier music.
Question 3

Explain the source–filter motivation behind cepstral features, list the seven steps to compute MFCCs, and interpret what , , and represent.

Source–filter motivation: Speech/audio is , so . Taking turns multiplication into addition, and DCT separates the slowly varying spectral envelope (vocal tract / body timbre) into low-index cepstral coefficients away from fast pitch harmonics.

Seven MFCC steps: (1) Pre-emphasis filter, (2) framing and Hamming windowing, (3) FFT power spectrum, (4) triangular Mel filterbank summation, (5) log compression, (6) DCT-II decorrelation, and (7) selecting low coefficients (plus optional deltas).

Interpretation of coefficients: measures overall log-energy; reflects spectral tilt (bass vs. treble balance); capture formant and timbral envelope curvature while remaining largely pitch-invariant.

Answer:

Logarithm + DCT separates the smooth vocal/instrument filter envelope from harmonic pitch excitation via: pre-emphasis framing/Hamming FFT Mel filterbank log DCT truncation. encodes energy, spectral slope, and broad spectral timbre.

Question 4

Explain octave equivalence and how a chromagram represents 12-tone harmonic content. Verify mathematically that and map to the exact same pitch class index despite lying two octaves apart.

Octave equivalence and chromagram representation: Pitches separated by powers of two in frequency () are perceived as the same note name (octave equivalence). A chromagram folds all spectral energy across octaves into normalized pitch-class bins () over time, revealing chords and keys regardless of octave voicing or instrument timbre.

Pitch-class mapping for and :

Answer:

A chromagram folds multi-octave spectral energy into pitch classes (–) per frame for timbre- and octave-invariant harmony analysis. Both (MIDI ) and (MIDI ) yield (pitch class A).

Question 5

Why does Spectral Flux use half-wave rectification for onset detection? Also, at with hop size , if an ODF autocorrelation peak occurs at lag frames, calculate the estimated tempo in BPM.

Role of half-wave rectification in Spectral Flux: Note attacks produce a sudden increase in spectral magnitude (), whereas note releases/decays produce negative differences. Half-wave rectification keeps only positive magnitude rises and zeros out decays so note offsets never trigger false onset peaks.

Tempo calculation: Frame rate .

Answer:

Half-wave rectification suppresses negative magnitude changes so only rising attack transients trigger onset peaks. For , , and frames, the estimated tempo is .

← Frequency Domain and Non-Linear Effects (part 2: 4.4–4.6)Audio system and DAW Integration →