PadhaikoAll subjects

Frequency Domain and Non-Linear Effects (part 1: 4.1–4.3)

Audio Processing (ENCT 389)

Open in the Padhaiko app

4.1 Sinusoidal model

Many musical and speech sounds—such as bowed strings, brass instruments, and voiced vowels—consist primarily of slowly varying periodic oscillations. Instead of storing raw time-domain samples or rigid fixed-grid STFT bins, the sinusoidal model (introduced by McAulay and Quatieri for speech and extended by Serra for music) represents a sound as a sum of time-varying sinusoidal partials (tracks) whose amplitudes, frequencies, and phases evolve smoothly over time.

Definition

Additive Sinusoidal Signal Model

In continuous time, the sinusoidal model approximates an audio signal as a sum of active partials:

where is the instantaneous amplitude envelope of the -th partial and is its instantaneous phase, obtained by integrating its instantaneous frequency :

In discrete-time frame processing (frame index , hop size ), parameters are estimated once per frame and smoothly interpolated across samples .

Analysis–Transformation–Synthesis Pipeline

  1. Windowed STFT and Zero-Padding: Each frame is windowed (commonly Blackman-Harris or Kaiser for deep sidelobe suppression) and zero-padded () so spectral peaks are finely sampled.
  2. Spectral Peak Detection & Parabolic Interpolation: Local maxima in the log-magnitude spectrum above a noise threshold are refined beyond bin spacing by fitting a parabola through the peak bin and its two neighbors , :
  1. Partial Tracking (Peak Continuation): Peaks in frame are linked to peaks in frame within a maximum frequency-jump tolerance ("guides" for harmonic series ), creating continuous sinusoidal tracks with clean birth and death ramps.
  2. Modification & Resynthesis:
    • Time-scaling: Stretch or compress the frame time axis without changing (alters duration without changing pitch).
    • Pitch-shifting: Multiply by a pitch factor without changing frame timestamps.
    • Synthesis: Use either a time-domain oscillator bank (linear amplitude interpolation, cubic polynomial phase interpolation across samples to avoid phase discontinuities) or fast Inverse-FFT (IFFT) synthesis.

4.2 Deterministic plus residual model

To overcome the limitation of pure sinusoidal analysis on mixed pitched-and-noisy audio, the Deterministic + Residual Model splits any audio signal into two complementary parts: a predictable deterministic (sinusoidal) component and an unpredictable residual component .

Definition

Deterministic Plus Residual Decomposition

The signal is modeled as:

where:

How the Residual Is Extracted and What It Contains

Because requires phase-accurate cancellation of the sinusoids, the synthesizer must match the exact amplitude, frequency, and instantaneous phase of every tracked partial in —otherwise, residual energy will still contain leftover sinusoidal leakage. Alternatively, spectral subtraction synthesizes the windowed sinusoidal spectrum using the analysis window's main-lobe shape and subtracts it directly from before an ISTFT.

The resulting residual captures:

  1. Stochastic / turbulent noise: Breath noise in flutes and vocals, bow-hair scrape in strings, snare wire sizzle.
  2. Sharp percussive transients: Drumstick strikes, piano hammer impacts, and consonant plosives whose rapid energy burst violates frame-level sinusoidal stationarity.
  3. Residual analysis error: Weak partials below the peak-picking threshold.

Strengths and Remaining Drawback: Separating and preserves natural breathiness and transient punch during mild audio edits and guarantees exact reconstruction when and are re-summed untouched. However, because is still a raw waveform (not parametrically modeled), directly time-stretching by repeating frames introduces audible metallic periodicity, motivating the Deterministic + Stochastic model.


4.3 Deterministic plus stochastic model

Proposed by Xavier Serra and Julius O. Smith (1990) as Spectral Modeling Synthesis (SMS), the Deterministic + Stochastic Model replaces the raw waveform residual with a compact parametric stochastic (noise) model .

Definition

Deterministic Plus Stochastic Model (SMS)

An audio signal is decomposed into deterministic sinusoids plus time-varying filtered white noise:

where is unit-variance white noise and is a slowly time-varying shaping filter whose frequency response matches the smooth spectral envelope of the residual in frame .

Stochastic Analysis and Synthesis Steps

  1. Residual Calculation: Subtract the phase-matched deterministic sinusoids from to obtain residual .
  2. Spectral Envelope Approximation: Compute the residual magnitude spectrum and smooth out rapid bin-to-bin random fluctuations to extract a low-dimensional stochastic spectral envelope using:
    • Piecewise-linear interpolation across frequency bands (line-segment approximation),
    • Cepstral smoothing (low-pass liftering the log magnitude spectrum), or
    • Linear Predictive Coding (LPC) (fitting an all-pole filter ). Fine-grained phase of is discarded—human hearing is insensitive to the exact micro-phase of broadband noise.
  3. Stochastic Synthesis: For each synthesis frame , generate fresh uncorrelated white noise (random phase ), multiply its spectrum by the smoothed envelope , and overlap-add via ISTFT (or pass white noise through the time-varying LPC filter).
Comparison of Spectral Audio Models (4.1–4.3)
ModelRepresentationReconstructionHandling of Breath / NoiseTime-Stretching Quality
Sinusoidal ModelSinusoidal tracks onlyApproximate (loses noise)Poor ("phasiness" artifacts)Good for pure tones; poor for noisy/percussive audio
Deterministic + ResidualSinusoids + raw difference Exact ()Exact when unmodifiedRaw shows metallic repetition under heavy stretching
Deterministic + Stochastic (SMS)Sinusoids + smoothed noise envelope Perceptually equivalentNatural (freshly synthesized shaped noise)Excellent for both harmonic tones and smooth breath/friction

Worked Examples

Example 4.1

4.1

In a -point zero-padded FFT at for sinusoidal peak analysis, a local maximum occurs at bin with log-magnitude values:

Apply parabolic interpolation to estimate the refined frequency () and peak amplitude ().

Fractional bin offset :

Refined frequency : With bin spacing :

(Without interpolation, bin alone would report .)

Refined peak amplitude :

Answer:

Refined bin offset , estimated frequency , and refined peak level .

Example 4.2

4.2

At , a -sample audio segment () contains a tracked sinusoid plus unmodeled residual noise:

  1. Compute the synthesized deterministic samples for when the model accurately recovers , , .
  2. Verify the residual and compute the residual-to-signal energy ratio .

Synthesizing the deterministic sinusoidal component : Normalized angular frequency is :

Extracting the residual : Given , phase-matched subtraction cancels the deterministic sinusoid completely:

Computing deterministic energy , residual energy , and their ratio:

Answer:
  1. .
  2. , with ().
Example 4.3

4.3

Explain how to design an exam-ready answer comparing Deterministic + Residual versus Deterministic + Stochastic (SMS) modeling when asked to describe how a flute tone with audible breath noise is time-stretched to twice its original duration ().

Decomposing the signal in both models: A flute tone combines a harmonic pitched body (fundamental and integer harmonics) with broadband turbulent air noise from the embouchure hole.

  • Both models extract the pitched harmonics into via STFT peak picking, parabolic refinement, and partial tracking, then subtract from to isolate the breath residual .

Comparing time-stretching () behavior:

  • Deterministic + Residual: Time-stretching simply spaces the parameter frames twice as far apart while oscillating at unchanged frequencies —clean and pitch-invariant. However, stretching the raw waveform by overlap-adding or repeating grains creates periodic comb-like buzzes in the breath noise.
  • Deterministic + Stochastic (SMS): Rather than keeping raw , SMS extracts its smooth spectral magnitude envelope (via cepstral/LPC/piecewise smoothing) and discards its random phase. When stretching , SMS interpolates across double the time frames and filters freshly generated uncorrelated white noise, yielding seamless, artifact-free breathiness alongside crisp harmonics.
Answer:

In Deterministic + Residual, the raw residual waveform introduces periodic artifacts if stretched; in Deterministic + Stochastic (SMS), modeling the residual by its smooth spectral envelope driven by fresh white noise allows the flute's harmonics and breath noise to both stretch smoothly by without metallic ringing.


Exercises

Practice Exercises — Sinusoidal & Deterministic-Stochastic Models

Question 1

State the mathematical formulation of the sinusoidal model and explain the four stages of its analysis–transformation–synthesis pipeline. Why is phase accuracy critical during synthesis?

Mathematical formulation:

Four pipeline stages:

  1. Windowed Zero-Padded STFT: Compute high-resolution spectra per frame.
  2. Peak Picking & Parabolic Interpolation: Locate local maxima and refine sub-bin frequencies and amplitudes.
  3. Peak Continuation (Tracking): Link peaks across consecutive frames into continuous birth-to-death partial tracks.
  4. Transformation & Synthesis: Modify (pitch) or frame spacing (time) and reconstruct via an oscillator bank or IFFT synthesis.

Importance of phase continuity: Phase jumps between consecutive hop frames introduce waveform discontinuities that are heard as clicks, roughness, or buzz. Cubic phase interpolation guarantees both phase and instantaneous frequency match smoothly at frame boundaries.

Answer:

The sinusoidal model sums time-varying partials through zero-padded STFT, parabolic peak estimation, frame-to-frame peak tracking, and phase-continuous synthesis; matching instantaneous phase and frequency across hops prevents audible clicks and roughness.

Question 2

In a -point FFT at , a spectral peak is located at bin with neighbors:

Estimate the true partial frequency () using parabolic interpolation.

Compute fractional bin shift :

Convert refined bin index to Hz: With :

Answer:

bins, giving refined frequency .

Question 3

Compare the Deterministic + Residual model with the Deterministic + Stochastic (SMS) model. Explain what information is retained in the residual, how SMS approximates it, and why SMS is superior for time-stretching.

Deterministic + Residual vs. Deterministic + Stochastic (SMS):

  • Deterministic + Residual (): Keeps the exact waveform difference , preserving breath noise, friction, and attack transients for bit-exact reconstruction, but leaves unparameterized.
  • Deterministic + Stochastic (): Replaces raw with a smoothed spectral magnitude envelope (via line-segment approximation, cepstral smoothing, or LPC) that shapes freshly generated white noise.

Why SMS outperforms for time-stretching: Stretching a raw residual directly repeats or overlaps noise grains, introducing artificial periodicity. SMS generates new uncorrelated random phases on every synthesis frame while stretching only the smooth spectral envelope, keeping noise textured and artifact-free at arbitrary stretch factors.

Answer:

Deterministic + Residual keeps the exact residual waveform , whereas SMS parameterizes the residual's smooth spectral envelope and drives it with fresh white noise, avoiding metallic grain repetition during time-stretching.

Question 4

A sinusoidal synthesis frame of hop size samples at updates a partial's amplitude from to using linear interpolation:

  1. Compute the amplitude at the midpoint and at .
  2. If an SMS file stores sinusoidal tracks ( at bits each) plus stochastic envelope coefficients ( bits each) per frame at , calculate the approximate parametric bitrate in and compare it to -bit mono PCM at .

Linear amplitude interpolation: At (): . At (): .

Parametric bitrate vs. PCM bitrate: Using the frame rate :

  • Parameters per frame: values/frame.
  • Bits per frame: .
  • SMS bitrate: .
  • Mono PCM bitrate: ( compression).
Answer:
  1. , .
  2. Parametric SMS bitrate compared to for -bit mono PCM ( reduction).
Question 5

Why does a Deterministic + Stochastic synthesizer often add a separate Transient component (forming a Sines + Transients + Noise model) for percussive instruments like piano or acoustic guitar?

Why transients break both Sines and Noise:

  1. Sharp attacks (hammer strikes, plucks) are localized impulses lasting only a few milliseconds—too brief to form coherent sinusoidal tracks across STFT frames, so sinusoidal trackers either miss them or smear them across the window length.
  2. Putting transients into the stochastic noise model preserves their spectral energy only as a blurred envelope with random phase, destroying the crisp phase-aligned snap of the attack ("pre-echo" or washed-out pluck).

Role of explicit transient separation: By detecting transient frames (e.g., via spectral flux or energy bursts) and preserving their short time-domain waveform or phase intact—moving them as rigid unstretched grains during time-scaling—the Sines + Transients + Noise model keeps attacks punchy while stretching sustained notes and noise smoothly.

Answer:

Attack transients violate both slow sinusoidal stationarity and stationary random-phase noise assumptions; isolating them into a third transient component prevents smearing and pre-echo during time-scaling and pitch modifications.

← Spectral RepresentationFrequency Domain and Non-Linear Effects (part 2: 4.4–4.6) →