PadhaikoAll subjects

Recent Trend

Audio Processing (ENCT 389)

Open in the Padhaiko app

7.1 Machine learning in audio

Classical audio engineering relied on hand-crafted signal descriptors—such as MFCCs, ZCR, Spectral Centroid, and Chroma—fed into statistical models like Gaussian Mixture Models (GMMs), Hidden Markov Models (HMMs), and Support Vector Machines (SVMs). Modern audio intelligence combines domain-aware time–frequency representations with deep neural networks that learn hierarchical representations directly from data.

Evolution of Machine Learning Architectures in Audio Processing
ParadigmTypical InputCore MechanismPrimary Strengths & Audio Tasks
Classical ML (GMM / SVM / HMM)Hand-crafted MFCC, Chroma, ZCRStatistical decision boundaries or sequence statesLow compute, works on small datasets (speaker ID, basic chord/beat tracking)
2D CNNsLog-Mel Spectrogram / CQT2D convolution filters over time & frequencyShift-tolerant local pattern detection (acoustic event detection, genre/instrument tagging)
1D CNNs & Neural CodecsRaw PCM waveformLearned time-domain filterbanks (e.g., WaveNet, Demucs)Preserves phase information for source separation and neural synthesis
RNN / LSTM / BiLSTMFrame sequence (MFCC / Mel)Recurrent hidden state across time stepsCaptures temporal context in speech recognition (ASR), onset detection, and melody tracking
Transformers & SSLSpectrogram patches or raw audio framesMulti-head self-attention + masked pretraining (AST, wav2vec 2.0, HuBERT)Global context modeling and strong performance with scarce labeled data

Core Audio ML Tasks


7.2 Audio foundation models for bioacoustic and healthcare

An audio foundation model is a large neural encoder pretrained via self-supervised learning (e.g., masked spectrogram reconstruction in AudioMAE, contrastive language–audio alignment in CLAP) on millions of diverse unlabeled audio recordings. Once pretrained, its learned embeddings generalize across niche tasks with minimal task-specific labels (few-shot learning or linear probing).

1. Bioacoustics Applications

Ecological monitoring increasingly relies on Passive Acoustic Monitoring (PAM)—autonomous weatherproof microphones left in rainforests, oceans, and wetlands for months:

2. Healthcare Applications

Human physiology generates rich acoustic biomarkers—subtle changes in vocal fold vibration, airflow turbulence, and valve mechanics—that foundation models (such as Google's HeAR — Health Acoustic Representations) capture non-invasively:


7.3 Spatial audio

Traditional mono ( channel) and stereo ( channels) flatten sound onto a horizontal line between two speakers. Spatial audio recreates a full three-dimensional ( azimuth , elevation , and distance ) soundfield for headphones, cinema arrays, virtual reality (VR), and gaming.

Psychoacoustic Cues of Human 3D Localization

  1. ITD (Interaural Time Difference): Sound from the right reaches the right ear earlier than the left ear by up to (dominant below ). Using Woodworth's spherical-head approximation (head radius , speed , angle from front):
  1. ILD (Interaural Level Difference): Above , wavelengths become shorter than head diameter, casting an acoustic shadow that makes sounds quieter at the farther ear.
  2. Pinna & Torso Spectral Filtering: The asymmetric folds of the outer ear (pinna) introduce angle-dependent spectral notches () that resolve elevation (up vs. down) and front–back ambiguity.

Three Major Spatial Audio Technologies

Comparison of Spatial Audio Representations
ApproachCore PrincipleChannel / Data ModelKey Use Cases
Binaural Audio (HRTF)Convolves mono sources with left/right Head-Related Transfer Functions capturing ITD, ILD, and pinna cuesAlways outputs channels (L/R) tailored specifically for headphonesVR/AR headsets, 3D podcasts, headphone spatialization
AmbisonicsDecomposes the 3D soundfield into orthogonal spherical harmonics; order uses channelsFirst-Order (): channels (); speaker-layout independent & smoothly rotatable VR video (with IMU head-tracking), acoustic field capture
Object-Based AudioCombines bed channels with independent audio objects carrying 3D coordinates and size metadataE.g., Dolby Atmos (up to tracks/objects), MPEG-H; rendered live to target room speakersCinema, Apple Spatial Audio, home theater (), gaming

7.4 Audio DSP ICs

While general-purpose CPUs handle DAW mixing on laptops, battery-powered devices—smartphones, Bluetooth TWS earbuds, hearing aids, smart speakers, and automotive amplifiers—rely on dedicated Audio DSP Integrated Circuits (ICs) for deterministic, ultra-low-latency, power-efficient signal processing.

Definition

Key Architectural Features of Dedicated Audio DSP ICs

Unlike general-purpose microcontrollers, Audio DSP ICs (such as Analog Devices SHARC / SigmaDSP, Texas Instruments C55x/C67x, and Cadence Tensilica HiFi cores) optimize silicon specifically for inner-product loops :

  1. Single-Cycle Hardware MAC & SIMD Units: Execute Multiply-Accumulate () in a single clock cycle, often with multiple parallel MAC lanes.
  2. Modified Harvard Architecture: Separate program and dual data memory buses allow fetching an instruction, a filter coefficient , and a sample simultaneously in one cycle.
  3. Zero-Overhead Hardware Looping & Circular Buffers: Dedicated address generators wrap ring-buffer pointers automatically for FIR filters and delay lines without conditional branch penalties, plus bit-reversed addressing for fast in-place FFTs.
  4. Guard Bits & Saturation Arithmetic: Wide accumulators (e.g., -bit or -bit for -bit audio, or -bit IEEE floating-point) prevent wrap-around overflow distortion ("digital clicks") via hardware saturation clamping.
  5. Native Audio Peripherals: Built-in , TDM, PDM microphone interfaces, hardware Sample Rate Converters (ASRCs), and ultra-low-power always-on wake-word / VAD accelerators ( standby).

Common embedded Audio DSP workloads include Active Noise Cancellation (ANC) in headphones (sub- latency), Acoustic Echo Cancellation (AEC) and beamforming in smart speakers, hearing-aid multiband compression, and crossover/room EQ in powered studio monitors.


7.5 Generative AI in music

Generative AI in music synthesizes original audio waveforms, accompaniment stems, or symbolic MIDI compositions conditioned on text descriptions ("upbeat acoustic folk guitar at 110 BPM"), melody hums, or style references.

Major Generative Architectures

  1. Autoregressive Audio Token Transformers (e.g., MusicLM, MusicGen):
    • A neural audio codec (SoundStream or EnCodec) compresses raw audio frames (e.g., at frame rate) into parallel streams of discrete integer tokens using Residual Vector Quantization (RVQ)—where each codebook quantizes the error left by the previous codebook.
    • A Transformer decoder predicts future audio tokens autoregressively, conditioned on text embeddings (from T5 or CLAP) and optional melody chromagrams, before the codec decoder reconstructs the PCM waveform.
  2. Latent Diffusion & Flow-Matching Models (e.g., AudioLDM, Stable Audio):
    • Compresses waveforms or Mel spectrograms into a continuous VAE latent space.
    • During inference, starts from pure Gaussian noise and iteratively denoises the latent representation over steps guided by text cross-attention and timing embeddings (start/total duration), then decodes to stereo audio.
  3. Symbolic Generation & Hybrid DDSP:
    • Symbolic Transformers generate editable MIDI scores (notes, chords, velocities), while Differentiable DSP (DDSP) integrates trainable oscillators, filters, and reverbs directly into neural networks for expressive, lightweight instrument synthesis.

Technical, Ethical, and Legal Challenges


Worked Examples

Example 7.1

7.1

  1. Using Woodworth's spherical-head formula with head radius () and speed of sound , calculate the Interaural Time Difference (ITD) in milliseconds for a sound source directly to the right ().
  2. Compare the required number of audio channels for First-Order () versus Third-Order () 3D Ambisonics.

Interaural Time Difference at :

(At , this equals samples of delay between left and right ears.)

Ambisonics channel count :

  • First-Order Ambisonics (): channels ( — -format).
  • Third-Order Ambisonics (): channels (sharper spatial directivity and wider sweet spot).
Answer:
  1. Maximum at is .
  2. First-Order Ambisonics requires channels; Third-Order requires channels.
Example 7.2

7.2

An embedded Audio DSP IC clocked at executes MAC per clock cycle and processes stereo (-channel) audio at . Each channel passes through a -tap FIR filter ( MACs/sample) followed by cascaded biquad IIR sections ( MACs/sample per biquad).

  1. Calculate the total MMACs/s (million MACs per second) required and the percentage of DSP clock capacity consumed.
  2. What is the smallest positive non-zero value representable in signed -bit Q1.15 fixed-point arithmetic ( sign bit, fractional bits)?

Total MACs per sample and MMACs/s:

  • MACs per mono sample .
  • For channels at :

DSP clock utilization: With capacity:

(Leaving headroom for additional reverb, limiter, or control logic, or allowing deep clock-gating for battery savings.)

Smallest positive step in signed Q1.15 fixed-point: In Q1.15, bits represent fractional powers down to , so equals:

(Representable range is .)

Answer:
  1. Required throughput ( of the DSP capacity).
  2. Smallest positive Q1.15 step .
Example 7.3

7.3

  1. A generative music model uses an RVQ neural audio codec operating at a frame rate of with residual codebooks per frame, each having vocabulary size ( bits/code). Calculate the discrete token bitrate in and the compression ratio compared to mono PCM.
  2. Outline how to answer an exam question comparing Audio Foundation Models in bioacoustics versus healthcare.

Computing RVQ neural codec bitrate and compression ratio:

  • Bits per frame .
  • Token bitrate .
  • mono PCM bitrate .
  • Compression ratio .

Exam comparison — Foundation models in bioacoustics vs. healthcare:

  • Common Backbone: Both pretrain large self-supervised encoders (e.g., masked spectrogram reconstruction / contrastive audio-text) on massive unlabeled audio, enabling accurate fine-tuning with very small labeled datasets.
  • Bioacoustics: Passive acoustic sensors monitor ecosystems for species classification (birds, whales), biodiversity indexing, and illegal logging/poaching; challenges include overlapping animal choruses and weather/wind noise.
  • Healthcare: Acoustic biomarkers from coughs, lung/heart auscultation, and voice screen for TB, COPD, heart murmurs, and Parkinson's; challenges center on patient privacy, on-device edge execution, device calibration, and clinical safety.
Answer:
  1. RVQ token bitrate ( compression vs. mono PCM).
  2. Foundation models transfer pretrained acoustic embeddings to label-scarce bioacoustic monitoring (species/conservation under outdoor noise) and medical screening (cough/heart/voice biomarkers under strict privacy and clinical validation constraints).

Exercises

Practice Exercises — Recent Trends in Audio Processing

Question 1

Compare classical machine learning (MFCC + GMM/SVM) with modern deep learning (2D CNNs on spectrograms, RNNs, and Self-Supervised Audio Transformers) in audio processing, stating one suitable application for each.

Classical ML vs. Deep Learning:

  • Classical ML (MFCC + GMM/SVM): Relies on manually engineered features (MFCC, ZCR, Spectral Centroid); lightweight, interpretable, and effective on small datasets (e.g., embedded voice command or simple speaker verification).
  • 2D CNNs on Mel/CQT Spectrograms: Automatically learn shift-invariant local time–frequency filters; ideal for environmental sound tagging and instrument recognition.
  • RNNs / LSTMs & Self-Supervised Transformers (wav2vec 2.0 / AST): Model long-range sequential dependencies and leverage unlabeled pretraining; dominate ASR, music source separation, and audio captioning.
Answer:

Classical ML pairs hand-crafted MFCCs with GMM/SVM for small-data or low-power tasks, whereas 2D CNNs, RNNs, and Self-Supervised Transformers learn hierarchical time–frequency representations end-to-end for ASR, sound event detection, and source separation.

Question 2

What is an audio foundation model? Explain two applications in bioacoustics and two in healthcare, and list two deployment challenges unique to clinical audio AI.

Audio foundation model: A large audio encoder pretrained via self-supervised or contrastive objectives on massive diverse audio corpora so its representations adapt rapidly to downstream tasks with minimal labeled data.

Bioacoustics & Healthcare applications:

  • Bioacoustics: (1) Automated wildlife species classification & biodiversity tracking via passive acoustic sensors; (2) real-time acoustic detection of chainsaws/gunshots for anti-poaching conservation.
  • Healthcare: (1) Cough/breath screening for respiratory diseases (TB, COPD, asthma); (2) digital stethoscope heart/lung auscultation and voice biomarkers for neurological conditions.

Clinical deployment challenges:

  1. Privacy & On-Device Inference: Avoiding transmission of identifiable patient speech/health audio to cloud servers.
  2. Domain Shift & Clinical Validation: Handling diverse smartphone microphones/room acoustics while maintaining high clinical sensitivity.
Answer:

An audio foundation model is a large self-supervised encoder transferable across tasks. It powers biodiversity monitoring and anti-poaching in bioacoustics, plus cough/auscultation screening and vocal biomarkers in healthcare, where patient privacy, device mismatch, and clinical safety are critical.

Question 3
  1. Explain the roles of ITD, ILD, and HRTF (pinna cues) in human 3D spatial hearing.
  2. Using Woodworth's formula with and , compute at (), and find how many channels Second-Order Ambisonics () uses.

Spatial hearing cues:

  • ITD: Arrival time difference between ears (dominant ).
  • ILD: Interaural loudness difference caused by head shadowing (dominant ).
  • HRTF / Pinna cues: Angle-dependent outer-ear spectral notches resolving elevation and front–back direction.

Numerical ITD at and Second-Order Ambisonics channels:

For Second-Order Ambisonics (): channels.

Answer:
  1. ITD ( timing), ILD ( level shadow), and HRTF pinna notches (elevation/front-back) establish 3D localization.
  2. ; Second-Order Ambisonics requires channels.
Question 4
  1. State four architectural hardware features that make an Audio DSP IC far more power-efficient for real-time audio filtering than a general-purpose CPU.
  2. A single-MAC Audio DSP processes -channel audio at with a -tap FIR filter per channel ( MACs/sample/channel) plus biquads total ( MACs/sample each). Calculate the required MMACs/s and DSP utilization percentage.

Four Audio DSP IC architectural features:

  1. Single-cycle hardware MAC & SIMD pipelines.
  2. Modified Harvard architecture (simultaneous instruction + dual data bus fetches).
  3. Hardware circular & bit-reversed addressing (zero-overhead delay lines and FFTs).
  4. Wide accumulator guard bits + saturation logic plus integrated audio buses.

DSP throughput and utilization: MACs per sample frame .

Answer:
  1. Single-cycle MAC/SIMD, Modified Harvard dual data buses, hardware circular/bit-reversed addressing, and saturation guard bits + interfaces.
  2. Required throughput ( utilization).
Question 5

Compare autoregressive audio-token Transformers (e.g., MusicGen) and latent diffusion models (e.g., AudioLDM / Stable Audio) for generative AI in music, and list two key ethical or technical challenges. If an RVQ audio codec emits codebooks of bits each at , compute its bitrate in .

Autoregressive Token Transformers vs. Latent Diffusion:

  • Autoregressive Token Models: Quantize audio via an RVQ neural codec (EnCodec/SoundStream) into discrete tokens and predict tokens sequentially with a Transformer decoder; naturally suit causal continuation and melody conditioning.
  • Latent Diffusion Models: Iteratively denoise compressed VAE latents conditioned via text cross-attention; excel at holistic global timing/structure and fast parallel batch generation.

Challenges and RVQ bitrate: Key challenges include copyright/attribution of training recordings and long-form multi-minute musical coherence (plus watermarking/consent). For , codebooks, :

Answer:

Autoregressive models predict discrete RVQ codec tokens sequentially, while latent diffusion models iteratively denoise continuous VAE audio latents; major challenges are copyright/attribution and long-horizon musical structure. The -codebook RVQ stream runs at .

← Audio system and DAW Integration