PadhaikoAll subjects

Multimedia Application

Multimedia System (ENCT 466)

Open in the Padhaiko app

8.1 Media preparation and composition

Every functional multimedia application begins by acquiring raw media assets and assembling them coherently across space and time.

  1. Media Preparation (Acquisition & Conditioning):
    • Capture & Digitization: Recording sound via microphones/ADC, capturing photographs/video via cameras, or scanning physical documents.
    • Cleaning & Editing: Removing background noise, normalizing audio loudness, cropping/color-correcting images, and trimming video clips.
    • Encoding & Formatting: Compressing assets into standardized representations (PNG/WebP, AAC/MP3, H.264/MP4) matched to target storage and network budgets.
  2. Media Composition (Spatial & Temporal Assembly):
    • Spatial Composition: Arranging visual elements on the display canvas—defining coordinates , bounding dimensions, -order layering, alpha transparency, and responsive scaling.
    • Temporal Composition: Scheduling when each medium starts, pauses, loops, or ends along a master timeline (parallel <par> tracks, sequential <seq> transitions, and relative event triggers).

8.2 Media integration and communication

Once media components are prepared and composed, the application must integrate them smoothly at playback and communicate them reliably across networks.

Media Integration & Synchronization

Media Communication Protocols & QoS

Key Multimedia Communication Protocols
ProtocolLayer / RolePrimary Function
RTP / RTCPTransport (over UDP)Carries timestamped real-time audio/video packets (RTP) and monitors packet loss/jitter feedback (RTCP)
RTSPSession ControlProvides VCR-like commands (PLAY, PAUSE, TEARDOWN) for live/stored streaming servers
SIP / WebRTCSignalingEstablishes, modifies, and terminates interactive voice/video calls and NAT traversal
HLS / MPEG-DASHAdaptive HTTP StreamingDelivers media chunks over standard HTTP/CDN, dynamically switching bitrate to match network speed

Network communication must manage Quality of Service (QoS) across four parameters: throughput (bandwidth), end-to-end delay, jitter (delay variance absorbed by playout buffers), and packet loss rate.


8.3 Media Entertainment

Entertainment is one of the largest drivers of multimedia hardware and compression innovation:

  1. Streaming Media & Digital TV (IPTV / OTT): On-demand movies, live sports broadcasts, and multichannel surround sound delivered via adaptive HTTP streaming and hardware-accelerated decoders.
  2. Interactive Video Games: Real-time 3D GPU rendering at , spatialized 3D audio, physics simulation, and ultra-low-latency multiplayer state synchronization.
  3. Virtual Reality (VR) & Augmented Reality (AR): Head-mounted stereoscopic displays requiring motion-to-photon latency to prevent motion sickness, combined with spatial audio and real-world camera overlays.

8.4 Telemedicine

Definition

Telemedicine

Telemedicine is the delivery of remote clinical healthcare, diagnostics, and medical education using multimedia capture, storage, and telecommunication technologies.

Two Primary Modes of Telemedicine

  1. Store-and-Forward (Asynchronous) Telemedicine: Local clinics capture high-resolution medical images (DICOM X-rays, CT/MRI scans, dermatology photos, pathology slides) and patient records, then transmit them securely to remote specialists for offline review.
  2. Real-Time (Interactive) Telemedicine: Live two-way HD video consultations, remote stethoscope/ultrasound telemetry, ICU tele-monitoring, and robotic tele-surgery.

Strict Technical Requirements


8.5 E-learning

E-learning (Electronic Learning) uses multimedia systems and networks to deliver structured instruction, interactive simulations, and assessment beyond physical classroom walls.

AspectSynchronous E-LearningAsynchronous E-Learning
TimingLive, real-time scheduled classesSelf-paced, on-demand anytime access
Media & ToolsVideo conferencing, shared digital whiteboards, live screen sharing, instant pollsRecorded video lectures, interactive HTML5 animations, PDFs, LMS quizzes (SCORM/xAPI)
InteractionImmediate voice/chat Q&A with instructor and peersDiscussion forums, automated quiz feedback, peer review
Bandwidth SensitivityHigh: requires stable low-latency bidirectional broadbandFlexible: supports caching, offline downloads, and adaptive bitrates

8.6 Digital video editing and production systems

Modern video production relies on Non-Linear Editing (NLE) systems rather than legacy tape-to-tape Linear Editing.

AspectLinear (Tape-Based) EditingNon-Linear Editing (NLE)
Storage MediumSequential analog/digital magnetic tapeRandom-access SSD / RAID disk arrays
Access & WorkflowMust fast-forward/rewind sequentially; inserting a clip forces re-recording downstreamInstant random access to any frame; drag-and-drop non-destructive timeline edits
FlexibilityDestructive and rigidUnlimited undo, multi-track compositing, easy re-ordering

Standard NLE Production Pipeline

  1. Ingest & Proxy Generation: Importing camera footage and optionally generating lightweight low-resolution proxy clips for smooth scrubbing on modest workstations.
  2. Timeline Assembly & Trimming: Arranging multiple synchronized video and audio tracks, trimming in/out points, and creating J-cuts / L-cuts (overlapping audio across visual cuts).
  3. Transitions, Effects & Compositing: Adding cross-dissolves, wipes, chroma keying (green-screen background replacement), titles, and motion graphics.
  4. Color Grading & Audio Mixing: Balancing exposure/contrast across shots, applying LUTs, and mixing dialogue, music, and Foley sound effects to broadcast loudness standards.
  5. Rendering & Export: Rendering effects at full resolution and encoding into final delivery containers (MP4/H.264/HEVC).

8.7 Video conferencing

Video conferencing enables real-time interactive audio, video, and document/screen sharing among two or more geographically separated participants.

Multi-Party Conferencing Architectures

  1. Full Mesh (Peer-to-Peer): Every participant sends a separate stream directly to every other participant. Works for users without a central media server, but uplink bandwidth grows as per user.
  2. MCU (Multipoint Control Unit): A central server decodes, composites/mixes, and re-encodes all participants' streams into one combined mosaic video + mixed audio stream sent back to each user. Keeps client downlink low ( stream), at high server CPU/transcoding cost and added latency.
  3. SFU (Selective Forwarding Unit): A lightweight central router receives each participant's stream (often in simulcast low/med/high layers) and forwards selected streams directly to other participants without decoding/re-encoding. Minimizes server latency and CPU load—used by modern Zoom, Google Meet, and Jitsi.

Core Technical Stack

Signaling (SIP or WebRTC SDP/ICE) + Media (RTP/SRTP over UDP) + Acoustic Echo Cancellation (AEC) + Noise Suppression + Adaptive Jitter Buffer, targeting one-way latency .


8.8 Video-on-demand

Definition

Video-on-Demand (VoD)

Video-on-Demand (VoD) allows subscribers to browse a catalog of stored video titles, select any program at any time, and control playback interactively (play, pause, fast-forward, rewind, seek).

True VoD (TVoD) vs. Near VoD (NVoD)

Modern VoD System Architecture

Videos are pre-encoded at multiple quality tiers (e.g., ) and sliced into HTTP segments; the client measures available throughput after each chunk and dynamically selects the highest bitrate that avoids rebuffering.


Worked Examples

Example 1

1 (6 marks)

A -participant video conference uses per video/audio stream. Compare the per-participant uplink, per-participant downlink, and total server outgoing bandwidth across three architectures:

  1. Full Mesh (Peer-to-Peer)
  2. MCU (Multipoint Control Unit) sending mixed composite stream per user
  3. SFU (Selective Forwarding Unit) forwarding each participant's stream without mixing

Full Mesh (Peer-to-Peer, ): Each participant sends directly to the other peers and receives from peers (no central server):

MCU (Central Mixing Server): Each user uploads stream to the MCU and downloads pre-mixed composite stream from the MCU:

SFU (Selective Forwarding Unit): Each user uploads only stream to the SFU, and the SFU forwards the other streams to each participant:

Answer:
  • Full Mesh: Uplink , Downlink , Server .
  • MCU: Uplink , Downlink , Server Egress .
  • SFU: Uplink , Downlink , Server Egress .
Example 2

2 (5 marks)

A Video-on-Demand (VoD) platform delivers MPEG-DASH streams sliced into chunks at two adaptive tiers: High () and Medium ().

  1. Calculate the aggregate CDN egress bandwidth (in , ) required to serve concurrent viewers streaming the High tier.
  2. Compute the file size (in , ) of a single chunk at each quality tier.

Aggregate CDN bandwidth for concurrent viewers:

Size of each DASH chunk:

Answer:
  1. Aggregate CDN egress ().
  2. chunk sizes: High (); Medium ().
Example 3

3 (5 marks)

In a store-and-forward telemedicine clinic, a CT scan produces slices at pixels with grayscale precision.

  1. Compute the uncompressed size of the CT scan in ().
  2. Using lossless JPEG-LS compression achieving a ratio, compute the compressed transfer size and the transmission time over a () clinic link. Why is lossy compression avoided here?

Compute uncompressed and lossless-compressed sizes:

Compute transfer time and justify lossless coding:

Lossy compression is avoided for diagnostic radiology because quantization artifacts can mask subtle micro-fractures or small lesions, risking misdiagnosis and legal liability.

Answer:
  1. Uncompressed CT size .
  2. Lossless compressed size ; transfer time at . Lossless compression preserves exact diagnostic pixel values.

Exercises

Practice Exercises — Multimedia Applications

Question 1

Explain the difference between media preparation, media composition, and media integration & communication. Distinguish between intra-stream and inter-stream synchronization, and name the primary streaming protocols used in multimedia communication.

Preparation, Composition, Integration, and Communication:

  • Media Preparation: Capturing, cleaning, formatting, and compressing individual media assets.
  • Media Composition: Laying out assets spatially (coordinates, layers, alpha) and temporally (timelines, parallel/sequential schedules).
  • Media Integration & Communication: Synchronizing multiple streams during playback and delivering them across networks with QoS control.

Synchronization & Protocols:

  • Intra-stream sync: Steady frame pacing within one continuous stream (e.g., per video frame).
  • Inter-stream sync: Cross-stream alignment such as audio–video lip-sync ().
  • Protocols: RTP/RTCP (real-time media + feedback), RTSP (stream control), SIP/WebRTC (call signaling), and HLS/MPEG-DASH (adaptive HTTP chunk streaming).
Answer:

Preparation digitizes/compresses assets, composition arranges them spatially and temporally, and integration/communication maintains intra-stream (frame pacing) and inter-stream (lip-sync) alignment over networks using RTP/RTCP, RTSP, SIP/WebRTC, and HLS/MPEG-DASH.

Question 2

Compare Telemedicine and E-learning as networked multimedia applications. Distinguish store-and-forward vs. real-time telemedicine, and synchronous vs. asynchronous e-learning.

Telemedicine Modes:

  • Store-and-Forward: Asynchronous transfer of lossless medical images (DICOM X-ray/CT/MRI) and records for specialist diagnosis.
  • Real-Time Interactive: Live HD clinical video consultations, remote monitoring, and tele-surgery requiring low latency and encryption.

E-Learning Modes:

  • Synchronous E-Learning: Live scheduled virtual classes using video conferencing, shared whiteboards, and instant Q&A.
  • Asynchronous E-Learning: Self-paced access to recorded lectures, interactive animations, and LMS quizzes (SCORM/xAPI), resilient to intermittent bandwidth.
Answer:

Telemedicine delivers remote healthcare via store-and-forward lossless imaging or real-time video consultations, whereas E-learning delivers education via live synchronous virtual classrooms or self-paced asynchronous multimedia modules.

Question 3

Contrast Linear Editing with Non-Linear Editing (NLE) in digital video production, and outline the main stages of an NLE workflow. Also distinguish between True VoD (TVoD) and Near VoD (NVoD), and between MCU and SFU video conferencing servers.

Linear vs. Non-Linear Editing (NLE): Linear editing copies tape-to-tape sequentially (rigid, destructive), whereas NLE edits digital clips on random-access disks via a non-destructive timeline: Ingest/Proxy Multi-track Assembly & J/L-Cuts Transitions/Chroma Key Color Grading & Audio Mixing Render/Export.

TVoD vs. NVoD and MCU vs. SFU:

  • TVoD vs. NVoD: True VoD grants every viewer an instant private stream with full seek control; Near VoD broadcasts a movie at staggered intervals across shared channels.
  • MCU vs. SFU: An MCU mixes all streams centrally (low client downlink, high server CPU/latency), whereas an SFU forwards selected streams without transcoding (ultra-low latency, low server CPU).
Answer:

NLE enables random-access non-destructive timeline editing (ingest, assembly, effects, grading/mixing, export). TVoD provides dedicated on-demand streams while NVoD staggers shared broadcasts; MCU servers transcode/mix streams centrally while SFU servers route streams without mixing.

Question 4

An -person engineering video conference uses per video/audio stream. Calculate:

  1. Per-user uplink, per-user downlink, and total SFU server egress bandwidth (in ) when using an SFU.
  2. Per-user uplink, per-user downlink, and total MCU server egress bandwidth (in ) if an MCU sends one mixed stream to each participant.

SFU bandwidths (): Each user uploads stream () and downloads forwarded streams:

MCU bandwidths (): Each user uploads stream () and receives combined stream ():

Answer:
  1. SFU: Uplink , Downlink , Server Egress .
  2. MCU: Uplink , Downlink , Server Egress .
Question 5

A movie on a VoD platform is encoded for HLS streaming at () and segmented into chunks.

  1. How many chunks compose the entire movie?
  2. Compute the size of one chunk and the total size of the movie in ().

Count total segments and compute single-chunk size:

Compute total movie size:

Answer:
  1. Total chunks .
  2. Each chunk ; total movie size ().
← Multimedia DesignApplications of AI in Multimedia →