Multimedia Application
Open in the Padhaiko app8.1 Media preparation and composition
Every functional multimedia application begins by acquiring raw media assets and assembling them coherently across space and time.
- Media Preparation (Acquisition & Conditioning):
- Capture & Digitization: Recording sound via microphones/ADC, capturing photographs/video via cameras, or scanning physical documents.
- Cleaning & Editing: Removing background noise, normalizing audio loudness, cropping/color-correcting images, and trimming video clips.
- Encoding & Formatting: Compressing assets into standardized representations (PNG/WebP, AAC/MP3, H.264/MP4) matched to target storage and network budgets.
- Media Composition (Spatial & Temporal Assembly):
- Spatial Composition: Arranging visual elements on the display canvas—defining coordinates , bounding dimensions, -order layering, alpha transparency, and responsive scaling.
- Temporal Composition: Scheduling when each medium starts, pauses, loops, or ends along a master timeline (parallel
<par>tracks, sequential<seq>transitions, and relative event triggers).
8.2 Media integration and communication
Once media components are prepared and composed, the application must integrate them smoothly at playback and communicate them reliably across networks.
Media Integration & Synchronization
- Intra-Stream Synchronization: Maintains uniform timing within a single continuous stream (e.g., delivering video frames strictly every without jitter).
- Inter-Stream Synchronization: Maintains temporal alignment between related streams—most critically lip-synchronization between spoken audio and video ( skew) and pointer/subtitle sync between slides, captions, and narration.
- Multimedia Databases & Hypermedia Containers: Stores media BLOBs alongside indexed metadata, timestamps, and cross-document hyperlinks (HTML5, SMIL, MPEG-4 containers).
Media Communication Protocols & QoS
| Protocol | Layer / Role | Primary Function |
|---|---|---|
| RTP / RTCP | Transport (over UDP) | Carries timestamped real-time audio/video packets (RTP) and monitors packet loss/jitter feedback (RTCP) |
| RTSP | Session Control | Provides VCR-like commands (PLAY, PAUSE, TEARDOWN) for live/stored streaming servers |
| SIP / WebRTC | Signaling | Establishes, modifies, and terminates interactive voice/video calls and NAT traversal |
| HLS / MPEG-DASH | Adaptive HTTP Streaming | Delivers media chunks over standard HTTP/CDN, dynamically switching bitrate to match network speed |
Network communication must manage Quality of Service (QoS) across four parameters: throughput (bandwidth), end-to-end delay, jitter (delay variance absorbed by playout buffers), and packet loss rate.
8.3 Media Entertainment
Entertainment is one of the largest drivers of multimedia hardware and compression innovation:
- Streaming Media & Digital TV (IPTV / OTT): On-demand movies, live sports broadcasts, and multichannel surround sound delivered via adaptive HTTP streaming and hardware-accelerated decoders.
- Interactive Video Games: Real-time 3D GPU rendering at , spatialized 3D audio, physics simulation, and ultra-low-latency multiplayer state synchronization.
- Virtual Reality (VR) & Augmented Reality (AR): Head-mounted stereoscopic displays requiring motion-to-photon latency to prevent motion sickness, combined with spatial audio and real-world camera overlays.
8.4 Telemedicine
Telemedicine
Telemedicine is the delivery of remote clinical healthcare, diagnostics, and medical education using multimedia capture, storage, and telecommunication technologies.
Two Primary Modes of Telemedicine
- Store-and-Forward (Asynchronous) Telemedicine: Local clinics capture high-resolution medical images (DICOM X-rays, CT/MRI scans, dermatology photos, pathology slides) and patient records, then transmit them securely to remote specialists for offline review.
- Real-Time (Interactive) Telemedicine: Live two-way HD video consultations, remote stethoscope/ultrasound telemetry, ICU tele-monitoring, and robotic tele-surgery.
Strict Technical Requirements
- Diagnostic Image Integrity: Radiology scans ( grayscale) must use lossless compression (e.g., Lossless JPEG / JPEG-LS / JPEG 2000 lossless) so subtle tumors or fractures are never obscured by compression artifacts.
- Security & Privacy: End-to-end encryption (TLS/SRTP) and strict access control to protect confidential patient health records.
- Reliability & QoS: Low-latency, jitter-controlled links for live surgical guidance and rural clinical consultation.
8.5 E-learning
E-learning (Electronic Learning) uses multimedia systems and networks to deliver structured instruction, interactive simulations, and assessment beyond physical classroom walls.
| Aspect | Synchronous E-Learning | Asynchronous E-Learning |
|---|---|---|
| Timing | Live, real-time scheduled classes | Self-paced, on-demand anytime access |
| Media & Tools | Video conferencing, shared digital whiteboards, live screen sharing, instant polls | Recorded video lectures, interactive HTML5 animations, PDFs, LMS quizzes (SCORM/xAPI) |
| Interaction | Immediate voice/chat Q&A with instructor and peers | Discussion forums, automated quiz feedback, peer review |
| Bandwidth Sensitivity | High: requires stable low-latency bidirectional broadband | Flexible: supports caching, offline downloads, and adaptive bitrates |
- Pedagogical Advantage: Combining visual diagrams, animations, and concise audio narration engages both visual and auditory cognitive channels, improving comprehension of complex engineering processes.
8.6 Digital video editing and production systems
Modern video production relies on Non-Linear Editing (NLE) systems rather than legacy tape-to-tape Linear Editing.
| Aspect | Linear (Tape-Based) Editing | Non-Linear Editing (NLE) |
|---|---|---|
| Storage Medium | Sequential analog/digital magnetic tape | Random-access SSD / RAID disk arrays |
| Access & Workflow | Must fast-forward/rewind sequentially; inserting a clip forces re-recording downstream | Instant random access to any frame; drag-and-drop non-destructive timeline edits |
| Flexibility | Destructive and rigid | Unlimited undo, multi-track compositing, easy re-ordering |
Standard NLE Production Pipeline
- Ingest & Proxy Generation: Importing camera footage and optionally generating lightweight low-resolution proxy clips for smooth scrubbing on modest workstations.
- Timeline Assembly & Trimming: Arranging multiple synchronized video and audio tracks, trimming in/out points, and creating J-cuts / L-cuts (overlapping audio across visual cuts).
- Transitions, Effects & Compositing: Adding cross-dissolves, wipes, chroma keying (green-screen background replacement), titles, and motion graphics.
- Color Grading & Audio Mixing: Balancing exposure/contrast across shots, applying LUTs, and mixing dialogue, music, and Foley sound effects to broadcast loudness standards.
- Rendering & Export: Rendering effects at full resolution and encoding into final delivery containers (MP4/H.264/HEVC).
8.7 Video conferencing
Video conferencing enables real-time interactive audio, video, and document/screen sharing among two or more geographically separated participants.
Multi-Party Conferencing Architectures
- Full Mesh (Peer-to-Peer): Every participant sends a separate stream directly to every other participant. Works for users without a central media server, but uplink bandwidth grows as per user.
- MCU (Multipoint Control Unit): A central server decodes, composites/mixes, and re-encodes all participants' streams into one combined mosaic video + mixed audio stream sent back to each user. Keeps client downlink low ( stream), at high server CPU/transcoding cost and added latency.
- SFU (Selective Forwarding Unit): A lightweight central router receives each participant's stream (often in simulcast low/med/high layers) and forwards selected streams directly to other participants without decoding/re-encoding. Minimizes server latency and CPU load—used by modern Zoom, Google Meet, and Jitsi.
Core Technical Stack
Signaling (SIP or WebRTC SDP/ICE) + Media (RTP/SRTP over UDP) + Acoustic Echo Cancellation (AEC) + Noise Suppression + Adaptive Jitter Buffer, targeting one-way latency .
8.8 Video-on-demand
Video-on-Demand (VoD)
Video-on-Demand (VoD) allows subscribers to browse a catalog of stored video titles, select any program at any time, and control playback interactively (play, pause, fast-forward, rewind, seek).
True VoD (TVoD) vs. Near VoD (NVoD)
- True VoD (TVoD): Allocates an independent, dedicated stream to each viewer immediately upon request with instant VCR trick-play control (standard on modern Internet platforms via CDN + DASH/HLS).
- Near VoD (NVoD): Broadcasts popular titles repeatedly on multiple staggered channels (e.g., starting the same movie every across channels), serving thousands of viewers on shared broadcast bandwidth with at most a short wait time.
Modern VoD System Architecture
Videos are pre-encoded at multiple quality tiers (e.g., ) and sliced into HTTP segments; the client measures available throughput after each chunk and dynamically selects the highest bitrate that avoids rebuffering.
Worked Examples
1
A -participant video conference uses per video/audio stream. Compare the per-participant uplink, per-participant downlink, and total server outgoing bandwidth across three architectures:
- Full Mesh (Peer-to-Peer)
- MCU (Multipoint Control Unit) sending mixed composite stream per user
- SFU (Selective Forwarding Unit) forwarding each participant's stream without mixing
Full Mesh (Peer-to-Peer, ): Each participant sends directly to the other peers and receives from peers (no central server):
MCU (Central Mixing Server): Each user uploads stream to the MCU and downloads pre-mixed composite stream from the MCU:
SFU (Selective Forwarding Unit): Each user uploads only stream to the SFU, and the SFU forwards the other streams to each participant:
- Full Mesh: Uplink , Downlink , Server .
- MCU: Uplink , Downlink , Server Egress .
- SFU: Uplink , Downlink , Server Egress .
2
A Video-on-Demand (VoD) platform delivers MPEG-DASH streams sliced into chunks at two adaptive tiers: High () and Medium ().
- Calculate the aggregate CDN egress bandwidth (in , ) required to serve concurrent viewers streaming the High tier.
- Compute the file size (in , ) of a single chunk at each quality tier.
Aggregate CDN bandwidth for concurrent viewers:
Size of each DASH chunk:
- Aggregate CDN egress ().
- chunk sizes: High (); Medium ().
3
In a store-and-forward telemedicine clinic, a CT scan produces slices at pixels with grayscale precision.
- Compute the uncompressed size of the CT scan in ().
- Using lossless JPEG-LS compression achieving a ratio, compute the compressed transfer size and the transmission time over a () clinic link. Why is lossy compression avoided here?
Compute uncompressed and lossless-compressed sizes:
Compute transfer time and justify lossless coding:
Lossy compression is avoided for diagnostic radiology because quantization artifacts can mask subtle micro-fractures or small lesions, risking misdiagnosis and legal liability.
- Uncompressed CT size .
- Lossless compressed size ; transfer time at . Lossless compression preserves exact diagnostic pixel values.
Exercises
Practice Exercises — Multimedia Applications
Explain the difference between media preparation, media composition, and media integration & communication. Distinguish between intra-stream and inter-stream synchronization, and name the primary streaming protocols used in multimedia communication.
Preparation, Composition, Integration, and Communication:
- Media Preparation: Capturing, cleaning, formatting, and compressing individual media assets.
- Media Composition: Laying out assets spatially (coordinates, layers, alpha) and temporally (timelines, parallel/sequential schedules).
- Media Integration & Communication: Synchronizing multiple streams during playback and delivering them across networks with QoS control.
Synchronization & Protocols:
- Intra-stream sync: Steady frame pacing within one continuous stream (e.g., per video frame).
- Inter-stream sync: Cross-stream alignment such as audio–video lip-sync ().
- Protocols: RTP/RTCP (real-time media + feedback), RTSP (stream control), SIP/WebRTC (call signaling), and HLS/MPEG-DASH (adaptive HTTP chunk streaming).
Preparation digitizes/compresses assets, composition arranges them spatially and temporally, and integration/communication maintains intra-stream (frame pacing) and inter-stream (lip-sync) alignment over networks using RTP/RTCP, RTSP, SIP/WebRTC, and HLS/MPEG-DASH.
Compare Telemedicine and E-learning as networked multimedia applications. Distinguish store-and-forward vs. real-time telemedicine, and synchronous vs. asynchronous e-learning.
Telemedicine Modes:
- Store-and-Forward: Asynchronous transfer of lossless medical images (DICOM X-ray/CT/MRI) and records for specialist diagnosis.
- Real-Time Interactive: Live HD clinical video consultations, remote monitoring, and tele-surgery requiring low latency and encryption.
E-Learning Modes:
- Synchronous E-Learning: Live scheduled virtual classes using video conferencing, shared whiteboards, and instant Q&A.
- Asynchronous E-Learning: Self-paced access to recorded lectures, interactive animations, and LMS quizzes (SCORM/xAPI), resilient to intermittent bandwidth.
Telemedicine delivers remote healthcare via store-and-forward lossless imaging or real-time video consultations, whereas E-learning delivers education via live synchronous virtual classrooms or self-paced asynchronous multimedia modules.
Contrast Linear Editing with Non-Linear Editing (NLE) in digital video production, and outline the main stages of an NLE workflow. Also distinguish between True VoD (TVoD) and Near VoD (NVoD), and between MCU and SFU video conferencing servers.
Linear vs. Non-Linear Editing (NLE): Linear editing copies tape-to-tape sequentially (rigid, destructive), whereas NLE edits digital clips on random-access disks via a non-destructive timeline: Ingest/Proxy Multi-track Assembly & J/L-Cuts Transitions/Chroma Key Color Grading & Audio Mixing Render/Export.
TVoD vs. NVoD and MCU vs. SFU:
- TVoD vs. NVoD: True VoD grants every viewer an instant private stream with full seek control; Near VoD broadcasts a movie at staggered intervals across shared channels.
- MCU vs. SFU: An MCU mixes all streams centrally (low client downlink, high server CPU/latency), whereas an SFU forwards selected streams without transcoding (ultra-low latency, low server CPU).
NLE enables random-access non-destructive timeline editing (ingest, assembly, effects, grading/mixing, export). TVoD provides dedicated on-demand streams while NVoD staggers shared broadcasts; MCU servers transcode/mix streams centrally while SFU servers route streams without mixing.
An -person engineering video conference uses per video/audio stream. Calculate:
- Per-user uplink, per-user downlink, and total SFU server egress bandwidth (in ) when using an SFU.
- Per-user uplink, per-user downlink, and total MCU server egress bandwidth (in ) if an MCU sends one mixed stream to each participant.
SFU bandwidths (): Each user uploads stream () and downloads forwarded streams:
MCU bandwidths (): Each user uploads stream () and receives combined stream ():
- SFU: Uplink , Downlink , Server Egress .
- MCU: Uplink , Downlink , Server Egress .
A movie on a VoD platform is encoded for HLS streaming at () and segmented into chunks.
- How many chunks compose the entire movie?
- Compute the size of one chunk and the total size of the movie in ().
Count total segments and compute single-chunk size:
Compute total movie size:
- Total chunks .
- Each chunk ; total movie size ().