PadhaikoAll subjects

Introduction

Multimedia System (ENCT 466)

Open in the Padhaiko app

1.1 Multimedia and its applications

Definition

Multimedia

Multimedia is the computer-controlled integration of multiple forms of media—specifically at least one discrete (time-independent) medium (such as text or still graphics) and at least one continuous (time-dependent) medium (such as audio, video, or animation)—that can be represented, stored, transmitted, and processed digitally.

Unlike traditional single-medium systems (such as a plain text editor or an analog television broadcast), digital multimedia combines diverse information channels into an interactive, synchronized computing environment.

Major Application Areas of Multimedia

Multimedia systems are deployed across nearly every domain of computing and communication:

Key Applications of Multimedia Systems
DomainRepresentative ApplicationsPrimary Media Used
Education & TrainingE-learning portals, virtual laboratories, flight/medical simulators, interactive CBT (Computer-Based Training)Text, graphics, narration audio, instructional video, 3D animation
Entertainment & GamingVideo-on-Demand (VoD), streaming platforms, 3D video games, Virtual/Augmented Reality (VR/AR)High-definition video, surround audio, interactive 3D graphics
Business & CommunicationVideo conferencing, corporate presentations, digital signage, collaborative document sharingLive audio/video streams, slides, shared whiteboards
Healthcare (Telemedicine)Remote diagnosis, surgical simulation, PACS medical imaging (MRI/CT visualization)High-resolution medical images, live consultation video, patient records
Journalism & PublishingElectronic books, interactive web magazines, digital newsroomsFormatted text, photographs, embedded clips, infographics
Engineering & IndustryCAD/CAM visualization, architectural walkthroughs, digital twinsVector graphics, 3D models, simulation animations

1.2 Global structure of Multimedia

The global structure of multimedia (following Steinmetz & Nahrstedt) organizes multimedia technology as a layered architecture connecting human-facing applications down to physical hardware and transmission networks, complemented by the distinction between hypertext and hypermedia information structures.

Layered Global Architecture

  1. Application Domain Layer (Top): End-user applications such as video conferencing, Video-on-Demand, e-learning, and digital authoring tools.
  2. System / Middleware Layer: Synchronization engines, multimedia databases, QoS (Quality of Service) managers, and programming abstractions (toolboxes, APIs, OS extensions).
  3. Compression & Media Processing Layer: Audio, image, and video codecs (JPEG, MPEG, MP3, H.264), signal processing, and format conversion.
  4. Device & Network Infrastructure Layer (Bottom): Capture/display hardware (microphones, cameras, GPUs, sound cards), high-speed storage (SSD, optical media), and broadband communication networks (ATM, IP/RTP, Fiber).

Hypertext vs. Hypermedia

AspectHypertextHypermedia
Core ContentNon-linearly linked text nodes connected by hyperlinksNon-linearly linked multimedia nodes (text, images, audio, video, animation)
NavigationClickable text anchors jumping across documentsClickable hotspots on images, timeline markers in audio/video, and text anchors
Representative ExampleStatic text-only wiki articles or HTML text linksInteractive web pages, multimedia encyclopedias, streaming portals with linked clips

1.3 Medium

Definition

Medium (Plural: Media)

In communication and computing, a medium is a means for the distribution, representation, storage, or presentation of information. Based on how information is handled at each stage from human perception to physical transport, Steinmetz and Nahrstedt classify media into six distinct categories.

Six Classifications of Media (Steinmetz & Nahrstedt)
Medium CategoryMeaning / RoleConcrete Examples
1. Perception MediumHow humans perceive information through sensory organsSeeing (visual: text, images, video); Hearing (auditory: speech, music)
2. Representation MediumHow information is internally encoded inside a computerASCII/UTF-8 (text), PCM (audio), JPEG/PNG (image), MPEG/H.264 (video)
3. Presentation MediumPhysical tools/devices used to input or output information to/from humansInput: keyboard, microphone, camera; Output: monitor, speaker, printer
4. Storage MediumPhysical data carriers that retain encoded media persistentlyHard disk drives (HDD), SSD, optical discs (DVD/Blu-ray), magnetic tape
5. Transmission MediumPhysical channels carrying signals over distanceCoaxial cable, twisted pair, optical fiber, radio/wireless links
6. Information Exchange MediumCombined carriers used to interchange multimedia across systemsStorage media (USB/DVD) and transmission networks (Internet, LAN)

1.4 Multimedia system and properties

Definition

Multimedia System

A multimedia system is any computing system capable of processing, storing, transmitting, and presenting multimedia data and applications in an integrated, synchronized manner.

Four Fundamental Properties of a Multimedia System

According to the standard Steinmetz & Nahrstedt definition, a true multimedia system must exhibit four properties:

  1. Combination of Media: It must combine both discrete (time-independent: text, graphics) and continuous (time-dependent: audio, video) media. A system handling only text and still images is not a true multimedia system.
  2. Independence: Different media streams must be independent of each other at the data level (e.g., an audio track and a video track can be edited, compressed, or stored separately) while still capable of being synchronized during playback.
  3. Computer-Supported Integration: A computer must control, integrate, and manipulate the media dynamically—enabling user interactivity rather than passive linear playback.
  4. Communication System Capability: Multimedia data should be transmittable across communication networks (LAN, WAN, Internet) for distributed applications such as conferencing and streaming.

Discrete vs. Continuous Media

AspectDiscrete (Time-Independent) MediaContinuous (Time-Dependent) Media
Time DependencyMeaning does not depend on time or playback rateMeaning strictly depends on time and continuous playback speed
ExamplesText, still images, vector graphicsDigital audio, speech, video, real-time animation
Timing ConstraintNo real-time deadline for correctnessStrict real-time deadlines; jitter or delay distorts output
Data VolumeRelatively small (KB to a few MB)Very large (MB to GB), requiring high bandwidth and compression

1.5 Characteristics of a Multimedia system

A multimedia system possesses four core operational characteristics that distinguish it from conventional data-processing systems:

  1. Computer Controlled: All media capture, editing, storage, synchronization, and presentation operations are managed programmatically by a computer rather than standalone analog hardware.
  2. Integrated: Multiple media types are seamlessly combined inside a unified user interface and data model rather than requiring separate, disconnected devices.
  3. Digital Representation: Information is stored and processed in binary digital form ( and ), allowing lossless copying, algorithmic compression, error correction, and software manipulation.
  4. Interactive Interface: The user can actively control what, when, and how media is presented (e.g., pausing, seeking, branching in hypermedia, adjusting playback parameters), as opposed to passive broadcast media.

1.6 Challenges for Multimedia Systems

Developing and operating multimedia systems introduces strict technical challenges beyond conventional computing:

  1. Huge Data Volume & Storage Demand: Uncompressed audio and high-resolution video consume gigabytes per minute, demanding aggressive lossless/lossy compression and high-capacity storage.
  2. Real-Time Processing & Scheduling: Continuous media requires strict deadlines—for example, video must deliver one frame every . Conventional general-purpose OS schedulers are designed for fairness, not real-time multimedia deadlines.
  3. High Network Bandwidth & QoS Guarantees: Networks must guarantee minimum bandwidth, bounded end-to-end delay (latency), low jitter (delay variation), and acceptable packet loss rates.
  4. Media Synchronization (Lip-Sync & Skew): Related streams (such as a speaker's audio and video) must stay aligned within tight human perception limits (typically for lip synchronization).
  5. Hardware & Platform Heterogeneity: Different devices have different screen resolutions, color depths, CPU/GPU capabilities, and supported codecs, requiring scalable formats and transcoding.
  6. Data Modeling & Content-Based Retrieval: Searching inside unstructured audio/video streams (e.g., finding a specific scene or spoken phrase) is far harder than indexing structured relational text tables.

1.7 Components of Multimedia System

A complete multimedia system consists of hardware, software, and network/communication components working together across the media lifecycle (capture storage processing transmission presentation).

Hardware and Software Components of a Multimedia System
Component GroupSpecific Components & Functions
1. Capture DevicesMicrophones, digital video cameras, scanners, MIDI keyboards, frame grabbers, digitizers (ADC)
2. Storage DevicesHigh-speed SSDs, RAID disk arrays, optical discs (CD/DVD/Blu-ray), high-capacity RAM buffers
3. Processing HardwareHigh-speed CPUs, GPUs (for 2D/3D rendering & shaders), DSPs (Digital Signal Processors), hardware video codecs
4. Output / Display DevicesHigh-resolution LCD/OLED monitors, VR headsets, stereo/surround speakers, DACs, projectors
5. Communication NetworksHigh-speed Ethernet, Wi-Fi, optical fiber, streaming protocols (RTP/RTSP, HTTP/DASH)
6. System Software & MiddlewareReal-time OS extensions, device drivers, synchronization managers, media frameworks (DirectShow, GStreamer, FFmpeg)
7. Authoring & Application SoftwareAudio editors, image/vector suites, non-linear video editors (NLEs), 3D animation tools, web/multimedia players

1.8 Multimedia building blocks

Every multimedia application is constructed from five core building blocks (media elements), which fall into discrete and continuous categories:

Definition

Five Multimedia Building Blocks

  1. Text: The foundational discrete medium for titles, body content, labels, captions, and hyperlinks; encoded using ASCII, Unicode (UTF-8/UTF-16), or formatted markup (HTML, RTF, PDF).
  2. Image / Graphics: Still visual representations divided into raster (bitmap) images (pixel grids like JPEG, PNG, BMP) and vector graphics (mathematical primitives like SVG, EPS).
  3. Audio (Sound): Continuous acoustic signals—speech, music, and sound effects—digitized via sampling and quantization (WAV, MP3, AAC) or synthesized via MIDI event streams.
  4. Video: A continuous sequence of photographic frames captured at a fixed frame rate (e.g., , , or ) creating the illusion of smooth motion (MP4/H.264, WebM/VP9).
  5. Animation: Computer-generated or artist-drawn synthetic visual motion created frame-by-frame or via keyframe interpolation (tweening), in 2D or 3D space.

1.9 Scope of Multimedia

The scope of multimedia defines the breadth of technological, economic, and societal domains shaped by integrated digital media:

  1. Education & Open Knowledge: Democratizes learning through MOOCs, interactive e-books, virtual labs, and adaptive tutoring systems accessible globally.
  2. Telecommunications & Collaboration: Transforms remote work, telemedicine, and distance governance through low-latency HD video conferencing and shared digital workspaces.
  3. Digital Entertainment & Creative Economy: Drives streaming media, interactive gaming, CGI cinema, virtual production, and user-generated content ecosystems.
  4. Commerce, Marketing & Journalism: Powers e-commerce product visualization (360° views, AR try-on), digital advertising, and multimedia data journalism.
  5. Scientific, Medical & Industrial Visualization: Enables real-time telemetry dashboards, medical imaging diagnostics, flight/surgical simulation, and digital twin monitoring.
  6. AI-Enhanced Multimedia (Emerging Scope): Integrates generative AI, automated speech-to-text captioning, real-time translation, and intelligent media indexing.

Worked Examples

Example 1

1 (6 marks)

A multimedia presentation combines uncompressed CD-quality stereo audio ( sampling rate, , ) and uncompressed RGB video ( pixels, , ).

  1. Calculate the bit rate of the audio stream (in ) and the video stream (in , using ).
  2. Find the combined multimedia stream bandwidth in and the total storage required for a uncompressed clip in ().

Audio stream bit rate: The bit rate of uncompressed PCM audio is the product of sampling rate, bit depth, and number of channels:

Video stream bit rate:

Combined bandwidth and storage requirement:

This illustrates why huge data volume is a primary challenge for multimedia systems and why compression is mandatory.

Answer:
  1. Audio bit rate (); Video bit rate .
  2. Combined stream bandwidth ; uncompressed storage .
Example 2

2 (5 marks)

In an online university lecture portal, a student watches a recorded lecture video over fiber-optic broadband on a laptop screen while listening through headphones; the lecture is stored on a cloud SSD server as an MP4 (H.264/AAC) file alongside UTF-8 lecture notes. Classify every element in this scenario into the six media categories defined by Steinmetz and Nahrstedt.

Map the human sensory and physical I/O stages:

  • Perception Medium: Visual perception (reading slides/notes, watching video) and auditory perception (hearing speech/sound).
  • Presentation Medium: Laptop display screen and headphones (output); microphone and camera used earlier by the lecturer (input).

Map the internal encoding, persistence, and network transport stages:

  • Representation Medium: UTF-8 (for lecture text), H.264 (for video frames), AAC (for audio).
  • Storage Medium: Cloud SSD array holding the lecture files.
  • Transmission Medium: Fiber-optic broadband cable (and local Wi-Fi/Ethernet link) carrying data packets.
  • Information Exchange Medium: The Internet / IP communication network transferring the lecture between the server and the student's laptop.
Answer:
  • Perception: Visual and auditory senses.
  • Representation: UTF-8, H.264, AAC.
  • Presentation: Laptop monitor and headphones (output); camera and microphone (capture).
  • Storage: Cloud SSD array.
  • Transmission: Optical fiber cable / Wi-Fi link.
  • Information Exchange: The Internet communication network.
Example 3

3 (4 marks)

A video conferencing system streams video at and audio continuously.

  1. Compute the inter-frame display interval (in ).
  2. If human lip-synchronization tolerance is , how many video frames of drift correspond to the maximum permissible lip-sync skew?
  3. Explain which two challenges for multimedia systems this calculation highlights.

Calculate the inter-frame period:

Convert the skew budget into frames:

So audio and video must stay aligned within !

Connect to multimedia system challenges: This highlights (1) Media Synchronization (maintaining tight intra- and inter-stream timing across independent audio/video tracks) and (2) Real-Time Processing & Network QoS (guaranteeing bounded latency and jitter so frames do not miss their presentation deadlines).

Answer:
  1. Inter-frame interval .
  2. Maximum permissible lip-sync drift .
  3. Highlights the challenges of Media Synchronization and Real-Time Processing / Network QoS.

Exercises

Practice Exercises — Introduction to Multimedia Systems

Question 1

Define a multimedia system and explain its four fundamental properties. Distinguish clearly between discrete and continuous media with examples.

Definition: A multimedia system is a computer-controlled system capable of processing, storing, transmitting, and presenting integrated multimedia content combining discrete and continuous media.

Four Fundamental Properties:

  1. Combination of Media: Must include at least one discrete medium (text/image) and one continuous medium (audio/video).
  2. Independence: Different media streams remain independent at the data/storage level while being coordinated during playback.
  3. Computer-Supported Integration: A computer integrates, processes, and allows interactive control over the media.
  4. Communication Capability: Media can be transmitted over communication networks for distributed use.

Discrete vs. Continuous Media:

  • Discrete (time-independent) media such as text and still graphics have meaning independent of time and do not require continuous real-time playback deadlines.
  • Continuous (time-dependent) media such as audio, video, and animation depend directly on time and playback rate; delays or jitter immediately degrade user perception.
Answer:

A multimedia system integrates discrete and continuous media under computer control. Its four properties are combination of media, independence, computer-supported integration, and communication capability. Discrete media (text, images) are time-independent, whereas continuous media (audio, video, animation) depend on real-time playback rate and timing.

Question 2

Explain the six categories of media according to Steinmetz and Nahrstedt, giving two concrete examples for each category.

Sensory, Encoding, and I/O Categories:

  1. Perception Medium: How humans sense information (e.g., sight/visual text and images; hearing/speech and music).
  2. Representation Medium: How data is encoded digitally in a computer (e.g., ASCII/UTF-8 for text; PCM/MP3 for audio; JPEG/MPEG for visuals).
  3. Presentation Medium: Physical devices used for input/output (e.g., monitor, speaker for output; microphone, camera for input).

Persistence, Transport, and Interchange Categories:

  1. Storage Medium: Physical carriers storing encoded bits (e.g., SSD, HDD, optical DVD/Blu-ray).
  2. Transmission Medium: Physical channels carrying signals (e.g., optical fiber, twisted-pair Ethernet, wireless radio).
  3. Information Exchange Medium: Combined storage/transmission carriers used to move multimedia between systems (e.g., USB flash drives, the Internet).
Answer:

The six media categories are: Perception (sight, hearing), Representation (UTF-8, PCM, JPEG, MPEG), Presentation (monitor, speaker, microphone, camera), Storage (SSD, HDD, DVD), Transmission (optical fiber, coaxial cable, Wi-Fi), and Information Exchange (portable storage media and communication networks).

Question 3

Compare hypertext and hypermedia, and describe the layered global structure of multimedia along with the five multimedia building blocks.

Hypertext vs. Hypermedia:

  • Hypertext links non-linear text nodes via textual hyperlinks (e.g., classic text documents with cross-references).
  • Hypermedia generalizes hypertext to link all media types—text, graphics, audio, video, and animation—through hotspots, anchors, and timeline triggers.

Layered Global Structure & Five Building Blocks: The global structure stacks four layers: (1) Application Domain (VoD, conferencing, e-learning), (2) System/Middleware (synchronization, QoS, APIs), (3) Compression & Media Processing (codecs), and (4) Device & Network Infrastructure (I/O hardware, storage, networks). The five multimedia building blocks are Text, Image/Graphics, Audio, Video, and Animation.

Answer:

Hypertext connects non-linear text nodes, whereas hypermedia connects text, images, audio, video, and animation. The global structure consists of Application, System/Middleware, Compression/Processing, and Device/Network layers built from the five building blocks: Text, Images/Graphics, Audio, Video, and Animation.

Question 4

An uncompressed HD multimedia stream consists of RGB video () at and stereo audio sampled at with .

  1. Calculate the total required bandwidth in ().
  2. Calculate the storage needed for of this uncompressed stream in ().

Video and audio bit rates:

Total bandwidth and storage:

Answer:
  1. Total uncompressed bandwidth .
  2. Storage for ().
Question 5

Discuss the major challenges for multimedia systems and list the key hardware and software components needed to address them. In addition, suppose a () multimedia packet travels over a link () with a propagation + queuing delay of and codec processing delay of . Calculate the total end-to-end packet delay (in ) and express it as a multiple of one video frame period.

Challenges and Components:

  • Key Challenges: Huge data volumes, real-time OS scheduling deadlines, high network bandwidth with QoS (delay, jitter, loss), inter/intra-stream synchronization, platform heterogeneity, and content-based retrieval.
  • Addressing Components: Capture devices (ADC, cameras, mics), fast storage (SSDs, RAID), processing hardware (GPUs, DSPs, hardware codecs), broadband networks with streaming protocols (RTP/RTSP), and real-time middleware/synchronization engines.

End-to-End Delay Calculation:

Answer:

Key challenges are data volume, real-time deadlines, network QoS, synchronization, heterogeneity, and retrieval, addressed by capture/output hardware, SSD/RAID storage, GPUs/DSPs, broadband networks, and real-time middleware. Total packet delay is ( at ).

Sound/Audio System →