PadhaikoAll subjects

Interactive Visualization, Distributed Rendering, AR/VR/MR & Games

Computer Graphics and Visualization (ENCT 201)

Open in the Padhaiko app

8.1 Interactive Visualization

Computer visualization is the transformation of abstract, numerical, or physical data into visual representations to facilitate human cognition, analysis, hypothesis testing, and pattern discovery. It leverages the high bandwidth of the human visual system to convey complex relationships that are difficult to discern from raw numerical tables.

Visualizations are broadly divided into two major disciplines based on the nature of the underlying data:

Definition

Scientific Visualization vs. Information Visualization

  1. Scientific Visualization (SciVis): Focuses on data that has an inherent, physically grounded spatial coordinate system (such as three-dimensional space and time).
    • The spatial arrangement in the visualization directly mirrors the physical geometry of the phenomenon being studied.
    • Examples:
      • Computational fluid dynamics (CFD) airflow simulation around aircraft wings.
      • Medical volumetric rendering of 3D CT (computed tomography) and MRI (magnetic resonance imaging) scans.
      • Meteorological atmospheric pressure, temperature, and hurricane wind velocity vector fields.
      • Molecular modeling of protein folding and atomic crystal lattices.
  2. Information Visualization (InfoVis): Focuses on non-spatial, abstract, or categorical data that has no natural physical geometric structure.
    • The designer must invent an artificial spatial mapping (spatialization) to position data points meaningfully.
    • Examples:
      • Social network connection graphs and telecommunication network topologies.
      • Hierarchical tree structures (e.g. file system treemaps).
      • Financial transaction flows and multi-dimensional statistical data.

Principles of Interactive Exploration

Effective visualization systems enable real-time human interaction following the classic information-seeking mantra: "Overview first, zoom and filter, then details-on-demand" (Ben Shneiderman).


8.2 Distributed Scene Rendering

Rendering highly detailed scenes (containing hundreds of millions of polygons, complex lighting simulations, or ultra-high resolutions for tiled display walls and CAVE immersive rooms) exceeds the memory and processing capacity of any single computer. Distributed scene rendering (parallel rendering) partitions the graphics rendering pipeline across a network cluster of multiple computing nodes or GPUs.

Definition

Decomposition Strategies in Parallel Rendering

Parallel rendering architectures are classified by Molnar et al. according to where in the graphics pipeline the sorting from object space (3D primitives) to image space (2D pixels) occurs:

  1. Sort-First (Screen-Space Partitioning):
    • The 2D screen viewport is subdivided into disjoint 2D rectangular tiles or scanline strips.
    • Each cluster node is assigned one or more screen tiles and renders only the primitives that project into its assigned region.
    • Primitives are sorted to screen tiles during the early geometry stage.
    • Advantage: Low communication overhead during rasterization; highly scalable for ultra-high pixel resolutions.
  2. Sort-Last (Object-Space Partitioning):
    • The entire 3D scene database (geometry) is partitioned evenly among all rendering nodes.
    • Every node independently transforms and rasterizes its local subset of geometry to a full-screen local buffer containing RGB color and depth () values.
    • A dedicated image compositing network blends and depth-tests all pixel streams to produce the final composited image.
    • Advantage: Excellent load balancing for massive geometric datasets; completely insensitive to camera viewpoint changes.
  3. Sort-Middle (Pipeline Stage Partitioning):
    • Geometry transformation is distributed across geometry processors, and rasterization is distributed across rasterizers.
    • A communication crossbar sorts screen-coordinate primitives between the transformation and rasterization stages.
Comparison of Distributed Rendering Architectures
ParameterSort-FirstSort-LastSort-Middle
Partitioning Domain2D screen tiles (image-space)3D geometry subsets (object-space)Geometry / rasterizer split
Sorting StageEarly (before geometry processing)Late (after rasterization)Middle (between geometry and rasterization)
Network Traffic2D primitives to screen tile nodesFull-screen RGB + depth pixel streamsScreen-space transformed primitives
Scalability StrengthHigh display pixel resolutionMassive polygon counts ( primitives)Balanced moderate clusters

8.3 Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR)

Spatial computing technologies blend the physical and digital domains. They are situated along Milgram's Reality-Virtuality Continuum:

Definition

Definitions of VR, AR, and MR

  1. Virtual Reality (VR): A fully immersive computer-generated three-dimensional simulation that visually and acoustically isolates the user from the physical environment. The user perceives complete presence inside a synthetic digital world.
  2. Augmented Reality (AR): A technology that superimposes computer-generated digital assets (2D text, 3D graphics, annotations) onto the live view of the physical world. The digital elements appear as overlays; they do not physically interact with or anchor behind physical objects.
  3. Mixed Reality (MR): An advanced spatial computing paradigm where the physical world and virtual elements are seamlessly integrated. Physical and virtual objects co-exist in the same space, anchor to real surfaces (spatial mapping), and interact in real time with mutual occlusion (a real table can hide a virtual object).
Comparison of Virtual Reality, Augmented Reality, and Mixed Reality
DimensionVirtual Reality (VR)Augmented Reality (AR)Mixed Reality (MR)
Visual Environment synthetic; physical world is completely occludedLive physical world with synthetic digital overlaysLive physical world integrated with interactive synthetic objects
Occlusion HandlingVirtual objects occlude virtual objectsVirtual graphics overlay on top of physical objectsMutual occlusion: real objects can hide virtual objects and vice versa
Spatial AwarenessTracks user within a play boundaryTracks surfaces or 2D image markersFull real-time 3D spatial mapping and environmental meshing (LiDAR / SLAM)
Display HardwareOpaque Head-Mounted Displays (HMDs, e.g. Meta Quest, HTC Vive)Optical see-through smart glasses or mobile screens (ARKit, ARCore)Spatial computing headsets with video pass-through or waveguide optics

Key Components of a Virtual Reality (VR) System

An immersive VR system engages multiple sensory pathways to create a convincing sense of presence:

Theorem

Five Essential Components of a VR System

  1. Visual Display Engine (Head-Mounted Display - HMD):
    • High-resolution stereoscopic displays (OLED/LCD) providing independent left-eye and right-eye views with binocular parallax for depth perception.
    • Wide Field of View () and high refresh rates ( to ) with low-persistence pixel switching to eliminate motion blur and simulator sickness.
  2. Tracking and Sensing Subsystem (6 Degrees of Freedom - 6-DoF):
    • Tracks 3 positional coordinates and 3 rotational orientations (pitch, roll, yaw).
    • Inside-Out Tracking: Headset-mounted optical cameras and Inertial Measurement Units (IMUs) compute spatial location using Simultaneous Localization and Mapping (SLAM).
    • Outside-In Tracking: Stationary external base stations track infrared markers on the headset.
  3. Interactive Input Devices:
    • Ergonomic hand controllers with capacitive touch sensors, analog thumbsticks, and precision haptic actuators providing tactile feedback.
    • Optical hand tracking and eye-tracking modules for gaze-directed interaction.
  4. Real-Time Graphics and Physics Engine:
    • High-throughput rendering pipeline maintaining ultra-low motion-to-photon latency () between head movement and display update.
    • Employs foveated rendering (rendering high resolution only where the user's fovea is gazing) and asynchronous reprojection (time-warping frames if the GPU drops a frame).
  5. Spatial Audio System:
    • 3D binaural acoustic rendering using Head-Related Transfer Functions (HRTF) to position sound sources realistically in 3D space relative to head orientation.

8.4 Game Development and Real-Time Graphics

Modern video games represent the pinnacle of real-time computer graphics, delivering photorealistic rendering, complex physics simulations, and responsive user interaction within strict hardware budgets.


Frame Budget in Real-Time Graphics

A real-time graphics application must render frames at a fixed cadence:

During each frame budget window, two hardware processors work concurrently:

  1. CPU Workload: Evaluates game logic, artificial intelligence, skeletal animation bone transforms, physics simulation, spatial culling, and builds draw-call command buffers.
  2. GPU Workload: Executes vertex shaders, tessellation, rasterization, fragment (pixel) shading, shadow mapping, and post-processing compositing.

If either processor takes longer than , a frame drop (stutter) occurs.


The Programmable GPU Pipeline

Modern graphics hardware executes rendering through a configurable pipeline:


Level of Detail (LOD) Management

In complex open-world scenes, objects near the camera require dense, high-polygon meshes, while distant objects cover only a few screen pixels. Rendering distant objects with full geometric complexity wastes valuable vertex processing bandwidth.

Definition

Level of Detail (LOD)

Level of Detail (LOD) generates multiple simplified versions of each 3D mesh at varying geometric resolutions:


8.5 Applications of AR, VR and Gaming

The integration of advanced graphics, spatial tracking, and simulation has expanded beyond entertainment into vital professional fields:

1. Medicine and Healthcare

2. Engineering, Architecture and Manufacturing

3. Education and Civil Training

4. Interactive Entertainment and Serious Games


Worked Examples

Example 8.1

8.1

A real-time game engine renders scenes at Quad HD resolution ( pixels):

  1. Calculate the available frame time budget in milliseconds for target refresh rates of and .
  2. If the scene exhibits an average pixel overdraw factor of (meaning each screen pixel is shaded on average times per frame due to overlapping geometry), calculate the required fragment shading throughput in megapixels per second () at .

Evaluating frame time budgets:

  • For :
  • For :

Calculating pixel count and fragment workload per frame:

Accounting for an overdraw factor of :

Evaluating required throughput at 60 fps:

Converting to megapixels per second ():

Answer:
  1. Frame budgets: at ; at .
  2. Required fragment shading throughput at : .
Example 8.2

8.2

A 3D simulation scene contains instances of an architectural building model. The model has three Level of Detail (LOD) representations:

From a given camera viewpoint:

  1. Calculate the total number of polygons rendered using the LOD system.
  2. Calculate the total number of polygons that would have been rendered without LOD (all instances rendered at LOD 0).
  3. Compute the percentage reduction in geometric rendering workload achieved by LOD.

Evaluating polygon count with LOD management:

  • LOD 0 instances:
  • LOD 1 instances:
  • LOD 2 instances:

Evaluating baseline polygon count without LOD:

Calculating percentage workload reduction:

Answer:
  1. Total polygons rendered with LOD: .
  2. Total polygons without LOD: .
  3. Workload reduction: fewer polygons rendered.
Example 8.3

8.3

A distributed Sort-Last parallel rendering cluster generates 4K Ultra HD frames ( pixels) at . Each rendering node outputs its local framebuffer comprising of RGB color and of depth () data per pixel across a high-speed network to an image compositor:

  1. Calculate the total number of bits transmitted per frame by each node.
  2. Determine the required network transmission bandwidth per node in gigabits per second ().

Evaluating bits per pixel and bits per frame:

Calculating network transmission bandwidth at 60 fps:

Converting to decimal gigabits per second ():

Answer:
  1. Bits transmitted per frame: .
  2. Required network bandwidth per node: .

Exercises

Latest Trends in Computer Graphics Exercises

Question 1

Differentiate between scientific visualization and information visualization. Provide two practical examples of each discipline.

Contrasting underlying data foundations:

  • Scientific Visualization (SciVis) handles data possessing an intrinsic physical spatial coordinate system (such as 3D physical coordinates and time). The visualization represents physical phenomena.
  • Information Visualization (InfoVis) deals with abstract, non-spatial, and categorical data with no natural physical geometry. The visualization system must establish an artificial visual encoding.

Detailing practical examples:

  • SciVis examples:
    1. 3D volumetric reconstruction of human organs from MRI/CT medical scans.
    2. Computational fluid dynamics (CFD) airflow velocity vector fields over automotive vehicles.
  • InfoVis examples:
    1. Interactive graph visualization of global financial transaction networks.
    2. Treemap visualizations of computer disk storage usage.
Answer:

Scientific visualization maps data with intrinsic physical spatial coordinates (e.g. medical MRI volumes, weather velocity fields). Information visualization creates spatial layouts for abstract non-spatial data (e.g. network graphs, financial flow diagrams).

Question 2

Explain the three primary architectures used in distributed scene rendering: Sort-First, Sort-Last, and Sort-Middle.

Describing Sort-First and Sort-Last:

  • Sort-First: Partitions the 2D screen into rectangular tiles. Primitives are sorted to screen tiles early in the pipeline, and assigned cluster nodes render only their allocated tiles. Ideal for ultra-high pixel resolutions.
  • Sort-Last: Partitions 3D geometry evenly across nodes. Each node independently renders its geometry to a full-screen local color and depth buffer. An image compositor blends depth streams to produce the final image. Ideal for massive geometry.

Describing Sort-Middle:

  • Sort-Middle: Geometry transformation is parallelized across primitive nodes, and rasterization is parallelized across display nodes. A communication crossbar sorts screen-transformed primitives between the two stages.
Answer:

Sort-First sorts primitives to 2D screen tiles before geometry transformation; Sort-Last distributes 3D geometry across nodes and sorts pixels via depth-compositing after rasterization; Sort-Middle sorts screen-space primitives between transformation and rasterization stages.

Question 3

Differentiate between Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) with respect to environment, user isolation, and occlusion behavior.

Contrasting environment and immersion:

  • VR completely replaces the physical world with a synthetic 3D environment, isolating the user visually and auditorily.
  • AR superimposes 2D/3D digital graphics onto the physical world as a HUD overlay without physical interaction or real-world surface anchoring.
  • MR integrates virtual and physical worlds where virtual objects anchor to real surfaces, respect physical lighting, and support mutual occlusion with physical structures.

Analyzing occlusion behavior:

  • In VR, all occlusion is synthetic between virtual objects.
  • In AR, virtual content appears directly on top of physical objects.
  • In MR, physical objects can occlude virtual objects (e.g. a real wall hides a virtual avatar walking behind it).
Answer:

VR fully replaces physical reality with a synthetic world; AR overlays digital data over the real world; MR blends physical and digital worlds with mutual occlusion and spatial surface anchoring.

Question 4

Explain the five key components that constitute an immersive Virtual Reality (VR) system.

Hardware display, tracking, and audio:

  1. Visual Display Engine (HMD): Stereoscopic high-refresh () displays providing binocular depth cues with wide field of view.
  2. Tracking Subsystem: 6-DoF position and orientation tracking using inside-out SLAM optical cameras or outside-in sensors.
  3. Spatial Audio: 3D binaural sound rendering using HRTF filters to simulate directional acoustic cues.

Interaction and rendering engines:

  1. Input Devices: Tracked handheld controllers with haptic tactile actuators and hand-tracking systems.
  2. Real-Time Graphics Engine: High-performance renderer maintaining motion-to-photon latency under via foveated rendering and asynchronous reprojection.
Answer:

The five components are: (1) Head-Mounted Display (stereoscopic visual engine), (2) 6-DoF tracking subsystem (position and orientation), (3) Interactive input controllers with haptics, (4) Real-time graphics engine with low-latency reprojection, and (5) 3D spatial binaural audio.

Question 5

A VR headset operates at , and an advanced model operates at :

  1. Compute the maximum allowable frame time budget in milliseconds for both refresh rates.
  2. Explain how Level of Detail (LOD) management prevents frame drops in real-time graphics engines.

Calculating frame time budgets:

  • At :
  • At :

Explaining LOD prevention of frame drops: When scenes contain dense geometry, distant objects generate millions of vertex transformation and rasterization calculations that do not contribute to visual quality. LOD switches distant models to simplified low-polygon meshes, reducing the vertex processing load on the GPU and keeping rendering time within the strict frame budget window ().

Answer:
  1. Frame budgets: at ; at .
  2. LOD dynamically substitutes distant complex meshes with low-polygon versions, drastically lowering GPU vertex processing workloads and ensuring frame rendering finishes within the frame budget without stutters.
Question 6

A distributed rendering farm has nodes generating Full HD () frames for a Sort-Last system at . Each pixel transmits of RGB color and of depth ( total):

  1. Compute the transmission data rate per node in megabytes per second ().
  2. Compute the total aggregate data rate arriving at the compositor from all nodes.

Evaluating data rate per node:

At :

Evaluating aggregate compositor bandwidth:

Answer:
  1. Data rate per node: .
  2. Total aggregate bandwidth at compositor: .

Past exam questions for Computer Graphics and Visualization

← Animation Functions, Raster Animation, Key-Frames & Motion Specification