PadhaikoAll subjects

System-Level I/O and Performance Optimization

Network and Systems Programming (ENCT 386)

Open in the Padhaiko app

6.1 Advanced file I/O (Read/write optimization)

Standard read() and write() calls become bottlenecks when servers handle high-throughput logs, database pages, or multi-part HTTP responses. Linux/POSIX provides specialized I/O calls and caching controls to reduce system call overhead, memory copies, and lock contention.

1. Kernel Page Cache, fsync(), and fdatasync()

By default, POSIX file I/O goes through the kernel Page Cache: write() copies bytes into RAM pages marked "dirty" and returns immediately while the kernel flushes them to disk asynchronously later. For crash-safe databases and transaction logs:

2. Vectored (Scatter-Gather) I/O: readv() and writev()

When a server builds an HTTP response (a 200-byte header buffer + a 4 KB JSON body buffer), calling write() twice incurs two syscalls and risks sending two tiny TCP packets; calling memcpy() into one combined buffer wastes CPU. Vectored I/O transfers an array of struct iovec buffers in one atomic system call:

Vectored Output with writev()C
#include <sys/uio.h>

struct iovec iov[2];
iov[0].iov_base = http_header;
iov[0].iov_len  = hdr_len;
iov[1].iov_base = json_body;
iov[1].iov_len  = body_len;

ssize_t total_sent = writev(sock_fd, iov, 2); /* single syscall, atomic write */

3. Positional I/O: pread() and pwrite()

Normal lseek() + read() is not thread-safe when multiple threads share the same file descriptor, because another thread can shift the shared file offset between lseek() and read(). pread(fd, buf, count, offset) and pwrite(fd, buf, count, offset) atomically read/write at an explicit byte offset in one syscall without modifying the file descriptor's offset.

4. Cache Hints (posix_fadvise()) and Direct I/O (O_DIRECT)


6.2 Non-blocking I/O and asynchronous I/O concepts

It is vital not to confuse Non-Blocking Multiplexed I/O with True Asynchronous I/O:

AspectNon-Blocking Multiplexed I/O (epoll)True Asynchronous I/O (POSIX AIO / io_uring)
Notification TimingReadiness-based: notifies before syscall ("socket is ready to read now")Completion-based: notifies after kernel finished copying data into your buffer
Syscall Costepoll_wait() + separate recv()/send() syscallsBatched submissions & completions via shared ring buffers (zero syscalls with SQPOLL)
Regular Disk File SupportNo (disk read() still stalls on cache miss)Yes (works uniformly for disk files, pipes, and sockets)

POSIX AIO vs. Linux io_uring

  1. POSIX AIO (<aio.h>: aio_read, aio_write, aio_error, aio_return): Portable specification using struct aiocb; on Linux glibc it is implemented using a user-space thread pool rather than true kernel async hooks.
  2. Linux io_uring: Modern Linux asynchronous interface built around two lockless ring buffers mapped directly between user space and kernel space—a Submission Queue (SQ) and a Completion Queue (CQ). Applications push dozens of read, write, accept, or send entries into the SQ and harvest finished results from the CQ with minimal or zero mode switches.

6.3 Memory-mapped files (Mmap)

mmap() maps a file (or anonymous RAM / POSIX shared memory) directly into the calling process's virtual address space so file bytes can be read and modified using ordinary C memory pointers instead of read()/write() syscalls.

mmap() Signature and Companion Calls (<sys/mman.h>)C
void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);
int   msync(void *addr, size_t length, int flags);   /* flush dirty pages to disk */
int   munmap(void *addr, size_t length);             /* unmap region */

Key Parameters and Rules of mmap()

  1. Protection (prot): PROT_READ, PROT_WRITE, PROT_EXEC, or PROT_NONE.
  2. Visibility (flags):
    • MAP_SHARED: Writes modify the underlying kernel page cache directly, becoming visible to other processes mapping the file and flushed back to disk.
    • MAP_PRIVATE: Uses Copy-on-Write (CoW)—modifications stay private to the process and never write back to the file (used by the dynamic linker for loading .so shared libraries).
  3. Mandatory Page Alignment: offset MUST be an exact multiple of the system page size (sysconf(_SC_PAGESIZE), typically 4096 bytes on x86-64); otherwise mmap() fails with EINVAL.
  4. Demand Paging: mmap() sets up virtual page-table entries lazily without reading the whole file upfront; physical 4 KB pages are loaded on demand when first touched (minor/major page fault).

6.4 File locking and concurrency

When multiple processes concurrently read or update a shared configuration file, log, or database file, race conditions corrupt file contents unless access is coordinated with file locks.

Advisory vs. Mandatory Locking

flock() vs. fcntl() Byte-Range Locking

Aspectflock() (BSD Whole-File Lock)fcntl() (POSIX Byte-Range Lock)
GranularityEntire file onlyArbitrary byte range (l_start, l_len)
Lock TypesLOCK_SH (shared), LOCK_EX (exclusive), LOCK_UNF_RDLCK (shared), F_WRLCK (exclusive), F_UNLCK
Non-Blocking FlagLOCK_NB flag (fails with EWOULDBLOCK)F_SETLK (fails with EAGAIN/EACCES) vs F_SETLKW (waits)
Lock OwnershipAssociated with Open File Table entry (preserved across dup/fork)Associated with (PID, inode) pair!
Classic PitfallDoes not work reliably on legacy NFS mountsClosing ANY fd to that file in the process silently releases all fcntl locks held by that process on the file!

6.5 Performance tuning and profiling basics

Optimizing system and network software relies on measurement before tuning—identifying whether an application is CPU-bound, syscall-bound, I/O-bound, or lock-contended.

Essential Linux Profiling and Performance Analysis Tools
ToolWhat It MeasuresTypical Diagnosis
time ./progWall (real), user-space (user), and kernel (sys) CPU timeHigh sys time relative to user indicates excessive syscalls or page faults
strace -c ./progExact count, total time, and error count per system callReveals 1-byte read/write loops, unnecessary stat()/lseek(), or high futex lock waits
perf top / perf recordHardware CPU sampling of functions (user + kernel) & Flame GraphsPinpoints hot loops, serialization overhead, or kernel TCP copy bottlenecks
vmstat 1 / iostat -xz 1Context switches (cs), CPU %wa (I/O wait), disk queue depth, throughputDistinguishes disk bandwidth saturation from CPU starvation
valgrind (cachegrind/massif)L1/LLC cache misses, branch mispredictions, peak heap usageDetects pointer-chasing cache misses and memory leaks

Core Tuning Principles

  1. Batch and Coalesce System Calls: Replace loops of tiny read()/write() calls with 4–64 KB buffers, writev(), sendfile(), or io_uring batching to minimize user-kernel mode switches.
  2. Respect Page and Cache-Line Boundaries: Align O_DIRECT buffers to 4096 bytes; pad per-thread counters to 64-byte CPU cache lines (alignas(64)) to prevent false sharing across cores.
  3. Eliminate Lock Contention: Shrink critical sections, use per-core sharded data structures or lock-free queues instead of one global mutex, and keep slow disk/network I/O outside locks.

Worked Examples

Example

Calculating Page-Aligned Offsets and Lengths for mmap()

On a Linux system with page size PAGESIZE = 4096 bytes, an application wants to mmap() and read a 512-byte binary header starting at file byte offset 10,000.

  1. Why will passing offset = 10000 directly to mmap() fail?
  2. Compute the required aligned_offset, the delta inside the mapped window, and the minimum map_len to pass to mmap(), and write the C pointer expression to access byte 10,000.

Why Direct offset = 10000 Fails: mmap() requires offset to be an exact integer multiple of PAGESIZE (4096). Because 10000 % 4096 = 1808 != 0, calling mmap(..., offset = 10000) fails with -1 (MAP_FAILED) and errno = EINVAL.

Computing Aligned Offset, Delta, and Map Length:

  • aligned_offset = (10000 // 4096) * 4096 = 2 * 4096 = 8192.
  • delta = 10000 - 8192 = 1808 bytes.
  • map_len = delta + 512 = 1808 + 512 = 2320 bytes (the kernel rounds this up internally to 1 page of 4096 bytes, since 2320 <= 4096).
  • In C: char *base = mmap(NULL, 2320, PROT_READ, MAP_PRIVATE, fd, 8192); char *hdr = base + 1808;
Answer:
  1. 10,000 is not a multiple of 4096, causing mmap() to fail with EINVAL.
  2. aligned_offset = 8192, delta = 1808, map_len = 2320 bytes; access via base + 1808 (and later unmap with munmap(base, 2320)).
Example

Atomic Append with Advisory fcntl() Byte-Range Lock and writev()

Write a C function safe_log_record(int fd, const char *tag, const char *msg) that acquires an exclusive blocking POSIX write lock (F_WRLCK via F_SETLKW) across the file, writes tag and msg atomically in one syscall using writev(), flushes data with fdatasync(), and unlocks.

Configure struct flock and Vectored iovec: Set l_type = F_WRLCK, l_whence = SEEK_SET, l_start = 0, l_len = 0 (l_len = 0 means lock from l_start to EOF, including future appends!).

C Implementation (safe_log_record):

POSIX Byte-Range Lock + writev() + fdatasync()C
#include <fcntl.h>
#include <string.h>
#include <sys/uio.h>
#include <unistd.h>

int safe_log_record(int fd, const char *tag, const char *msg) {
    struct flock fl = {
        .l_type = F_WRLCK,
        .l_whence = SEEK_SET,
        .l_start = 0,
        .l_len = 0 /* 0 locks through EOF */
    };

    if (fcntl(fd, F_SETLKW, &fl) < 0) return -1;

    lseek(fd, 0, SEEK_END);
    struct iovec iov[3] = {
        { .iov_base = (void *)tag, .iov_len = strlen(tag) },
        { .iov_base = (void *)msg, .iov_len = strlen(msg) },
        { .iov_base = (void *)"\n", .iov_len = 1 }
    };
    ssize_t rc = writev(fd, iov, 3);
    if (rc >= 0) fdatasync(fd);

    fl.l_type = F_UNLCK;
    fcntl(fd, F_SETLK, &fl);
    return (rc < 0) ? -1 : 0;
}
Answer:

fcntl(fd, F_SETLKW, &fl) with F_WRLCK blocks until exclusive access is granted; writev() writes tag, msg, and "\n" in a single syscall; fdatasync() persists the data efficiently without unnecessary metadata flushes; and F_UNLCK releases the lock.

Example

Diagnosing Syscall Bottlenecks from strace -c Output

Running time and strace -c on a file-processing utility over a 10 MB workload reports real = 0.60 s, user = 0.15 s, sys = 0.45 s, and 100,000 read()/write() syscalls averaging 4.0 microseconds each in kernel/mode-switch time.

  1. Is the program CPU-bound or kernel/syscall-bound? Compute the total time spent in those read()/write() syscalls.
  2. If you batch transfers using 20x larger buffers (reducing syscalls from 100,000 to 5,000 at the same 4.0 us/call), compute the new syscall time and the speedup on syscall overhead.

Part 1 — Bottleneck Diagnosis & Baseline Syscall Time:

  • sys = 0.45 s is 3x larger than user = 0.15 s, accounting for 75% of total wall time -> the program is syscall/kernel-bound rather than user-CPU-bound.
  • Baseline syscall time = 100,000 * 4.0 us = 400,000 us = 0.40 seconds.

Part 2 — Impact of 20x Batching:

  • Batched syscall count = 100,000 / 20 = 5,000 syscalls.
  • New syscall time = 5,000 * 4.0 us = 20,000 us = 0.02 seconds.
  • Syscall time drops from 0.40 s to 0.02 s (a 20x speedup on syscall overhead, cutting total runtime from 0.60 s to ~0.22 s).
Answer:
  1. Syscall/kernel-bound (sys = 0.45 s vs user = 0.15 s); baseline syscall time = 0.40 s.
  2. Batching reduces syscall time to 0.02 s (20x reduction in syscall overhead).

Exercises

Practice Exercises — System-Level I/O & Optimization

Question 1

Explain the purpose of writev() / readv(), pread() / pwrite(), and fdatasync() vs. fsync(). Give one concrete network or storage server scenario where each outperforms standard read()/write()/fsync().

Role of Each Advanced I/O Call:

  • writev() / readv() (Vectored I/O): Scatters/gathers multiple non-contiguous buffers (struct iovec[]) in one syscall. Scenario: transmitting an HTTP header + separate response body buffer over a TCP socket in one atomic syscall without memcpy().
  • pread() / pwrite() (Positional I/O): Reads/writes at an explicit byte offset without changing the shared file offset. Scenario: multiple worker threads reading random B-tree database pages concurrently from a single shared fd without mutex locking around lseek().
  • fdatasync() vs. fsync(): fsync() flushes data plus all metadata (mtime), whereas fdatasync() flushes data and only size/retrieval metadata. Scenario: pre-allocated database Write-Ahead Logs (WAL) gain lower commit latency using fdatasync().
Answer:

writev()/readv() coalesce non-contiguous buffers in one syscall (e.g., HTTP header + body); pread()/pwrite() perform thread-safe offset I/O on a shared fd without locks; fdatasync() skips non-essential timestamp metadata flushes for faster WAL durability than fsync().

Question 2

Distinguish between non-blocking multiplexed I/O (epoll) and true asynchronous I/O (io_uring / POSIX AIO). Why can epoll prevent thread blocking on TCP sockets, yet still block on regular disk files?

Readiness Notification vs. Completion Notification:

  • epoll (Non-blocking multiplexed): Readiness-based. epoll_wait() alerts the thread when a socket can be read/written without blocking, and the application still invokes recv()/send() syscalls to move the bytes.
  • True Asynchronous I/O (io_uring / POSIX AIO): Completion-based. The application submits I/O requests into a submission queue; the kernel executes the transfer asynchronously in the background and posts a completion event only after bytes are already copied into/out of the user buffer.

Why Disk Files Still Block Under epoll: In POSIX/Linux, regular disk files do not track socket-style readiness buffers: select()/poll()/epoll() either reject regular files (EPERM) or always report them as ready, and O_NONBLOCK is ignored for regular file read()—so a page-cache miss forces the thread to sleep waiting on storage hardware. True AIO (io_uring) offloads disk reads inside the kernel without stalling the caller's event loop.

Answer:

epoll reports readiness before you call recv()/send(), whereas true AIO (io_uring) performs the I/O asynchronously and reports completion afterward. Regular disk files ignore O_NONBLOCK and stall on page-cache misses under epoll, whereas io_uring executes disk and socket I/O asynchronously via shared SQ/CQ ring buffers.

Question 3

On a system with PAGESIZE = 4096 bytes, a program needs to read 3,000 bytes starting at file offset 25,000 using mmap().

  1. Calculate the page-aligned aligned_offset, the byte adjustment delta, the minimum map_len, and how many 4 KiB pages this mapping spans.
  2. Contrast MAP_SHARED and MAP_PRIVATE and name the signal raised if another process truncates the file shorter than the accessed mapping.

Part 1 — Page-Alignment Calculation:

  • aligned_offset = (25000 // 4096) * 4096 = 6 * 4096 = 24,576.
  • delta = 25000 - 24576 = 424 bytes.
  • map_len = 424 + 3000 = 3,424 bytes.
  • Pages spanned = ceil(3424 / 4096) = 1 page (spanning file bytes 24,576 to 28,671).

Part 2 — MAP_SHARED vs. MAP_PRIVATE and SIGBUS:

  • MAP_SHARED shares the kernel page cache so writes propagate to the file and other processes.
  • MAP_PRIVATE performs Copy-on-Write so modifications remain private in RAM and never modify the file.
  • Accessing a mapped address beyond the current file size after another process shrinks/truncates the file triggers SIGBUS.
Answer:
  1. aligned_offset = 24,576, delta = 424, map_len = 3,424 bytes (1 4-KiB page).
  2. MAP_SHARED writes back to the file; MAP_PRIVATE isolates writes via Copy-on-Write. Accessing past a truncated file raises SIGBUS.
Question 4

Distinguish between advisory and mandatory file locking. Compare flock() and fcntl() byte-range locking, and explain the classic POSIX fcntl() close-descriptor pitfall.

Advisory vs. Mandatory Locking:

  • Advisory: Enforced only by mutual cooperation—processes must explicitly acquire/check locks via flock()/fcntl(), while plain read()/write() calls ignore them.
  • Mandatory: Kernel blocks uncooperative read()/write() calls conflicting with a lock (rarely supported and deprecated on Linux).

flock() vs. fcntl() and the Close Pitfall:

  • flock() locks the entire file and attaches ownership to the Open File Table entry (dups/forks share it).
  • fcntl(F_SETLK) supports byte-range locking (l_start, l_len), yet attaches ownership to the (PID, inode) pair. Consequently, if a process has fd1 locked and opens and closes any other fd2 pointing to that same file, close(fd2) silently drops all fcntl locks held on that file! (Solved on Linux by F_OFD_SETLK.)
Answer:

Advisory locks require cooperating processes; mandatory locks are kernel-enforced. flock() locks whole files per Open File Table entry, whereas fcntl() locks byte ranges per (PID, inode)—meaning closing any descriptor to the same file in that process silently releases all its fcntl() locks.

Question 5
  1. A high-frequency service tracks stats for 8 worker threads in an array struct { long count; } stats[8];. Explain why false sharing degrades multi-core scaling even though there is no mutex, and compute the total array size if each counter is padded to 64-byte cache lines.
  2. An unbatched server takes 200 us (0.20 ms) per request (5,000 req/s). Syscall batching reduces the average cost to 50 us per request. Calculate the total time to process 4,000 requests after batching (in ms) and the resulting throughput (in req/s).

Part 1 — False Sharing & 64-Byte Cache-Line Padding:

  • Because sizeof(long) = 8 bytes, all 8 threads' counters fit inside a single 64-byte CPU L1 cache line (8 * 8 = 64 bytes). Whenever Thread 0 increments stats[0].count, hardware cache coherency (MESI) invalidates the entire cache line on Cores 1–7, causing continuous cache-line bouncing despite zero shared variables!
  • Padding each struct to 64 bytes isolates each counter on its own cache line; total array size = 8 * 64 = 512 bytes.

Part 2 — Batched Processing Latency & Throughput:

  • Total time for 4,000 requests at 50 us each = (50 * 4,000) / 1,000 = 200.0 ms (0.20 s).
  • Resulting throughput = 4,000 requests / 0.20 s = 20,000 req/s (4x improvement over 5,000 req/s).
Answer:
  1. False sharing occurs because adjacent 8-byte counters share one 64-byte cache line, triggering constant MESI invalidations; padding each entry to 64 bytes yields 512 bytes.
  2. Batched time for 4,000 requests = 200.0 ms, achieving 20,000 req/s throughput.
← Advanced Network ProgrammingNetwork Application Development and Case Studies →