System-Level I/O and Performance Optimization
Open in the Padhaiko app6.1 Advanced file I/O (Read/write optimization)
Standard read() and write() calls become bottlenecks when servers handle high-throughput logs, database pages, or multi-part HTTP responses. Linux/POSIX provides specialized I/O calls and caching controls to reduce system call overhead, memory copies, and lock contention.
1. Kernel Page Cache, fsync(), and fdatasync()
By default, POSIX file I/O goes through the kernel Page Cache: write() copies bytes into RAM pages marked "dirty" and returns immediately while the kernel flushes them to disk asynchronously later. For crash-safe databases and transaction logs:
fsync(fd): Flushes both dirty data blocks and file metadata (inode timestamps, size) to physical disk before returning.fdatasync(fd): Flushes dirty data blocks and only essential metadata needed to retrieve the data (skipping non-criticalmtime/atimeupdates), saving an extra disk seek/flush.
2. Vectored (Scatter-Gather) I/O: readv() and writev()
When a server builds an HTTP response (a 200-byte header buffer + a 4 KB JSON body buffer), calling write() twice incurs two syscalls and risks sending two tiny TCP packets; calling memcpy() into one combined buffer wastes CPU. Vectored I/O transfers an array of struct iovec buffers in one atomic system call:
#include <sys/uio.h>
struct iovec iov[2];
iov[0].iov_base = http_header;
iov[0].iov_len = hdr_len;
iov[1].iov_base = json_body;
iov[1].iov_len = body_len;
ssize_t total_sent = writev(sock_fd, iov, 2); /* single syscall, atomic write */3. Positional I/O: pread() and pwrite()
Normal lseek() + read() is not thread-safe when multiple threads share the same file descriptor, because another thread can shift the shared file offset between lseek() and read(). pread(fd, buf, count, offset) and pwrite(fd, buf, count, offset) atomically read/write at an explicit byte offset in one syscall without modifying the file descriptor's offset.
4. Cache Hints (posix_fadvise()) and Direct I/O (O_DIRECT)
posix_fadvise(fd, offset, len, advice): Hints future access patterns to the kernel (POSIX_FADV_SEQUENTIALdoubles read-ahead;POSIX_FADV_RANDOMdisables read-ahead;POSIX_FADV_DONTNEEDevicts streamed pages so huge backups do not pollute the page cache).O_DIRECT: Bypasses the kernel page cache entirely, transferring directly between aligned user buffers (posix_memalign) and the storage controller via DMA—used by databases (PostgreSQL, MySQL InnoDB) that maintain their own user-space buffer pools and want to avoid double-caching.
6.2 Non-blocking I/O and asynchronous I/O concepts
It is vital not to confuse Non-Blocking Multiplexed I/O with True Asynchronous I/O:
- Why
select()/poll()/epoll()Fail on Local Disk Files:O_NONBLOCKand readiness multiplexers monitor kernel socket/pipe buffers. Regular disk files are always reported as immediately "ready" byselect()/epoll(), yet callingread()on an uncached disk block still blocks the thread synchronously while waiting for physical NVMe/HDD I/O! - True Asynchronous I/O (AIO): The application submits an I/O request ("read 64 KB at offset X into
buf") and returns immediately; the kernel performs the entire disk/network transfer in the background and delivers a completion event once the bytes are already sitting inbuf.
| Aspect | Non-Blocking Multiplexed I/O (epoll) | True Asynchronous I/O (POSIX AIO / io_uring) |
|---|---|---|
| Notification Timing | Readiness-based: notifies before syscall ("socket is ready to read now") | Completion-based: notifies after kernel finished copying data into your buffer |
| Syscall Cost | epoll_wait() + separate recv()/send() syscalls | Batched submissions & completions via shared ring buffers (zero syscalls with SQPOLL) |
| Regular Disk File Support | No (disk read() still stalls on cache miss) | Yes (works uniformly for disk files, pipes, and sockets) |
POSIX AIO vs. Linux io_uring
- POSIX AIO (
<aio.h>:aio_read,aio_write,aio_error,aio_return): Portable specification usingstruct aiocb; on Linux glibc it is implemented using a user-space thread pool rather than true kernel async hooks. - Linux
io_uring: Modern Linux asynchronous interface built around two lockless ring buffers mapped directly between user space and kernel space—a Submission Queue (SQ) and a Completion Queue (CQ). Applications push dozens ofread,write,accept, orsendentries into the SQ and harvest finished results from the CQ with minimal or zero mode switches.
6.3 Memory-mapped files (Mmap)
mmap() maps a file (or anonymous RAM / POSIX shared memory) directly into the calling process's virtual address space so file bytes can be read and modified using ordinary C memory pointers instead of read()/write() syscalls.
void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);
int msync(void *addr, size_t length, int flags); /* flush dirty pages to disk */
int munmap(void *addr, size_t length); /* unmap region */Key Parameters and Rules of mmap()
- Protection (
prot):PROT_READ,PROT_WRITE,PROT_EXEC, orPROT_NONE. - Visibility (
flags):MAP_SHARED: Writes modify the underlying kernel page cache directly, becoming visible to other processes mapping the file and flushed back to disk.MAP_PRIVATE: Uses Copy-on-Write (CoW)—modifications stay private to the process and never write back to the file (used by the dynamic linker for loading.soshared libraries).
- Mandatory Page Alignment:
offsetMUST be an exact multiple of the system page size (sysconf(_SC_PAGESIZE), typically4096bytes on x86-64); otherwisemmap()fails withEINVAL. - Demand Paging:
mmap()sets up virtual page-table entries lazily without reading the whole file upfront; physical 4 KB pages are loaded on demand when first touched (minor/major page fault).
6.4 File locking and concurrency
When multiple processes concurrently read or update a shared configuration file, log, or database file, race conditions corrupt file contents unless access is coordinated with file locks.
Advisory vs. Mandatory Locking
- Advisory Locking (Default on Linux/Unix): Locks work only when all processes cooperate by calling
flock()orfcntl()before accessing the file. An uncooperative process calling plainwrite()ignores advisory locks completely. - Mandatory Locking: Enforced by the kernel on every
read()/write()syscall (deprecated/removed in modern Linux kernels due to race/DoS issues).
flock() vs. fcntl() Byte-Range Locking
| Aspect | flock() (BSD Whole-File Lock) | fcntl() (POSIX Byte-Range Lock) |
|---|---|---|
| Granularity | Entire file only | Arbitrary byte range (l_start, l_len) |
| Lock Types | LOCK_SH (shared), LOCK_EX (exclusive), LOCK_UN | F_RDLCK (shared), F_WRLCK (exclusive), F_UNLCK |
| Non-Blocking Flag | LOCK_NB flag (fails with EWOULDBLOCK) | F_SETLK (fails with EAGAIN/EACCES) vs F_SETLKW (waits) |
| Lock Ownership | Associated with Open File Table entry (preserved across dup/fork) | Associated with (PID, inode) pair! |
| Classic Pitfall | Does not work reliably on legacy NFS mounts | Closing ANY fd to that file in the process silently releases all fcntl locks held by that process on the file! |
6.5 Performance tuning and profiling basics
Optimizing system and network software relies on measurement before tuning—identifying whether an application is CPU-bound, syscall-bound, I/O-bound, or lock-contended.
| Tool | What It Measures | Typical Diagnosis |
|---|---|---|
time ./prog | Wall (real), user-space (user), and kernel (sys) CPU time | High sys time relative to user indicates excessive syscalls or page faults |
strace -c ./prog | Exact count, total time, and error count per system call | Reveals 1-byte read/write loops, unnecessary stat()/lseek(), or high futex lock waits |
perf top / perf record | Hardware CPU sampling of functions (user + kernel) & Flame Graphs | Pinpoints hot loops, serialization overhead, or kernel TCP copy bottlenecks |
vmstat 1 / iostat -xz 1 | Context switches (cs), CPU %wa (I/O wait), disk queue depth, throughput | Distinguishes disk bandwidth saturation from CPU starvation |
valgrind (cachegrind/massif) | L1/LLC cache misses, branch mispredictions, peak heap usage | Detects pointer-chasing cache misses and memory leaks |
Core Tuning Principles
- Batch and Coalesce System Calls: Replace loops of tiny
read()/write()calls with4–64 KBbuffers,writev(),sendfile(), orio_uringbatching to minimize user-kernel mode switches. - Respect Page and Cache-Line Boundaries: Align
O_DIRECTbuffers to 4096 bytes; pad per-thread counters to 64-byte CPU cache lines (alignas(64)) to prevent false sharing across cores. - Eliminate Lock Contention: Shrink critical sections, use per-core sharded data structures or lock-free queues instead of one global mutex, and keep slow disk/network I/O outside locks.
Worked Examples
Calculating Page-Aligned Offsets and Lengths for mmap()
On a Linux system with page size PAGESIZE = 4096 bytes, an application wants to mmap() and read a 512-byte binary header starting at file byte offset 10,000.
- Why will passing
offset = 10000directly tommap()fail? - Compute the required
aligned_offset, thedeltainside the mapped window, and the minimummap_lento pass tommap(), and write the C pointer expression to access byte10,000.
Why Direct offset = 10000 Fails: mmap() requires offset to be an exact integer multiple of PAGESIZE (4096). Because 10000 % 4096 = 1808 != 0, calling mmap(..., offset = 10000) fails with -1 (MAP_FAILED) and errno = EINVAL.
Computing Aligned Offset, Delta, and Map Length:
aligned_offset = (10000 // 4096) * 4096 = 2 * 4096 = 8192.delta = 10000 - 8192 = 1808bytes.map_len = delta + 512 = 1808 + 512 = 2320bytes (the kernel rounds this up internally to 1 page of 4096 bytes, since2320 <= 4096).- In C:
char *base = mmap(NULL, 2320, PROT_READ, MAP_PRIVATE, fd, 8192); char *hdr = base + 1808;
10,000is not a multiple of4096, causingmmap()to fail withEINVAL.aligned_offset = 8192,delta = 1808,map_len = 2320bytes; access viabase + 1808(and later unmap withmunmap(base, 2320)).
Atomic Append with Advisory fcntl() Byte-Range Lock and writev()
Write a C function safe_log_record(int fd, const char *tag, const char *msg) that acquires an exclusive blocking POSIX write lock (F_WRLCK via F_SETLKW) across the file, writes tag and msg atomically in one syscall using writev(), flushes data with fdatasync(), and unlocks.
Configure struct flock and Vectored iovec: Set l_type = F_WRLCK, l_whence = SEEK_SET, l_start = 0, l_len = 0 (l_len = 0 means lock from l_start to EOF, including future appends!).
C Implementation (safe_log_record):
#include <fcntl.h>
#include <string.h>
#include <sys/uio.h>
#include <unistd.h>
int safe_log_record(int fd, const char *tag, const char *msg) {
struct flock fl = {
.l_type = F_WRLCK,
.l_whence = SEEK_SET,
.l_start = 0,
.l_len = 0 /* 0 locks through EOF */
};
if (fcntl(fd, F_SETLKW, &fl) < 0) return -1;
lseek(fd, 0, SEEK_END);
struct iovec iov[3] = {
{ .iov_base = (void *)tag, .iov_len = strlen(tag) },
{ .iov_base = (void *)msg, .iov_len = strlen(msg) },
{ .iov_base = (void *)"\n", .iov_len = 1 }
};
ssize_t rc = writev(fd, iov, 3);
if (rc >= 0) fdatasync(fd);
fl.l_type = F_UNLCK;
fcntl(fd, F_SETLK, &fl);
return (rc < 0) ? -1 : 0;
}fcntl(fd, F_SETLKW, &fl) with F_WRLCK blocks until exclusive access is granted; writev() writes tag, msg, and "\n" in a single syscall; fdatasync() persists the data efficiently without unnecessary metadata flushes; and F_UNLCK releases the lock.
Diagnosing Syscall Bottlenecks from strace -c Output
Running time and strace -c on a file-processing utility over a 10 MB workload reports real = 0.60 s, user = 0.15 s, sys = 0.45 s, and 100,000 read()/write() syscalls averaging 4.0 microseconds each in kernel/mode-switch time.
- Is the program CPU-bound or kernel/syscall-bound? Compute the total time spent in those
read()/write()syscalls. - If you batch transfers using 20x larger buffers (reducing syscalls from
100,000to5,000at the same4.0 us/call), compute the new syscall time and the speedup on syscall overhead.
Part 1 — Bottleneck Diagnosis & Baseline Syscall Time:
sys = 0.45 sis3xlarger thanuser = 0.15 s, accounting for75%of total wall time -> the program is syscall/kernel-bound rather than user-CPU-bound.- Baseline syscall time =
100,000 * 4.0 us = 400,000 us = 0.40 seconds.
Part 2 — Impact of 20x Batching:
- Batched syscall count =
100,000 / 20 = 5,000syscalls. - New syscall time =
5,000 * 4.0 us = 20,000 us = 0.02 seconds. - Syscall time drops from
0.40 sto0.02 s(a 20x speedup on syscall overhead, cutting total runtime from0.60 sto~0.22 s).
- Syscall/kernel-bound (
sys = 0.45 svsuser = 0.15 s); baseline syscall time = 0.40 s. - Batching reduces syscall time to 0.02 s (20x reduction in syscall overhead).
Exercises
Practice Exercises — System-Level I/O & Optimization
Explain the purpose of writev() / readv(), pread() / pwrite(), and fdatasync() vs. fsync(). Give one concrete network or storage server scenario where each outperforms standard read()/write()/fsync().
Role of Each Advanced I/O Call:
writev()/readv()(Vectored I/O): Scatters/gathers multiple non-contiguous buffers (struct iovec[]) in one syscall. Scenario: transmitting an HTTP header + separate response body buffer over a TCP socket in one atomic syscall withoutmemcpy().pread()/pwrite()(Positional I/O): Reads/writes at an explicit byte offset without changing the shared file offset. Scenario: multiple worker threads reading random B-tree database pages concurrently from a single sharedfdwithout mutex locking aroundlseek().fdatasync()vs.fsync():fsync()flushes data plus all metadata (mtime), whereasfdatasync()flushes data and only size/retrieval metadata. Scenario: pre-allocated database Write-Ahead Logs (WAL) gain lower commit latency usingfdatasync().
writev()/readv() coalesce non-contiguous buffers in one syscall (e.g., HTTP header + body); pread()/pwrite() perform thread-safe offset I/O on a shared fd without locks; fdatasync() skips non-essential timestamp metadata flushes for faster WAL durability than fsync().
Distinguish between non-blocking multiplexed I/O (epoll) and true asynchronous I/O (io_uring / POSIX AIO). Why can epoll prevent thread blocking on TCP sockets, yet still block on regular disk files?
Readiness Notification vs. Completion Notification:
epoll(Non-blocking multiplexed): Readiness-based.epoll_wait()alerts the thread when a socket can be read/written without blocking, and the application still invokesrecv()/send()syscalls to move the bytes.- True Asynchronous I/O (
io_uring/ POSIX AIO): Completion-based. The application submits I/O requests into a submission queue; the kernel executes the transfer asynchronously in the background and posts a completion event only after bytes are already copied into/out of the user buffer.
Why Disk Files Still Block Under epoll: In POSIX/Linux, regular disk files do not track socket-style readiness buffers: select()/poll()/epoll() either reject regular files (EPERM) or always report them as ready, and O_NONBLOCK is ignored for regular file read()—so a page-cache miss forces the thread to sleep waiting on storage hardware. True AIO (io_uring) offloads disk reads inside the kernel without stalling the caller's event loop.
epoll reports readiness before you call recv()/send(), whereas true AIO (io_uring) performs the I/O asynchronously and reports completion afterward. Regular disk files ignore O_NONBLOCK and stall on page-cache misses under epoll, whereas io_uring executes disk and socket I/O asynchronously via shared SQ/CQ ring buffers.
On a system with PAGESIZE = 4096 bytes, a program needs to read 3,000 bytes starting at file offset 25,000 using mmap().
- Calculate the page-aligned
aligned_offset, the byte adjustmentdelta, the minimummap_len, and how many 4 KiB pages this mapping spans. - Contrast
MAP_SHAREDandMAP_PRIVATEand name the signal raised if another process truncates the file shorter than the accessed mapping.
Part 1 — Page-Alignment Calculation:
aligned_offset = (25000 // 4096) * 4096 = 6 * 4096 = 24,576.delta = 25000 - 24576 = 424bytes.map_len = 424 + 3000 = 3,424bytes.- Pages spanned =
ceil(3424 / 4096) = 1page (spanning file bytes24,576to28,671).
Part 2 — MAP_SHARED vs. MAP_PRIVATE and SIGBUS:
MAP_SHAREDshares the kernel page cache so writes propagate to the file and other processes.MAP_PRIVATEperforms Copy-on-Write so modifications remain private in RAM and never modify the file.- Accessing a mapped address beyond the current file size after another process shrinks/truncates the file triggers
SIGBUS.
aligned_offset = 24,576,delta = 424,map_len = 3,424bytes (1 4-KiB page).MAP_SHAREDwrites back to the file;MAP_PRIVATEisolates writes via Copy-on-Write. Accessing past a truncated file raisesSIGBUS.
Distinguish between advisory and mandatory file locking. Compare flock() and fcntl() byte-range locking, and explain the classic POSIX fcntl() close-descriptor pitfall.
Advisory vs. Mandatory Locking:
- Advisory: Enforced only by mutual cooperation—processes must explicitly acquire/check locks via
flock()/fcntl(), while plainread()/write()calls ignore them. - Mandatory: Kernel blocks uncooperative
read()/write()calls conflicting with a lock (rarely supported and deprecated on Linux).
flock() vs. fcntl() and the Close Pitfall:
flock()locks the entire file and attaches ownership to the Open File Table entry (dups/forks share it).fcntl(F_SETLK)supports byte-range locking (l_start,l_len), yet attaches ownership to the(PID, inode)pair. Consequently, if a process hasfd1locked and opens and closes any otherfd2pointing to that same file,close(fd2)silently drops allfcntllocks held on that file! (Solved on Linux byF_OFD_SETLK.)
Advisory locks require cooperating processes; mandatory locks are kernel-enforced. flock() locks whole files per Open File Table entry, whereas fcntl() locks byte ranges per (PID, inode)—meaning closing any descriptor to the same file in that process silently releases all its fcntl() locks.
- A high-frequency service tracks stats for 8 worker threads in an array
struct { long count; } stats[8];. Explain why false sharing degrades multi-core scaling even though there is no mutex, and compute the total array size if each counter is padded to 64-byte cache lines. - An unbatched server takes 200 us (0.20 ms) per request (
5,000 req/s). Syscall batching reduces the average cost to 50 us per request. Calculate the total time to process 4,000 requests after batching (in ms) and the resulting throughput (in req/s).
Part 1 — False Sharing & 64-Byte Cache-Line Padding:
- Because
sizeof(long) = 8 bytes, all 8 threads' counters fit inside a single 64-byte CPU L1 cache line (8 * 8 = 64 bytes). Whenever Thread 0 incrementsstats[0].count, hardware cache coherency (MESI) invalidates the entire cache line on Cores 1–7, causing continuous cache-line bouncing despite zero shared variables! - Padding each struct to
64bytes isolates each counter on its own cache line; total array size =8 * 64 = 512 bytes.
Part 2 — Batched Processing Latency & Throughput:
- Total time for
4,000requests at50 useach =(50 * 4,000) / 1,000 = 200.0 ms(0.20 s). - Resulting throughput =
4,000 requests / 0.20 s = 20,000 req/s(4ximprovement over5,000 req/s).
- False sharing occurs because adjacent 8-byte counters share one 64-byte cache line, triggering constant MESI invalidations; padding each entry to 64 bytes yields 512 bytes.
- Batched time for 4,000 requests = 200.0 ms, achieving 20,000 req/s throughput.