PadhaikoAll subjects

Advanced Network Programming

Network and Systems Programming (ENCT 386)

Open in the Padhaiko app

5.1 Concurrent server design: Iterative versus concurrent servers, multi-process and multi-threaded servers

A network server must choose how to schedule incoming client connections. The simplest design handles one client at a time; scalable servers handle thousands simultaneously.

AspectIterative ServerConcurrent Server
Execution FlowAccepts 1 client, finishes full request/response, closes, then accepts nextAccepts clients and processes multiple active connections simultaneously
Head-of-Line BlockingSevere — one slow or idle client stalls every waiting clientEliminated — slow clients do not block fast clients
ComplexityMinimal; single process/thread, no synchronization neededHigher; requires fork(), threads, synchronization, or event loops
Best FitShort stateless UDP lookups (e.g., NTP, simple DNS)HTTP servers, chat servers, SSH, database proxies, streaming

Concurrent Server Architectures

  1. Multi-Process Server (fork()-per-connection): The parent process loops on accept(). For each new conn_fd, it calls fork(): the child closes listen_fd, services conn_fd, and exits; the parent closes conn_fd immediately (decrementing kernel refcount so the socket closes when the child finishes) and reaps exited children via SIGCHLD + waitpid(-1, NULL, WNOHANG).
    • Pros: Strong fault isolation (one crashed worker does not crash other clients).
    • Cons: High fork() + IPC overhead if shared state is needed.
  2. Multi-Threaded Server (pthread-per-connection): Spawns a detached pthread per accepted client. Shares heap caches directly, with lower creation/switch overhead than fork(), yet spawning 10,000+ unbounded threads wastes gigabytes of stack RAM and triggers severe scheduler thrashing.
  3. Pre-Forked / Bounded Thread-Pool Server: Allocates a fixed pool of W worker processes or threads at startup. An acceptor pushes conn_fds onto a mutex/condvar-guarded queue (or workers share listen_fd using SO_REUSEPORT). Eliminates per-connection spawn latency and bounds memory usage.

5.2 Event-driven programming: Select, poll, epoll

Even with a thread pool, one-thread-per-connection hits a wall around 10,000 idle or slow connections (the C10K problem). Event-driven servers (the Reactor pattern, used by Nginx, Redis, and Node.js) handle tens of thousands of non-blocking sockets inside one thread (or one event loop per CPU core).

While select() and poll() re-scan the entire array of monitored descriptors (O(N)) on every single wakeup, Linux's epoll (<sys/epoll.h>) maintains an interest list inside the kernel (red-black tree) and a ready list (linked list), delivering O(1) readiness notification proportional only to the number of active sockets.

Core epoll System Calls

  1. int epoll_create1(int flags); — Creates an epoll kernel instance and returns epfd.
  2. int epoll_ctl(int epfd, int op, int fd, struct epoll_event *ev); — Registers (EPOLL_CTL_ADD), modifies (EPOLL_CTL_MOD), or removes (EPOLL_CTL_DEL) a socket fd with bitmask events (EPOLLIN, EPOLLOUT, EPOLLET, EPOLLRDHUP).
  3. int epoll_wait(int epfd, struct epoll_event *events, int maxevents, int timeout_ms); — Blocks until at least one registered fd becomes ready and fills events[0 .. nready-1].

Level-Triggered (LT) vs. Edge-Triggered (EPOLLET)

Definition

Level-Triggered vs. Edge-Triggered epoll

Comparison of select(), poll(), and epoll()
Featureselect()poll()epoll (Linux)
Kernel TrackingStateless (re-passed on every call)Stateless (re-passed on every call)Stateful (epoll_ctl registers once)
Time ComplexityO(max_fd) per callO(N) per callO(k) (k = ready fds only)
Max FD LimitFD_SETSIZE (1024)Unlimited (RLIMIT_NOFILE)Unlimited (RLIMIT_NOFILE)
Trigger ModesLevel-triggered onlyLevel-triggered onlyLevel-triggered & Edge-triggered (EPOLLET)
Best Scale< 100 sockets, max portability100–1,000 sockets, POSIX portable10,000–1,000,000+ connections on Linux

5.3 High-performance server design principles

Scaling a network server from hundreds to hundreds of thousands of requests per second combines kernel bypasses, lock avoidance, and protocol tuning:

  1. Zero-Copy Data Transfer (sendfile() / splice()): Traditional file serving calls read(file_fd, buf) (disk -> kernel page cache -> user buffer) followed by send(sock_fd, buf) (user buffer -> kernel socket buffer -> NIC). sendfile(out_fd, in_fd, &offset, count) transfers bytes directly inside kernel space from page cache to NIC DMA—cutting 4 user/kernel copies and 4 mode switches down to zero user copies!
  2. Thread-per-Core + SO_REUSEPORT: Pinning one event-loop worker process/thread per CPU core avoids cross-core mutex contention and cache-line bouncing. Setting SO_REUSEPORT allows every worker to bind its own listening socket on the exact same IP/port; the Linux kernel hashes incoming 4-tuples across workers directly, eliminating single-acceptor lock bottlenecks.
  3. TCP Protocol Tuning:
    • TCP_NODELAY: Disables Nagle's algorithm so small interactive packets (RPC messages, keystrokes, game state) are sent immediately instead of waiting up to 40–200 ms for delayed ACKs.
    • SOMAXCONN / net.core.somaxconn: Increases listen() Accept queue depth so sudden connection bursts do not drop SYNs.
  4. Bounded Queues & Backpressure: Never allow an unbounded queue between network I/O and slow disk/DB backends; cap queue length and shed load early (503 Service Unavailable) to prevent Out-Of-Memory (OOM) crashes.
  5. Memory Pools & Connection Pooling: Pre-allocate fixed-size packet/connection slabs to avoid malloc()/free() lock contention, and reuse persistent backend TCP connections instead of paying 3-way handshake + TLS costs per request.

5.4 Concepts of remote procedure call (RPC)

A Remote Procedure Call (RPC) allows a client program to invoke a function located in another address space (usually on a remote server across a network) using familiar local function-call syntax, hiding socket I/O, byte ordering, and packet framing behind generated stubs.

RPC Execution Flow (Stub Generation & Marshaling)Text
+----------------+                                       +----------------+
| Client App     |                                       | Server App     |
| sum = add(3,5) |                                       | int add(a, b)  |
+-------+--------+                                       +-------+--------+
        | (1) local call                                         ^ (6) local call
        v                                                        |
+----------------+                                       +----------------+
| Client Stub    |                                       | Server Stub    |
| Marshal (3, 5) |                                       | Unmarshal(3,5) |
+-------+--------+                                       +-------+--------+
        | (2) serialize (Protobuf/XDR)                           ^ (5) decode
        v                                                        |
+----------------+    (3) Request over TCP / HTTP2       +----------------+
| Client OS/Net  | ====================================> | Server OS/Net  |
| Socket Send    | <==================================== | Socket Recv    |
+----------------+    (4) Response (sum = 8)             +----------------+

Key RPC Components and Semantics


5.5 Introduction to RESTful APIs and microservices

RESTful APIs (Representational State Transfer)

REST is an architectural style for distributed hypermedia systems (defined by Roy Fielding) layered directly over standard HTTP/HTTPS:

  1. Resource-Oriented URIs: Endpoints identify nouns/resources (/api/v1/orders/42), not verbs (/doCreateOrder).
  2. Uniform HTTP Methods:
    • GET: Retrieve resource (safe and idempotent).
    • POST: Create new resource / trigger action (not idempotent).
    • PUT: Full replacement of a resource (idempotent).
    • PATCH: Partial update.
    • DELETE: Remove resource (idempotent).
  3. Statelessness: Every HTTP request carries all authentication (Authorization: Bearer <JWT>) and context needed; the server stores zero per-client session state in RAM, letting load balancers route any request to any backend instance.
  4. Standard Status Codes & Representations: 200 OK, 201 Created, 400 Bad Request, 401 Unauthorized, 404 Not Found, 500 Internal Server Error, typically carrying application/json payloads.
AspectRPC (e.g., gRPC / HTTP2)RESTful API (HTTP / JSON)
Design FocusAction / procedure-oriented (CreateUser())Resource / noun-oriented (POST /users)
Wire FormatBinary (Protocol Buffers) — compact, strongly typedText (JSON) — human-readable, schema-flexible
Transport & StreamingHTTP/2 multiplexed bidirectional streamingHTTP/1.1 or HTTP/2 request-response
Best Use CaseInternal high-throughput microservice-to-microservice callsPublic web APIs, browser/mobile clients, third-party integrations

Microservices Architecture

Rather than packaging an entire backend into a single monolith, a microservices architecture splits the application into small, autonomous services aligned with business domains (e.g., Auth Service, Catalog Service, Payment Service), each owning its own database and communicating over REST or gRPC.


5.6 Secure communication basics (SSL/TLS overview)

Plain TCP transmits every byte—passwords, tokens, and session cookies—in cleartext, exposing traffic to eavesdropping and tampering. Transport Layer Security (TLS) (successor to SSL) sits immediately above TCP (below HTTP/gRPC/SMTP) to guarantee three cryptographic properties:

  1. Confidentiality: Symmetric authenticated encryption (AES-GCM or ChaCha20-Poly1305) hides payload bytes from wire sniffers.
  2. Data Integrity: Cryptographic authentication tags (AEAD / HMAC) detect any bit modification or packet injection in transit.
  3. Authentication: Server presents an X.509 Digital Certificate containing its public key and domain name, cryptographically signed by a trusted Certificate Authority (CA) pre-installed in the OS/browser trust store.

How the TLS Handshake Works (TLS 1.3 vs. TLS 1.2)

TLS 1.3 1-RTT Handshake Over TCPText
Client                                                    Server
  | ---- TCP SYN -----------------------------------------> |
  | <--- TCP SYN-ACK -------------------------------------- |  (1 RTT TCP)
  | ---- TCP ACK + TLS ClientHello (ciphers + KeyShare) --> |
  |                                                         |  Derives shared secret
  | <--- ServerHello + KeyShare + EncryptedExtensions  ---- |  via ECDHE
  | <--- Certificate + CertificateVerify + Finished    ---- |  (1 RTT TLS 1.3)
  | ---- TLS Finished + [Encrypted HTTP Request] ---------> |
  | <=== Encrypted Application Data (AES-GCM / ChaCha20) => |

Worked Examples

Example

Little's Law Thread-Pool Sizing and sendfile() Copy Reduction

  1. A concurrent HTTP API server receives an average arrival rate of 6,000 requests/second, and each request takes an average of 25 ms (0.025 s) to process (including database query wait). Using Little's Law (L = lambda * W), estimate the minimum number of concurrent worker threads needed so requests do not queue indefinitely.
  2. Suppose serving a static 1 MiB file in 100 chunks via read() + send() requires 4 memory copies (disk -> kernel -> user -> kernel -> NIC) per chunk, whereas sendfile() reduces this to 2 DMA/kernel operations (disk -> kernel page cache -> NIC). Compare total copy operations across 100 chunks.

Part 1 — Applying Little's Law (L = lambda * W):

  • lambda = 6,000 req/s, W = 0.025 s.
  • Minimum concurrent in-flight workers L = 6,000 * 0.025 = 150 workers (plus safety headroom, e.g., 200 workers, or a single epoll event loop).

Part 2 — Comparing read()/send() vs. sendfile() Copy Counts:

  • Traditional read() + send() over 100 chunks: 100 * 4 = 400 data-copy steps (plus 200 user-kernel mode switches).
  • Zero-copy sendfile() over 100 chunks (or 1 single sendfile() call!): 100 * 2 = 200 DMA/internal operations with zero user-space buffer copies (cutting copies in half and eliminating user-space RAM allocation).
Answer:
  1. Minimum concurrent workers needed = 150 threads (6,000 * 0.025).
  2. read()/send() performs 400 copy steps vs. 200 kernel/DMA transfers with zero user-space copies for sendfile().
Example

Edge-Triggered epoll (EPOLLET) TCP Echo Server in C

Write a C TCP echo server on port 8080 using Linux epoll in Edge-Triggered (EPOLLET) mode, showing proper non-blocking accept() and recv() drain loops until EAGAIN.

Configure Non-Blocking Helper and epoll Instance: Every socket added with EPOLLET must be non-blocking (O_NONBLOCK), and both accept() and recv() must drain inside while (1) loops until errno == EAGAIN || errno == EWOULDBLOCK.

Complete epoll Edge-Triggered Server (epoll_et_srv.c):

Edge-Triggered epoll TCP Echo ServerC
#include <arpa/inet.h>
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <sys/epoll.h>
#include <sys/socket.h>
#include <unistd.h>

#define MAX_EVENTS 64

static void set_nonblock(int fd) {
    int f = fcntl(fd, F_GETFL, 0);
    fcntl(fd, F_SETFL, f | O_NONBLOCK);
}

int main(void) {
    int srv = socket(AF_INET, SOCK_STREAM, 0);
    int opt = 1;
    setsockopt(srv, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
    set_nonblock(srv);

    struct sockaddr_in addr = {0};
    addr.sin_family = AF_INET;
    addr.sin_addr.s_addr = htonl(INADDR_ANY);
    addr.sin_port = htons(8080);
    bind(srv, (struct sockaddr *)&addr, sizeof(addr));
    listen(srv, SOMAXCONN);

    int epfd = epoll_create1(0);
    struct epoll_event ev = { .events = EPOLLIN | EPOLLET, .data.fd = srv };
    epoll_ctl(epfd, EPOLL_CTL_ADD, srv, &ev);

    struct epoll_event events[MAX_EVENTS];
    while (1) {
        int nready = epoll_wait(epfd, events, MAX_EVENTS, -1);
        for (int i = 0; i < nready; i++) {
            int fd = events[i].data.fd;
            if (fd == srv) {
                while (1) {
                    int cli = accept(srv, NULL, NULL);
                    if (cli < 0) {
                        if (errno == EAGAIN || errno == EWOULDBLOCK) break;
                        break;
                    }
                    set_nonblock(cli);
                    struct epoll_event cev = { .events = EPOLLIN | EPOLLET, .data.fd = cli };
                    epoll_ctl(epfd, EPOLL_CTL_ADD, cli, &cev);
                }
            } else {
                char buf[1024];
                while (1) {
                    ssize_t n = recv(fd, buf, sizeof(buf), 0);
                    if (n > 0) {
                        send(fd, buf, (size_t)n, 0);
                    } else if (n == 0 || (errno != EAGAIN && errno != EWOULDBLOCK)) {
                        epoll_ctl(epfd, EPOLL_CTL_DEL, fd, NULL);
                        close(fd);
                        break;
                    } else {
                        break; /* drained all available bytes (EAGAIN) */
                    }
                }
            }
        }
    }
    close(epfd);
    close(srv);
    return 0;
}
Answer:

The server sets O_NONBLOCK on srv and every accepted cli, registers them with EPOLLIN | EPOLLET, and drains both accept() and recv() in loops until EAGAIN so no pending connection or buffered bytes are left behind between edge notifications.

Example

Computing Handshake Latency: TCP + TLS 1.2 vs. TLS 1.3

A mobile client in Kathmandu connects to an API server in Singapore with a round-trip time (RTT) of 30 ms. Compare the total time from initiating connect() until the first HTTP/REST JSON response byte arrives at the client over:

  1. TLS 1.2 (1 RTT TCP handshake + 2 RTT TLS 1.2 handshake + 1 RTT HTTP request/response).
  2. TLS 1.3 (1 RTT TCP handshake + 1 RTT TLS 1.3 handshake + 1 RTT HTTP request/response).

Tracing Round-Trips for TLS 1.2:

  • 0..30 ms: TCP SYN -> SYN-ACK (1 RTT; client sends ACK together with ClientHello).
  • 30..90 ms: Standard TLS 1.2 takes 2 RTTs (ClientHello/ServerHello + ClientKeyExchange/ChangeCipherSpec).
  • 90..120 ms: Encrypted GET /api/data -> 200 OK JSON response (1 RTT).
  • Total latency = (1 + 2 + 1) * 30 ms = 120 ms.

Tracing Round-Trips for TLS 1.3:

  • 0..30 ms: TCP SYN -> SYN-ACK (1 RTT).
  • 30..60 ms: TLS 1.3 sends client KeyShare inside ClientHello, completing the entire TLS handshake in 1 RTT.
  • 60..90 ms: Encrypted GET /api/data (pipelined right behind Finished) -> 200 OK JSON response (1 RTT).
  • Total latency = (1 + 1 + 1) * 30 ms = 90 ms (saving 30 ms / 25% per cold HTTPS connection).
Answer:
  1. TLS 1.2: 4 RTTs = 120 ms to first JSON response.
  2. TLS 1.3: 3 RTTs = 90 ms to first JSON response (saves 1 full RTT = 30 ms).

Exercises

Practice Exercises — Advanced Network Programming

Question 1

Contrast iterative, multi-process (fork), multi-threaded, and pre-allocated thread-pool server architectures. Explain how SO_REUSEPORT improves multi-core accept scalability on Linux.

Comparing the Four Server Models:

  • Iterative: Handles one client sequentially; simple, but suffers severe head-of-line blocking.
  • Multi-process (fork): Strong fault isolation per client; higher process creation and IPC overhead.
  • Multi-threaded (pthread): Lower creation/switch overhead and direct memory sharing; unbounded threads risk stack OOM and thrashing under C10K load.
  • Thread-pool: Pre-creates a fixed worker pool pulling from a bounded queue; eliminates spawn latency and caps memory/CPU thrashing.

How SO_REUSEPORT Scales Multi-Core Accepts: Normally, multiple workers sharing a single listen_fd contend on a single kernel socket accept lock. With SO_REUSEPORT, each worker process/thread creates its own socket and binds the same IP and port; the kernel distributes incoming SYN packets evenly across the workers' separate accept queues using a 4-tuple hash, achieving lock-free multi-core scaling.

Answer:

Iterative servers process 1 client at a time; fork-per-client isolates faults at higher overhead; thread-per-client reduces overhead but risks OOM at scale; thread pools bound concurrency and eliminate creation cost. SO_REUSEPORT lets each worker bind its own listening socket on the same port so the kernel load-balances incoming connections without accept-lock contention.

Question 2

Explain why epoll() outperforms select() and poll() for 20,000 predominantly idle connections. Next, explain what happens if a server uses EPOLLET (Edge-Triggered) mode on a blocking socket or calls recv(fd, buf, 100, 0) only once when 500 bytes arrive.

Why epoll Scales to 20,000 Idle Sockets: select() and poll() are stateless: every call copies 20,000 fds from user space to kernel space and scans all 20,000 (O(N)), even if only 5 sockets received data. epoll registers fds once in kernel space (epoll_ctl), and epoll_wait() returns only the 5 ready events in O(k) time.

Bugs with EPOLLET + Blocking Socket or Single recv():

  • Single recv(fd, buf, 100, 0) on 500 bytes: EPOLLET triggers only once when the buffer goes from empty to non-empty. After reading 100 bytes, 400 bytes remain unread in the kernel buffer, yet epoll_wait() will not notify again unless new packets arrive—stalling the connection permanently!
  • Blocking socket inside drain loop: If you loop while (recv(...) > 0) to drain all 500 bytes on a blocking socket, the 6th recv() (when the buffer is empty) blocks the entire event-loop thread, freezing all 19,999 other clients! Hence EPOLLET always requires O_NONBLOCK + draining until EAGAIN.
Answer:

epoll tracks fds statefully in kernel and returns only ready fds in O(k) instead of O(N) per call. In EPOLLET, reading only 100 of 500 bytes leaves 400 bytes stranded without another wakeup, while draining a blocking socket freezes the entire event loop on the final empty read.

Question 3

An API gateway handles 8,000 requests/second with an average processing latency of 15 ms (0.015 s).

  1. Using Little's Law, calculate the minimum number of concurrent worker threads required.
  2. If each worker thread reserves a 512 KiB stack, how many MiB of stack memory does that pool reserve?
  3. Name two TCP/kernel optimizations (besides thread sizing) used in high-performance servers and state what bottleneck each removes.

Part 1 & 2 — Little's Law & Pool Stack Memory:

  • Concurrent workers L = lambda * W = 8,000 * 0.015 = 120 workers.
  • Total stack memory = 120 * 512 KiB = 61,440 KiB = 60 MiB.

Part 3 — Two High-Performance Server Optimizations:

  1. sendfile() (Zero-Copy I/O): Transfers static file data directly from kernel page cache to NIC DMA, eliminating user-space copies and mode switches.
  2. TCP_NODELAY: Disables Nagle's buffering delay so small RPC/API packets transmit immediately without waiting 40–200 ms for delayed ACKs.
Answer:
  1. 120 concurrent workers.
  2. 60 MiB of stack memory.
  3. sendfile() eliminates user-space memory copies for static files; TCP_NODELAY disables Nagle buffering to eliminate artificial latency on small messages.
Question 4

Trace the steps of a Remote Procedure Call (RPC) from client invocation to server return. Distinguish between At-Least-Once and At-Most-Once RPC semantics, and state when you would choose gRPC over a RESTful JSON API in a microservices architecture.

RPC Execution Steps:

  1. Client invokes the local client stub function.
  2. Client stub marshals (serializes) arguments into a platform-independent wire message (e.g., Protobuf) and passes it to the OS network stack.
  3. Network delivers the packet to the server's transport endpoint.
  4. Server stub (skeleton) unmarshals parameters and invokes the actual server procedure.
  5. Return value is marshaled back by the server stub, sent across the network, unmarshaled by the client stub, and returned to the caller.

Call Semantics & gRPC vs. REST:

  • At-Least-Once: Client retries lost replies until an answer arrives; the server may execute the procedure multiple times (safe only for idempotent operations).
  • At-Most-Once: Server tracks unique request IDs and caches responses so duplicate retries return the saved result without re-executing non-idempotent side effects.
  • Choosing gRPC over REST: Choose gRPC for low-latency, strongly typed internal microservice-to-microservice communication or bidirectional streaming (HTTP/2 + compact binary Protobuf), and REST/JSON for public web/browser APIs needing universal human readability and HTTP caching.
Answer:

RPC flows through client call -> client stub marshaling -> network -> server stub unmarshaling -> server execution -> marshaled reply. At-least-once retries safely on idempotent ops; at-most-once deduplicates using request IDs. Use gRPC for high-throughput internal microservices/streaming and REST/JSON for public HTTP clients.

Question 5

Explain how TLS provides confidentiality, integrity, and server authentication during HTTPS communication, and define Forward Secrecy. Next, suppose a client reconnects to a server over a 40 ms RTT link: compute the total latency to receive the first HTTP reply for a cold TLS 1.3 (1-RTT) connection versus a TLS 1.3 0-RTT resumed session (where early HTTP data is sent inside the first ClientHello flight right after TCP handshake).

TLS Security Pillars & Forward Secrecy:

  • Confidentiality: Symmetric AEAD encryption (AES-GCM / ChaCha20-Poly1305) using ephemeral session keys derived during the handshake.
  • Integrity: AEAD authentication tags verify that no ciphertext bits were altered or forged.
  • Authentication: Server presents an X.509 certificate signed by a trusted CA and signs the handshake transcript (CertificateVerify) with its private key.
  • Forward Secrecy: Using ephemeral Diffie-Hellman (ECDHE) ensures session keys are never derived from the server's long-term private key alone, so stealing that private key later cannot decrypt past captured sessions.

Latency Calculation (RTT = 40 ms):

  • Cold TLS 1.3 (1-RTT): 1 RTT (TCP) + 1 RTT (TLS 1.3) + 1 RTT (HTTP) = 3 * 40 ms = 120 ms.
  • Resumed TLS 1.3 (0-RTT early data): 1 RTT (TCP) + 0 RTT (TLS) + 1 RTT (ClientHello + HTTP early data -> reply) = 2 * 40 ms = 80 ms.
Answer:

TLS pairs X.509 CA signatures (authentication) with ECDHE key exchange (forward secrecy) and AEAD ciphers (confidentiality + integrity). On a 40 ms RTT link, cold TLS 1.3 takes 120 ms (3 RTTs) while 0-RTT resumption takes 80 ms (2 RTTs).

← Socket Programming and Network APIs (part 2: 4.4–4.6)System-Level I/O and Performance Optimization →