PadhaikoAll subjects

Network Application Development and Case Studies

Network and Systems Programming (ENCT 386)

Open in the Padhaiko app

7.1 Design of scalable client-server systems

Building a prototype socket server for 5 clients is straightforward; scaling that system reliably to 50,000+ concurrent clients requires eliminating architectural bottlenecks across OS limits, state management, and network routing.

Definition

Horizontal vs. Vertical Scaling in Network Systems

Essential Principles of Scalable Client-Server Design

  1. Stateless Application Nodes: Keep per-user session state out of local server RAM (store sessions in Redis or signed JWT tokens) so any backend server behind a load balancer can handle any client request seamlessly.
  2. OS & Kernel Resource Sizing:
    • Raise per-process file descriptor limits (ulimit -n / RLIMIT_NOFILE) above expected concurrent connections (each TCP client consumes 1 fd; proxies consume 2 fds: client + upstream).
    • Tune net.core.somaxconn and net.ipv4.ip_local_port_range so reverse proxies do not exhaust ephemeral ports (~28k–64k per (src_ip, dst_ip, dst_port) tuple) when talking to upstreams.
  3. Connection Pooling & Keep-Alive: Reuse persistent TCP/TLS connections between proxies, API gateways, and databases to avoid paying 3-way handshake and TLS setup costs on every request.
  4. Rate Limiting, Timeouts & Circuit Breakers: Enforce strict read/write timeouts (SO_RCVTIMEO or event-loop timers) so Slowloris attacks cannot pin sockets indefinitely, and apply backpressure or circuit breaking when downstream databases degrade.
  5. Graceful Shutdown (SIGTERM): Stop accepting new connections (close(listen_fd)), wait up to a bounded grace window for in-flight requests to finish, flush logs, and exit cleanly without dropping active responses.

7.2 Development of network applications: Chat servers, file transfer systems

1. Multi-Client Chat Server Architecture

A production chat server combines I/O multiplexing (poll() / epoll()), application framing, and non-blocking fan-out buffering:

Non-Blocking Event-Driven Chat Server PipelineText
  [Client A] --("JOIN Alice\nMSG hi\n")--> [ epoll_wait (EPOLLIN) ]
                                                    |
                                          Frame Parser (line \n)
                                                    |
                                          Broadcast Loop (for each client != A)
                                           /                    \
                    (fast client B: send ok)        (slow client C: EAGAIN!)
                                                             |
                                            Append to C's outbound ring buffer
                                            & arm EPOLLOUT on C's fd!

2. Reliable File Transfer System Architecture

A custom TCP file-transfer system must solve metadata framing, partial-I/O looping, and security:

  1. Structured Header: Send a fixed binary header in network byte order (uint32_t name_len, uint64_t file_size, plus optional SHA-256 checksum) followed by the filename and raw file bytes.
  2. Looping read_exact() & write_all(): Never assume one recv() returns the entire header or file chunk—loop until bytes_received == file_size. Use sendfile() on Linux servers to stream large files directly from disk cache to the socket with zero user-space copies.
  3. Path Traversal Prevention: Always sanitize client-supplied filenames (reject / or .. components, or extract strictly the basename) so an attacker cannot overwrite ../../etc/passwd or ~/.ssh/authorized_keys.

7.3 Debugging and monitoring tools (Netstat, ss, tcpdump, wireshark)

Diagnosing real production network failures requires inspecting live kernel socket tables and raw wire packets.

Essential Network Debugging and Monitoring Tools
ToolLayer / ScopeKey Commands & FiltersPrimary Diagnostic Use
ss (Socket Statistics)Kernel socket tables (via netlink tcp_diag)ss -tulnp (listening sockets + PIDs); ss -ant (all TCP states); ss -s (summary)Fast O(1) inspection of open ports, Recv-Q/Send-Q backlogs, CLOSE_WAIT leaks
netstat (Legacy net-tools)Reads /proc/net/tcp (slow at scale)netstat -tulnp, netstat -antpLegacy equivalent of ss; still seen on older Unix/embedded hosts
tcpdump (CLI Packet Sniffer)Link / IP / TCP / UDP wire framestcpdump -i eth0 -nn -s0 -w trace.pcap 'tcp port 8080'Headless server packet capture; verify handshakes, RST, retransmits, DNS
Wireshark (GUI Analyzer)Deep packet inspection (700+ dissectors)Display filters: tcp.flags.reset == 1, tcp.analysis.retransmission, http.response.code >= 500Visual timeline analysis, "Follow TCP Stream", RTT/throughput graphs

Reading ss / netstat States and tcpdump Flags


7.4 Case studies of real-world systems (Web servers, distributed apps)

Case Study 1: High-Concurrency Web Server (Nginx Architecture)

Traditional process-per-connection servers (such as classic Apache prefork) struggled under thousands of slow keep-alive HTTP clients because each idle connection tied up an entire OS process and megabytes of RAM. Nginx solved this C10K challenge using a master-worker, event-driven architecture:

Nginx Master-Worker Event-Driven ArchitectureText
                  [ Master Process (root): binds :80/:443, reloads config, reaps workers ]
                      /                     |                     \
         [ Worker 1 (Core 0) ]    [ Worker 2 (Core 1) ]    [ Worker N (Core N-1) ]
         - epoll_wait() loop      - epoll_wait() loop      - epoll_wait() loop
         - Non-blocking sockets   - Non-blocking sockets   - Non-blocking sockets
         - sendfile() static I/O  - sendfile() static I/O  - sendfile() static I/O

Case Study 2: Distributed Systems & In-Memory Stores (Redis + Distributed Chat/Microservices)

Modern distributed applications combine event-driven single-node speed with horizontal partitioning:

  1. Redis Single-Threaded Event Loop: Redis processes hundreds of thousands of commands/second on one main epoll event loop. Because all data structures (hash tables, sorted sets, streams) live in RAM and execute sequentially on one thread, zero mutexes or race conditions occur on core operations (with optional background I/O threads for socket read/write parsing and fork() Copy-on-Write for RDB persistence snapshots).
  2. Distributed Chat / Web Cluster: When one server cannot hold 1 million WebSocket/TCP users, multiple stateless edge gateway nodes maintain client sockets and route cross-node messages over a pub/sub broker (Redis Pub/Sub, NATS, or Kafka), partitioned by consistent hashing across room or user IDs so adding/removing a node moves minimal state.

Worked Examples

Example

Calculating File Transfer Time and Proxy Ephemeral Port Capacity

  1. A client transfers a 500 MiB firmware image over a 100 Mbps Ethernet link whose effective TCP goodput (after IP/TCP/Ethernet framing overhead) is 94 Mbps (94 * 10^6 bits/s). How many seconds does the file transfer take?
  2. A reverse proxy connects to a single upstream backend (10.0.0.10:8080) from one local IP, using Linux's default ephemeral port range 32,768–60,999 (28,232 ports). If the proxy closes short-lived upstream connections without connection pooling and TIME_WAIT lasts 60 seconds, what is the maximum sustainable connection rate (connections/second) before ephemeral port exhaustion (EADDRNOTAVAIL) occurs?

Part 1 — File Transfer Time:

  • File size in bits = 500 * 1024 * 1024 * 8 = 4,194,304,000 bits.
  • Effective goodput = 94 * 10^6 bits/s = 94,000,000 bits/s.
  • Transfer time = 4,194,304,000 / 94,000,000 = 44.62 seconds.

Part 2 — Ephemeral Port Exhaustion Rate:

  • Max connections/sec = 28,232 ports / 60 s = 470.53 connections/second.

(Enabling HTTP keep-alive / connection pooling eliminates this limit by reusing a small pool of persistent connections!)

Answer:
  1. Transfer time = 44.62 seconds.
  2. Maximum unpooled rate = 470.53 connections/second before EADDRNOTAVAIL port exhaustion.
Example

Length-Prefixed Binary File Transfer Protocol Helper in C

Write a C sender and receiver helper pair (send_file_header and recv_exact) that transmits a filename and 32-bit file size cleanly in network byte order, avoiding TCP framing bugs and partial recv() truncation.

Implement Exact-Byte Reader (recv_exact) and Header Sender (send_file_header):

TCP Length-Prefixed File Header FramingC
#include <arpa/inet.h>
#include <stdint.h>
#include <string.h>
#include <sys/socket.h>

int recv_exact(int fd, void *buf, size_t n) {
    size_t got = 0;
    char *p = (char *)buf;
    while (got < n) {
        ssize_t r = recv(fd, p + got, n - got, 0);
        if (r <= 0) return -1; /* EOF or error before all n bytes arrived */
        got += (size_t)r;
    }
    return 0;
}

int send_file_header(int fd, const char *clean_name, uint32_t file_size) {
    uint32_t name_len = (uint32_t)strlen(clean_name);
    uint32_t hdr[2] = { htonl(name_len), htonl(file_size) };

    if (send(fd, hdr, sizeof(hdr), 0) != sizeof(hdr)) return -1;
    if (send(fd, clean_name, name_len, 0) != (ssize_t)name_len) return -1;
    return 0;
}
Answer:

send_file_header() encodes name_len and file_size using htonl() before sending the filename, and recv_exact() loops recv() until all n bytes arrive, preventing TCP stream fragmentation bugs.

Example

Diagnosing Production Outages with ss and tcpdump Output

Inspect the following three diagnostic observations captured on a Linux web server and identify the exact root cause and fix for each:

  1. ss -s shows TCP: 18450 (estab 120, closed 18200, orphaned 0, timewait 40) and ss -ant lists 18,200 sockets in CLOSE-WAIT.
  2. ss -lnt on port 8080 shows Recv-Q = 128, Send-Q = 128, while tcpdump shows incoming client SYN packets receiving no SYN-ACK.
  3. tcpdump on a chat server shows [P.] packets followed by TCP Window Full and repeated [ACK] with win 0 (TCP ZeroWindow) from one client.

Incident 1 — 18,200 CLOSE-WAIT Sockets: Remote clients already closed their connections (FIN received and ACKed by Linux), yet the server process never called close(cli_fd) (a file-descriptor leak on an EOF or error branch). Fix: ensure every recv() <= 0 and error handler calls close(fd).

Incident 2 — Recv-Q == Send-Q == 128 on LISTEN Socket: Send-Q = 128 is the server's listen() backlog, and Recv-Q = 128 means the Accept queue is 100% full! Either the application's accept loop is blocked (e.g., doing slow synchronous DB work inside the acceptor thread) or listen(fd, 128) / net.core.somaxconn is too small for the burst rate.

Incident 3 — TCP ZeroWindow (win 0) in tcpdump: The remote client's TCP receive buffer is completely full because the client application stopped calling recv() (stalled/slow consumer). Fix: enforce non-blocking sockets with bounded per-client outbound queues and drop unresponsive clients after a timeout.

Answer:
  1. 18,200 CLOSE-WAIT: Server leaked file descriptors by failing to call close(fd) after peer FIN.
  2. Recv-Q == Send-Q == 128: Full listen() Accept queue; unblock the accept() loop and raise backlog / somaxconn.
  3. TCP ZeroWindow: Stalled client application stopped draining recv(); bound outbound queues and apply write timeouts.

Exercises

Practice Exercises — Network Apps & Case Studies

Question 1

Explain five architectural techniques required to scale a client-server system from a single test node to 50,000+ concurrent clients (covering statelessness, OS limits, connection pooling, timeouts/backpressure, and graceful shutdown).

Five Scalability Techniques:

  1. Stateless Backend Tier: Store sessions in Redis or signed JWTs instead of node RAM so a load balancer can scale servers horizontally.
  2. OS Limit Tuning: Increase RLIMIT_NOFILE (ulimit -n), net.core.somaxconn, and ephemeral port ranges (or use persistent pools).
  3. Connection Pooling: Reuse long-lived TCP/TLS connections to databases and upstream services to eliminate repeated handshakes.
  4. Strict Timeouts & Backpressure: Bound read/write timeouts and queue lengths to prevent Slowloris hangs and OOM crashes.
  5. Graceful Shutdown (SIGTERM): Close listening sockets first, finish draining active requests, and exit cleanly without dropping client replies.
Answer:

Scaling to 50k+ clients requires (1) stateless nodes behind a load balancer, (2) raising RLIMIT_NOFILE and somaxconn, (3) connection pooling, (4) strict timeouts and bounded queues for backpressure, and (5) graceful SIGTERM connection draining.

Question 2
  1. In an epoll-based TCP chat server, why can a single slow client freeze message delivery for all other clients if the server calls blocking send() during broadcast, and how do per-client outbound buffers + EPOLLOUT solve this?
  2. In a TCP file transfer server, name two security or correctness bugs that arise if the server trusts a raw client filename string and calls recv() once per file block without looping.

Part 1 — Preventing Slow-Client Stalls in Chat Broadcasts: If Client X's TCP receive window drops to zero, its kernel send buffer on the server fills up; a blocking send() puts the single epoll event-loop thread to sleep, halting broadcasts to every other user! Using O_NONBLOCK sockets, any unsent remainder after EAGAIN is appended to Client X's private outbound buffer and EPOLLOUT is enabled on Client X's fd so the loop keeps serving everyone else and flushes X's buffer whenever X's socket becomes writable (disconnecting X if the buffer exceeds a max cap).

Part 2 — File Transfer Correctness and Security Bugs:

  1. Path Traversal Vulnerability: Without stripping directory components (or rejecting .. / /), a client sending ../../.ssh/authorized_keys can overwrite arbitrary server files.
  2. Short-Read Corruption: Calling recv(fd, buf, 4096, 0) once may return fewer than 4096 bytes due to TCP stream segmentation; without a recv_exact() / byte-counter loop, the saved file is truncated or misaligned.
Answer:
  1. Blocking send() on a slow client stalls the entire event loop; non-blocking sockets with per-client outbound queues and EPOLLOUT isolate slow consumers.
  2. Raw filenames allow directory traversal (../) attacks, and unlooped recv() causes short-read file truncation.
Question 3

A chat room has 1,000 active clients. On average, 5 messages per second are posted to the room, and each framed message (including sender tag and newline) is 200 bytes. Assume every message is broadcast to all 1,000 connected clients.

  1. Calculate the server's total outbound payload rate in bytes/second and Mbps (1 Mbps = 10^6 bits/s).
  2. Explain why chat servers scale outbound broadcasting horizontally using pub/sub clustering once a single machine hits CPU or NIC bandwidth limits.

Part 1 — Outbound Broadcast Fan-Out Bandwidth:

  • Messages sent out per second = 5 msg/s * 1,000 clients = 5,000 outbound deliveries/s.
  • Outbound payload rate = 5,000 * 200 bytes = 1,000,000 bytes/s (1 MB/s).
  • In megabits per second = (1,000,000 * 8 bits/s) / 1,000,000 = 8.0 Mbps (note O(N^2) growth if all N users also post proportionally!).

Part 2 — Horizontal Scaling via Pub/Sub: Because broadcast fan-out grows quadratically when active senders and listeners both increase, a single NIC or CPU core eventually saturates. Partitioning client sockets across K gateway nodes connected via a pub/sub broker (Redis Pub/Sub / NATS / Kafka) allows each gateway to fan out only to its local fraction (N / K) of users.

Answer:
  1. Outbound payload rate = 1,000,000 bytes/s = 8.0 Mbps.
  2. Quadratic broadcast fan-out saturates single-host NIC/CPU, so horizontal gateways connected via pub/sub split the fan-out work across K nodes.
Question 4

Compare ss and netstat, and write:

  1. A tcpdump command to capture full packets (without DNS resolution) on interface eth0 for TCP port 8443 into debug.pcap.
  2. A Wireshark display filter to isolate only TCP RST packets or retransmissions on port 8443.
  3. A one-sentence explanation distinguishing SYN-RECV, CLOSE-WAIT, and TIME-WAIT in ss -ant.

ss vs. netstat & Capture/Filter Commands:

  • ss reads kernel netlink tcp_diag tables directly (orders of magnitude faster than netstat parsing /proc/net/tcp on servers with tens of thousands of sockets).
  • tcpdump command: tcpdump -i eth0 -nn -s0 -w debug.pcap 'tcp port 8443' (-nn skips DNS/port name lookups; -s0 captures full packet payload).
  • Wireshark display filter: tcp.port == 8443 && (tcp.flags.reset == 1 || tcp.analysis.retransmission)

Distinguishing the Three TCP States:

  • SYN-RECV: Server received initial SYN, sent SYN-ACK, and is waiting for the client's final ACK.
  • CLOSE-WAIT: Peer already closed (FIN received), waiting for the local application to call close().
  • TIME-WAIT: Local side performed the active close and waits 2 * MSL so stray delayed packets expire before the 5-tuple is reused.
Answer:

ss queries kernel netlink directly and outperforms netstat at scale. Capture: tcpdump -i eth0 -nn -s0 -w debug.pcap 'tcp port 8443'; Wireshark filter: tcp.port == 8443 && (tcp.flags.reset == 1 || tcp.analysis.retransmission). SYN-RECV awaits handshake ACK, CLOSE-WAIT awaits local close(), and TIME-WAIT lingers 2 * MSL after active close.

Question 5

Compare Nginx's master-worker epoll architecture with a blocking thread-per-connection server. Suppose a server handles 40,000 concurrent keep-alive connections:

  1. Estimate the virtual stack memory required by a blocking thread-per-connection server using Linux's default 8 MiB thread stack.
  2. Estimate the connection-state memory required by an event-driven epoll worker storing 1 KiB of state per connection (in MiB), and explain why Nginx runs 1 worker process per CPU core rather than 1 worker total.

Part 1 & 2 — Memory Footprint Comparison (40,000 Connections):

  • Blocking Thread-per-Connection (8 MiB stack): 40,000 * 8 MiB = 320,000 MiB (312.5 GiB of virtual stack space + massive scheduler overhead).
  • Event-Driven epoll (1 KiB state/conn): 40,000 * 1 KiB = 40,000 KiB = 39.06 MiB (~8,192x smaller!).

Why Nginx Runs 1 Worker Process per CPU Core: Running a single worker process would utilize only 1 CPU core for HTTP parsing, TLS encryption, and compression. Running 1 single-threaded epoll worker per core saturates all CPU cores in parallel with zero lock contention or thread context-switching overhead, while isolating each core's memory/cache.

Answer:
  1. Thread-per-connection (8 MiB stacks) reserves 320,000 MiB (312.5 GiB), whereas epoll (1 KiB/conn) uses only 39.06 MiB.
  2. Nginx spawns 1 event-loop worker per CPU core to utilize all cores in parallel without inter-thread locking or context-switch thrashing.
← System-Level I/O and Performance Optimization