Network Application Development and Case Studies
Open in the Padhaiko app7.1 Design of scalable client-server systems
Building a prototype socket server for 5 clients is straightforward; scaling that system reliably to 50,000+ concurrent clients requires eliminating architectural bottlenecks across OS limits, state management, and network routing.
Horizontal vs. Vertical Scaling in Network Systems
- Vertical Scaling (Scale-Up): Increasing CPU cores, RAM, or NIC bandwidth on a single server host. Ultimately bounded by hardware limits and single-point-of-failure risk.
- Horizontal Scaling (Scale-Out): Running multiple stateless server instances behind a Load Balancer (L4 TCP/UDP or L7 HTTP/gRPC) so incoming traffic distributes across machines with high availability.
Essential Principles of Scalable Client-Server Design
- Stateless Application Nodes: Keep per-user session state out of local server RAM (store sessions in Redis or signed JWT tokens) so any backend server behind a load balancer can handle any client request seamlessly.
- OS & Kernel Resource Sizing:
- Raise per-process file descriptor limits (
ulimit -n/RLIMIT_NOFILE) above expected concurrent connections (each TCP client consumes 1 fd; proxies consume 2 fds: client + upstream). - Tune
net.core.somaxconnandnet.ipv4.ip_local_port_rangeso reverse proxies do not exhaust ephemeral ports (~28k–64kper(src_ip, dst_ip, dst_port)tuple) when talking to upstreams.
- Raise per-process file descriptor limits (
- Connection Pooling & Keep-Alive: Reuse persistent TCP/TLS connections between proxies, API gateways, and databases to avoid paying 3-way handshake and TLS setup costs on every request.
- Rate Limiting, Timeouts & Circuit Breakers: Enforce strict read/write timeouts (
SO_RCVTIMEOor event-loop timers) so Slowloris attacks cannot pin sockets indefinitely, and apply backpressure or circuit breaking when downstream databases degrade. - Graceful Shutdown (
SIGTERM): Stop accepting new connections (close(listen_fd)), wait up to a bounded grace window for in-flight requests to finish, flush logs, and exit cleanly without dropping active responses.
7.2 Development of network applications: Chat servers, file transfer systems
1. Multi-Client Chat Server Architecture
A production chat server combines I/O multiplexing (poll() / epoll()), application framing, and non-blocking fan-out buffering:
[Client A] --("JOIN Alice\nMSG hi\n")--> [ epoll_wait (EPOLLIN) ]
|
Frame Parser (line \n)
|
Broadcast Loop (for each client != A)
/ \
(fast client B: send ok) (slow client C: EAGAIN!)
|
Append to C's outbound ring buffer
& arm EPOLLOUT on C's fd!- Why Per-Client Outbound Queues Matter: If Client C has a slow 3G link and its kernel send buffer fills up, calling blocking
send()during a broadcast would freeze the entire chat room for Clients A, B, and D! Instead, every client struct holds a small outbound buffer; if non-blockingsend()returnsEAGAIN, remaining bytes are queued andEPOLLOUTis enabled on Client C's socket (or slow clients exceeding a max queue cap are disconnected). - Framing: Because TCP is a byte stream, accumulate incoming bytes per client until
'\n'(text protocol) or a 4-byte length header arrives before dispatching a complete chat command.
2. Reliable File Transfer System Architecture
A custom TCP file-transfer system must solve metadata framing, partial-I/O looping, and security:
- Structured Header: Send a fixed binary header in network byte order (
uint32_t name_len,uint64_t file_size, plus optional SHA-256 checksum) followed by the filename and raw file bytes. - Looping
read_exact()&write_all(): Never assume onerecv()returns the entire header or file chunk—loop untilbytes_received == file_size. Usesendfile()on Linux servers to stream large files directly from disk cache to the socket with zero user-space copies. - Path Traversal Prevention: Always sanitize client-supplied filenames (reject
/or..components, or extract strictly the basename) so an attacker cannot overwrite../../etc/passwdor~/.ssh/authorized_keys.
7.3 Debugging and monitoring tools (Netstat, ss, tcpdump, wireshark)
Diagnosing real production network failures requires inspecting live kernel socket tables and raw wire packets.
| Tool | Layer / Scope | Key Commands & Filters | Primary Diagnostic Use |
|---|---|---|---|
ss (Socket Statistics) | Kernel socket tables (via netlink tcp_diag) | ss -tulnp (listening sockets + PIDs); ss -ant (all TCP states); ss -s (summary) | Fast O(1) inspection of open ports, Recv-Q/Send-Q backlogs, CLOSE_WAIT leaks |
netstat (Legacy net-tools) | Reads /proc/net/tcp (slow at scale) | netstat -tulnp, netstat -antp | Legacy equivalent of ss; still seen on older Unix/embedded hosts |
tcpdump (CLI Packet Sniffer) | Link / IP / TCP / UDP wire frames | tcpdump -i eth0 -nn -s0 -w trace.pcap 'tcp port 8080' | Headless server packet capture; verify handshakes, RST, retransmits, DNS |
| Wireshark (GUI Analyzer) | Deep packet inspection (700+ dissectors) | Display filters: tcp.flags.reset == 1, tcp.analysis.retransmission, http.response.code >= 500 | Visual timeline analysis, "Follow TCP Stream", RTT/throughput graphs |
Reading ss / netstat States and tcpdump Flags
- Key
ssTCP States:LISTEN: Server socket awaitingSYN(Recv-Q= current pending Accept queue length;Send-Q= configuredlisten()backlog!).SYN-RECV: Half-open handshakes waiting for final clientACK(spikes during SYN floods or packet loss).ESTABLISHED: Active connected sockets.TIME-WAIT: Normal post-close linger (2 * MSL) on the side that initiated active close.CLOSE-WAIT: Remote peer sentFIN, waiting for local process to callclose()(growing count = fd leak in your code).
- Key
tcpdumpFlag Letters:[S]=SYN,[S.]=SYN-ACK,[.]=ACK,[P.]=PSH-ACK(data push),[F.]=FIN-ACK,[R]=RST(abrupt reset). Always pass-nnto disable slow reverse-DNS lookups during live troubleshooting.
7.4 Case studies of real-world systems (Web servers, distributed apps)
Case Study 1: High-Concurrency Web Server (Nginx Architecture)
Traditional process-per-connection servers (such as classic Apache prefork) struggled under thousands of slow keep-alive HTTP clients because each idle connection tied up an entire OS process and megabytes of RAM. Nginx solved this C10K challenge using a master-worker, event-driven architecture:
[ Master Process (root): binds :80/:443, reloads config, reaps workers ]
/ | \
[ Worker 1 (Core 0) ] [ Worker 2 (Core 1) ] [ Worker N (Core N-1) ]
- epoll_wait() loop - epoll_wait() loop - epoll_wait() loop
- Non-blocking sockets - Non-blocking sockets - Non-blocking sockets
- sendfile() static I/O - sendfile() static I/O - sendfile() static I/O- 1 Worker Process per CPU Core: Minimizes context switches and keeps CPU L1/L2 caches hot;
SO_REUSEPORTdistributes incoming connections cleanly across workers. - Non-Blocking Reactor +
epoll: One worker tracks tens of thousands of simultaneous keep-alive connections using just a few hundred bytes of connection state each instead of multi-megabyte stacks. - Zero-Copy & Upstream Proxying: Static files stream via
sendfile(), while dynamic PHP/Python/Node backends sit behind pooled HTTP/FastCGI/gRPC upstream connections.
Case Study 2: Distributed Systems & In-Memory Stores (Redis + Distributed Chat/Microservices)
Modern distributed applications combine event-driven single-node speed with horizontal partitioning:
- Redis Single-Threaded Event Loop: Redis processes hundreds of thousands of commands/second on one main
epollevent loop. Because all data structures (hash tables, sorted sets, streams) live in RAM and execute sequentially on one thread, zero mutexes or race conditions occur on core operations (with optional background I/O threads for socket read/write parsing andfork()Copy-on-Write for RDB persistence snapshots). - Distributed Chat / Web Cluster: When one server cannot hold 1 million WebSocket/TCP users, multiple stateless edge gateway nodes maintain client sockets and route cross-node messages over a pub/sub broker (Redis Pub/Sub, NATS, or Kafka), partitioned by consistent hashing across room or user IDs so adding/removing a node moves minimal state.
Worked Examples
Calculating File Transfer Time and Proxy Ephemeral Port Capacity
- A client transfers a 500 MiB firmware image over a 100 Mbps Ethernet link whose effective TCP goodput (after IP/TCP/Ethernet framing overhead) is 94 Mbps (
94 * 10^6 bits/s). How many seconds does the file transfer take? - A reverse proxy connects to a single upstream backend (
10.0.0.10:8080) from one local IP, using Linux's default ephemeral port range32,768–60,999(28,232 ports). If the proxy closes short-lived upstream connections without connection pooling andTIME_WAITlasts 60 seconds, what is the maximum sustainable connection rate (connections/second) before ephemeral port exhaustion (EADDRNOTAVAIL) occurs?
Part 1 — File Transfer Time:
- File size in bits =
500 * 1024 * 1024 * 8 = 4,194,304,000 bits. - Effective goodput =
94 * 10^6 bits/s = 94,000,000 bits/s. - Transfer time =
4,194,304,000 / 94,000,000 = 44.62 seconds.
Part 2 — Ephemeral Port Exhaustion Rate:
- Max connections/sec =
28,232 ports / 60 s = 470.53 connections/second.
(Enabling HTTP keep-alive / connection pooling eliminates this limit by reusing a small pool of persistent connections!)
- Transfer time = 44.62 seconds.
- Maximum unpooled rate = 470.53 connections/second before
EADDRNOTAVAILport exhaustion.
Length-Prefixed Binary File Transfer Protocol Helper in C
Write a C sender and receiver helper pair (send_file_header and recv_exact) that transmits a filename and 32-bit file size cleanly in network byte order, avoiding TCP framing bugs and partial recv() truncation.
Implement Exact-Byte Reader (recv_exact) and Header Sender (send_file_header):
#include <arpa/inet.h>
#include <stdint.h>
#include <string.h>
#include <sys/socket.h>
int recv_exact(int fd, void *buf, size_t n) {
size_t got = 0;
char *p = (char *)buf;
while (got < n) {
ssize_t r = recv(fd, p + got, n - got, 0);
if (r <= 0) return -1; /* EOF or error before all n bytes arrived */
got += (size_t)r;
}
return 0;
}
int send_file_header(int fd, const char *clean_name, uint32_t file_size) {
uint32_t name_len = (uint32_t)strlen(clean_name);
uint32_t hdr[2] = { htonl(name_len), htonl(file_size) };
if (send(fd, hdr, sizeof(hdr), 0) != sizeof(hdr)) return -1;
if (send(fd, clean_name, name_len, 0) != (ssize_t)name_len) return -1;
return 0;
}send_file_header() encodes name_len and file_size using htonl() before sending the filename, and recv_exact() loops recv() until all n bytes arrive, preventing TCP stream fragmentation bugs.
Diagnosing Production Outages with ss and tcpdump Output
Inspect the following three diagnostic observations captured on a Linux web server and identify the exact root cause and fix for each:
ss -sshowsTCP: 18450 (estab 120, closed 18200, orphaned 0, timewait 40)andss -antlists 18,200 sockets inCLOSE-WAIT.ss -lnton port8080showsRecv-Q = 128,Send-Q = 128, whiletcpdumpshows incoming clientSYNpackets receiving noSYN-ACK.tcpdumpon a chat server shows[P.]packets followed byTCP Window Fulland repeated[ACK]withwin 0(TCP ZeroWindow) from one client.
Incident 1 — 18,200 CLOSE-WAIT Sockets: Remote clients already closed their connections (FIN received and ACKed by Linux), yet the server process never called close(cli_fd) (a file-descriptor leak on an EOF or error branch). Fix: ensure every recv() <= 0 and error handler calls close(fd).
Incident 2 — Recv-Q == Send-Q == 128 on LISTEN Socket: Send-Q = 128 is the server's listen() backlog, and Recv-Q = 128 means the Accept queue is 100% full! Either the application's accept loop is blocked (e.g., doing slow synchronous DB work inside the acceptor thread) or listen(fd, 128) / net.core.somaxconn is too small for the burst rate.
Incident 3 — TCP ZeroWindow (win 0) in tcpdump: The remote client's TCP receive buffer is completely full because the client application stopped calling recv() (stalled/slow consumer). Fix: enforce non-blocking sockets with bounded per-client outbound queues and drop unresponsive clients after a timeout.
- 18,200
CLOSE-WAIT: Server leaked file descriptors by failing to callclose(fd)after peerFIN. Recv-Q == Send-Q == 128: Fulllisten()Accept queue; unblock theaccept()loop and raisebacklog/somaxconn.TCP ZeroWindow: Stalled client application stopped drainingrecv(); bound outbound queues and apply write timeouts.
Exercises
Practice Exercises — Network Apps & Case Studies
Explain five architectural techniques required to scale a client-server system from a single test node to 50,000+ concurrent clients (covering statelessness, OS limits, connection pooling, timeouts/backpressure, and graceful shutdown).
Five Scalability Techniques:
- Stateless Backend Tier: Store sessions in Redis or signed JWTs instead of node RAM so a load balancer can scale servers horizontally.
- OS Limit Tuning: Increase
RLIMIT_NOFILE(ulimit -n),net.core.somaxconn, and ephemeral port ranges (or use persistent pools). - Connection Pooling: Reuse long-lived TCP/TLS connections to databases and upstream services to eliminate repeated handshakes.
- Strict Timeouts & Backpressure: Bound read/write timeouts and queue lengths to prevent Slowloris hangs and OOM crashes.
- Graceful Shutdown (
SIGTERM): Close listening sockets first, finish draining active requests, and exit cleanly without dropping client replies.
Scaling to 50k+ clients requires (1) stateless nodes behind a load balancer, (2) raising RLIMIT_NOFILE and somaxconn, (3) connection pooling, (4) strict timeouts and bounded queues for backpressure, and (5) graceful SIGTERM connection draining.
- In an
epoll-based TCP chat server, why can a single slow client freeze message delivery for all other clients if the server calls blockingsend()during broadcast, and how do per-client outbound buffers +EPOLLOUTsolve this? - In a TCP file transfer server, name two security or correctness bugs that arise if the server trusts a raw client filename string and calls
recv()once per file block without looping.
Part 1 — Preventing Slow-Client Stalls in Chat Broadcasts: If Client X's TCP receive window drops to zero, its kernel send buffer on the server fills up; a blocking send() puts the single epoll event-loop thread to sleep, halting broadcasts to every other user! Using O_NONBLOCK sockets, any unsent remainder after EAGAIN is appended to Client X's private outbound buffer and EPOLLOUT is enabled on Client X's fd so the loop keeps serving everyone else and flushes X's buffer whenever X's socket becomes writable (disconnecting X if the buffer exceeds a max cap).
Part 2 — File Transfer Correctness and Security Bugs:
- Path Traversal Vulnerability: Without stripping directory components (or rejecting
..//), a client sending../../.ssh/authorized_keyscan overwrite arbitrary server files. - Short-Read Corruption: Calling
recv(fd, buf, 4096, 0)once may return fewer than 4096 bytes due to TCP stream segmentation; without arecv_exact()/ byte-counter loop, the saved file is truncated or misaligned.
- Blocking
send()on a slow client stalls the entire event loop; non-blocking sockets with per-client outbound queues andEPOLLOUTisolate slow consumers. - Raw filenames allow directory traversal (
../) attacks, and unloopedrecv()causes short-read file truncation.
A chat room has 1,000 active clients. On average, 5 messages per second are posted to the room, and each framed message (including sender tag and newline) is 200 bytes. Assume every message is broadcast to all 1,000 connected clients.
- Calculate the server's total outbound payload rate in bytes/second and Mbps (
1 Mbps = 10^6 bits/s). - Explain why chat servers scale outbound broadcasting horizontally using pub/sub clustering once a single machine hits CPU or NIC bandwidth limits.
Part 1 — Outbound Broadcast Fan-Out Bandwidth:
- Messages sent out per second =
5 msg/s * 1,000 clients = 5,000 outbound deliveries/s. - Outbound payload rate =
5,000 * 200 bytes = 1,000,000 bytes/s(1 MB/s). - In megabits per second =
(1,000,000 * 8 bits/s) / 1,000,000 = 8.0 Mbps(note O(N^2) growth if all N users also post proportionally!).
Part 2 — Horizontal Scaling via Pub/Sub: Because broadcast fan-out grows quadratically when active senders and listeners both increase, a single NIC or CPU core eventually saturates. Partitioning client sockets across K gateway nodes connected via a pub/sub broker (Redis Pub/Sub / NATS / Kafka) allows each gateway to fan out only to its local fraction (N / K) of users.
- Outbound payload rate = 1,000,000 bytes/s = 8.0 Mbps.
- Quadratic broadcast fan-out saturates single-host NIC/CPU, so horizontal gateways connected via pub/sub split the fan-out work across
Knodes.
Compare ss and netstat, and write:
- A
tcpdumpcommand to capture full packets (without DNS resolution) on interfaceeth0for TCP port8443intodebug.pcap. - A Wireshark display filter to isolate only TCP
RSTpackets or retransmissions on port8443. - A one-sentence explanation distinguishing
SYN-RECV,CLOSE-WAIT, andTIME-WAITinss -ant.
ss vs. netstat & Capture/Filter Commands:
ssreads kernel netlinktcp_diagtables directly (orders of magnitude faster thannetstatparsing/proc/net/tcpon servers with tens of thousands of sockets).tcpdumpcommand:tcpdump -i eth0 -nn -s0 -w debug.pcap 'tcp port 8443'(-nnskips DNS/port name lookups;-s0captures full packet payload).- Wireshark display filter:
tcp.port == 8443 && (tcp.flags.reset == 1 || tcp.analysis.retransmission)
Distinguishing the Three TCP States:
SYN-RECV: Server received initialSYN, sentSYN-ACK, and is waiting for the client's finalACK.CLOSE-WAIT: Peer already closed (FINreceived), waiting for the local application to callclose().TIME-WAIT: Local side performed the active close and waits2 * MSLso stray delayed packets expire before the 5-tuple is reused.
ss queries kernel netlink directly and outperforms netstat at scale. Capture: tcpdump -i eth0 -nn -s0 -w debug.pcap 'tcp port 8443'; Wireshark filter: tcp.port == 8443 && (tcp.flags.reset == 1 || tcp.analysis.retransmission). SYN-RECV awaits handshake ACK, CLOSE-WAIT awaits local close(), and TIME-WAIT lingers 2 * MSL after active close.
Compare Nginx's master-worker epoll architecture with a blocking thread-per-connection server. Suppose a server handles 40,000 concurrent keep-alive connections:
- Estimate the virtual stack memory required by a blocking thread-per-connection server using Linux's default 8 MiB thread stack.
- Estimate the connection-state memory required by an event-driven
epollworker storing 1 KiB of state per connection (in MiB), and explain why Nginx runs 1 worker process per CPU core rather than 1 worker total.
Part 1 & 2 — Memory Footprint Comparison (40,000 Connections):
- Blocking Thread-per-Connection (8 MiB stack):
40,000 * 8 MiB = 320,000 MiB(312.5 GiBof virtual stack space + massive scheduler overhead). - Event-Driven
epoll(1 KiB state/conn):40,000 * 1 KiB = 40,000 KiB = 39.06 MiB(~8,192xsmaller!).
Why Nginx Runs 1 Worker Process per CPU Core: Running a single worker process would utilize only 1 CPU core for HTTP parsing, TLS encryption, and compression. Running 1 single-threaded epoll worker per core saturates all CPU cores in parallel with zero lock contention or thread context-switching overhead, while isolating each core's memory/cache.
- Thread-per-connection (8 MiB stacks) reserves 320,000 MiB (312.5 GiB), whereas
epoll(1 KiB/conn) uses only 39.06 MiB. - Nginx spawns 1 event-loop worker per CPU core to utilize all cores in parallel without inter-thread locking or context-switch thrashing.