Socket Programming and Network APIs (part 2: 4.4–4.6)
Open in the Padhaiko app4.4 Blocking versus non-blocking sockets
By default, every POSIX socket created with socket() or returned by accept() operates in blocking mode. Understanding when a socket blocks—and how non-blocking mode alters every system call—is fundamental to building responsive network servers.
Blocking vs. Non-Blocking Socket Operations
- Blocking Socket: If a requested operation cannot complete immediately (e.g.,
recv()when the receive buffer is empty,send()when the kernel send buffer is full,accept()when no client is queued, orconnect()awaiting a 3-way handshake), the kernel puts the calling thread to sleep until the event completes or an error/signal occurs. - Non-Blocking Socket (
O_NONBLOCK): The operation never puts the thread to sleep. If it can complete immediately (even partially), it returns the byte count or new fd; otherwise it returns-1immediately and setserrnotoEAGAINorEWOULDBLOCK(orEINPROGRESSforconnect()).
Enabling Non-Blocking Mode in C
Two standard POSIX mechanisms enable O_NONBLOCK:
fcntl()on an existing socket:
int flags = fcntl(sockfd, F_GETFL, 0);
if (flags >= 0) {
fcntl(sockfd, F_SETFL, flags | O_NONBLOCK);
}- Atomic creation flags on Linux:
socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0)andaccept4(srv, addr, len, SOCK_NONBLOCK)avoid an extrafcntl()system call.
| System Call | Blocking Mode Behavior | Non-Blocking Mode (O_NONBLOCK) Behavior |
|---|---|---|
accept() | Sleeps until a client finishes the 3-way handshake | Returns -1 (EAGAIN / EWOULDBLOCK) if Accept queue is empty |
recv() / read() | Sleeps until at least 1 byte arrives or peer sends FIN (0) | Returns available bytes immediately, 0 on EOF, or -1 (EAGAIN) if empty |
send() / write() | Sleeps until all bytes fit in kernel send buffer (or error) | Copies as many bytes as fit; returns -1 (EAGAIN) if send buffer is full |
connect() | Sleeps until 3-way handshake finishes or fails | Initiates handshake and returns -1 with errno == EINPROGRESS |
4.5 I/O multiplexing (Select, poll basics)
I/O multiplexing enables a single thread to monitor dozens or thousands of file descriptors simultaneously and block in one kernel call until one or more of them are ready for reading, writing, or exception handling.
1. The select() System Call (<sys/select.h>)
int select(int nfds, fd_set *readfds, fd_set *writefds,
fd_set *exceptfds, struct timeval *timeout);
FD_ZERO(&set); /* clear all bits in set */
FD_SET(fd, &set); /* add fd to set */
FD_CLR(fd, &set); /* remove fd from set */
FD_ISSET(fd, &set); /* test whether fd is ready after select() returns */Critical select() Rules and Limitations
nfdsMUST bemax_fd + 1: Passing the count of sockets instead ofmax_fd + 1is a classic bug that causesselect()to silently ignore higher-numbered file descriptors!- In-Place Bitmask Mutation:
select()overwritesreadfds,writefds, andexceptfdsto report which fds are ready. You must copy a persistentmaster_setinto a temporaryworking_setbefore everyselect()call. FD_SETSIZEHard Limit: Standardfd_setis a fixed-size bit array (typically 1024 bits = 128 bytes on glibc Linux). Monitoring any fd>= 1024causes buffer overflow / undefined behavior.- Linear Scanning (
O(max_fd)): Every call copies the bitmasks user->kernel->user and forces user code to loop0 .. max_fdcheckingFD_ISSET().
2. The poll() System Call (<poll.h>)
poll() fixes two major design flaws of select() (FD_SETSIZE cap and bitmask destruction) by passing an explicit array of struct pollfd:
struct pollfd {
int fd; /* file descriptor to watch (ignore entry if < 0) */
short events; /* requested events: POLLIN, POLLOUT */
short revents; /* returned events filled by kernel: POLLIN, POLLOUT, POLLERR, POLLHUP */
};
int poll(struct pollfd fds[], nfds_t nfds, int timeout_ms);eventsvs.revents: Separating input (events) from output (revents) means the caller does not need to rebuildfds[]before every loop iteration.timeout_ms:-1blocks forever,0returns immediately (non-blocking check),> 0waits at mosttimeout_msmilliseconds.- No 1024 FD Limit:
fds[]can hold as many descriptors as the process'sRLIMIT_NOFILEpermits.
| Aspect | select() | poll() |
|---|---|---|
| Data Representation | Three fixed bitmasks (fd_set) | Caller-allocated array of struct pollfd |
| Max FD Number | Hard-capped at FD_SETSIZE (1024) on Linux | No fixed limit (bounded only by RLIMIT_NOFILE) |
| Re-initialization | Must copy master_set before every call (mutated in place) | Not needed (events preserved; kernel writes only revents) |
| Timeout Resolution | struct timeval (microseconds; select may modify it) | int timeout_ms (milliseconds; unmodified) |
| Scan Complexity | O(max_fd) even if only 2 high fds are open | O(N) where N is number of entries in fds[] |
4.6 Error handling and debugging techniques
Network programs fail constantly due to unreachable hosts, dropped links, interrupted syscalls, and half-closed peers. Robust error handling distinguishes recoverable transient conditions from fatal bugs.
Classifying Socket Errors
Condition / errno | Cause | Proper Server / Client Action |
|---|---|---|
recv() == 0 | Remote peer closed connection cleanly (FIN) | Not an error—close local socket and clean up client state |
EINTR | Blocking syscall interrupted by a caught signal | Retry the syscall (or check shutdown flag) |
EAGAIN / EWOULDBLOCK | Non-blocking socket has no data / buffer full | Stop reading/writing for now; wait via select/poll/epoll |
EPIPE (after SIGPIPE ignored) | Writing to a socket whose peer already closed | Close local fd and remove client cleanly |
ECONNRESET | Peer crashed or sent TCP RST | Log, close local fd, and release resources |
EADDRINUSE | Port already bound (or lingering in TIME_WAIT) | Enable SO_REUSEADDR or check conflicting process |
EINPROGRESS | Non-blocking connect() handshake in progress | Wait for writability (POLLOUT) in poll(), then inspect SO_ERROR |
Diagnostic APIs and CLI Debugging Tools
- API Diagnostics:
perror("msg")/strerror(errno)for standard syscalls;gai_strerror(rc)forgetaddrinfo()DNS failures (sincegetaddrinfo()returns its own error codes rather than settingerrno).getsockopt(fd, SOL_SOCKET, SO_ERROR, &err, &len)retrieves and clears pending asynchronous socket errors (essential after non-blockingconnect()).SO_RCVTIMEOandSO_SNDTIMEOattach kernel-enforced timeouts to blockingrecv()/send()calls.
- Command-Line Network Debugging Tools:
ss -tulpn/netstat -antp: Lists listening ports, owning PIDs, and TCP states. A growing pile of sockets inCLOSE_WAITindicates your server forgot to callclose()after receiving peerFIN; manyTIME_WAITsockets are normal on the side that closed first.strace -e trace=network ./server: Traces everysocket,bind,connect,send,recv, and exacterrnoreturned without recompiling.lsof -p <pid>: Lists all open file descriptors and sockets held by a running process to spot fd leaks.tcpdump/ Wireshark: Captures live wire packets to inspect 3-way handshakes, retransmissions,RSTflags, and malformed framing headers.
Worked Examples
Comparing select() and poll() Scan Overhead and Memory Usage
A process opens a few files and then holds 250 active sockets whose highest file descriptor number is max_fd = 900.
- How many bytes are occupied by three
fd_setstructures (readfds,writefds,exceptfds) passed toselect()whenFD_SETSIZE = 1024? What value must be passed asnfdstoselect(), and how many fd positions must afor (fd = 0; fd < nfds; fd++)loop check per wakeup? - How many bytes are occupied by
struct pollfd fds[250](assumingsizeof(struct pollfd) = 8bytes:4-byteint+ two2-byteshorts), and how many entries doespoll()check?
Analyzing select() Overhead:
- Each
fd_setis a bitmask ofFD_SETSIZE = 1024bits =1024 / 8 = 128 bytes. - Three sets (
readfds,writefds,exceptfds) total3 * 128 = 384 bytescopied perselect()call. nfdsmust bemax_fd + 1 = 900 + 1 = 901, so both the kernel and userFD_ISSET()loop inspect 901 positions (0..900), even though only 250 sockets are active (wasting checks on 651 unused slots).
Analyzing poll() Overhead:
- Array size =
250 * 8 bytes = 2,000 bytes. - Although
2,000 bytes > 384 bytes,poll()inspects only the 250 active entries (zero wasted scans over closed fd gaps) and never fails ifmax_fdgrows past1024.
select(): 384 bytes for 3fd_sets;nfds = 901; scans 901 fd slots per iteration.poll(): 2,000 bytes for250pollfdstructs; scans only 250 active sockets with no1024fd limit.
Single-Threaded Multi-Client TCP Echo Server Using select()
Write a C TCP echo server that handles multiple simultaneous clients in a single thread using select(), maintaining a master fd_set and tracking max_fd accurately.
Design the select() Event Loop:
- Initialize
masterwithFD_ZEROand addlisten_fd; setmax_fd = listen_fd. - In each iteration, copy
read_fds = masterbefore callingselect(max_fd + 1, &read_fds, NULL, NULL, NULL). - When
FD_ISSET(listen_fd, &read_fds)is true, callaccept(), addcli_fdtomaster, and updatemax_fd. - For every other ready
fd, read data; onrecv() <= 0, closefdand remove it withFD_CLR(fd, &master).
Complete C Implementation (select_echo.c):
#include <arpa/inet.h>
#include <stdio.h>
#include <sys/select.h>
#include <sys/socket.h>
#include <unistd.h>
int main(void) {
int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
int opt = 1;
setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
struct sockaddr_in addr = {0};
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = htonl(INADDR_ANY);
addr.sin_port = htons(9090);
bind(listen_fd, (struct sockaddr *)&addr, sizeof(addr));
listen(listen_fd, 32);
fd_set master, read_fds;
FD_ZERO(&master);
FD_SET(listen_fd, &master);
int max_fd = listen_fd;
while (1) {
read_fds = master; /* crucial copy: select() mutates read_fds */
if (select(max_fd + 1, &read_fds, NULL, NULL, NULL) < 0) continue;
for (int fd = 0; fd <= max_fd; fd++) {
if (!FD_ISSET(fd, &read_fds)) continue;
if (fd == listen_fd) {
int cli = accept(listen_fd, NULL, NULL);
if (cli >= 0 && cli < FD_SETSIZE) {
FD_SET(cli, &master);
if (cli > max_fd) max_fd = cli;
} else if (cli >= FD_SETSIZE) {
close(cli); /* guard against fd_set overflow */
}
} else {
char buf[1024];
ssize_t n = recv(fd, buf, sizeof(buf), 0);
if (n <= 0) {
close(fd);
FD_CLR(fd, &master);
} else {
send(fd, buf, (size_t)n, 0);
}
}
}
}
return 0;
}By copying master into read_fds before select(max_fd + 1, ...), checking FD_SETSIZE, adding newly accepted sockets, and removing disconnected clients with FD_CLR(), one thread serves many concurrent TCP clients without threads or busy waiting.
Non-Blocking connect() with Timeout Using poll() and SO_ERROR
Write a C function connect_with_timeout(const char *ip, uint16_t port, int timeout_ms) that initiates a non-blocking TCP connect(), waits up to timeout_ms milliseconds using poll(), and properly verifies connection success using getsockopt(..., SO_ERROR, ...).
Why poll() Writability Alone Is Not Enough: When a non-blocking connect() finishes—whether successfully or with an error like ECONNREFUSED—poll() marks the socket as writable (POLLOUT). The application must inspect SO_ERROR via getsockopt() to tell success (err == 0) apart from failure (err != 0).
Implementation (connect_with_timeout):
#include <arpa/inet.h>
#include <errno.h>
#include <fcntl.h>
#include <poll.h>
#include <sys/socket.h>
#include <unistd.h>
int connect_with_timeout(const char *ip, uint16_t port, int timeout_ms) {
int fd = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
if (fd < 0) return -1;
struct sockaddr_in srv = {0};
srv.sin_family = AF_INET;
srv.sin_port = htons(port);
inet_pton(AF_INET, ip, &srv.sin_addr);
int rc = connect(fd, (struct sockaddr *)&srv, sizeof(srv));
if (rc == 0) return fd; /* immediate local connect succeeded */
if (errno != EINPROGRESS) { close(fd); return -1; }
struct pollfd pfd = { .fd = fd, .events = POLLOUT, .revents = 0 };
rc = poll(&pfd, 1, timeout_ms);
if (rc <= 0) { close(fd); errno = (rc == 0) ? ETIMEDOUT : errno; return -1; }
int so_err = 0;
socklen_t len = sizeof(so_err);
if (getsockopt(fd, SOL_SOCKET, SO_ERROR, &so_err, &len) < 0 || so_err != 0) {
close(fd);
errno = so_err ? so_err : errno;
return -1;
}
return fd;
}Setting SOCK_NONBLOCK makes connect() return EINPROGRESS immediately; poll() waits for POLLOUT up to timeout_ms (returning ETIMEDOUT if rc == 0), and getsockopt(SOL_SOCKET, SO_ERROR) verifies whether the 3-way handshake succeeded (so_err == 0).
Exercises
Practice Exercises — Socket Programming (Part 2)
Compare blocking and non-blocking sockets. Explain the meaning of EAGAIN/EWOULDBLOCK and EINPROGRESS, and why polling a non-blocking socket in a tight while loop without select() or poll() degrades server performance.
Blocking vs. Non-Blocking Behavior:
- A blocking socket puts the calling thread to sleep in the kernel when an operation cannot finish immediately (empty receive buffer, full send buffer, empty accept queue).
- A non-blocking (
O_NONBLOCK) socket returns-1immediately instead of sleeping:recv()/accept()/send()seterrno = EAGAIN(orEWOULDBLOCK) when not ready, whileconnect()setserrno = EINPROGRESSbecause the TCP handshake proceeds asynchronously in the background.
Why Tight Busy-Polling Fails: Spinning in a tight loop (while (recv(...) < 0 && errno == EAGAIN)) forces the CPU core to remain at 100% utilization executing millions of wasted user-to-kernel mode switches per second, starving other runnable processes and wasting energy. Pairing O_NONBLOCK with select()/poll() lets the thread sleep cleanly until an I/O event actually occurs.
Blocking sockets sleep when unready; non-blocking sockets return -1 immediately with EAGAIN/EWOULDBLOCK (or EINPROGRESS on connect()). Tight busy-polling burns 100% CPU on wasted syscalls, so non-blocking sockets must be driven by I/O multiplexing (select/poll/epoll).
- On a system where
FD_SETSIZE = 2048, how many bytes does a singlefd_setbitmask occupy? Why canselect()still fail or corrupt memory if a server opens2,500concurrent sockets? - How much memory does a
struct pollfdarray (8 bytesper entry) take for 1,500 concurrent sockets, and what integer must be passed asnfdstopoll()versusselect()?
Part 1 — fd_set Size and Overflow Risk:
2048 bits / 8 bits/byte = 256 bytesperfd_set.- Because
FD_SET(fd, &set)writes bitfdinto a fixed 2048-bit array, opening2,500sockets creates descriptorsfd >= 2048, writing past the end offd_setand causing stack/heap memory corruption unless explicitly guarded.
Part 2 — pollfd Array Size and nfds Meaning:
1,500 * 8 bytes = 12,000 bytes(~11.7 KiB).- In
poll(fds, nfds, timeout),nfdsis simply the number of elements in thefdsarray (1500)—unlikeselect(), wherenfdsmust bemax_fd + 1regardless of how many sockets are open.
- One
fd_setwithFD_SETSIZE = 2048takes 256 bytes; descriptors>= 2048overflow the fixed bitmask. struct pollfd[1500]takes 12,000 bytes;poll()takesnfds = 1500(array length), whereasselect()takesnfds = max_fd + 1.
A student writes a select() server with 3 open clients (fd = 4, 5, 6), passes nfds = 3 (the client count) instead of max_fd + 1, and forgets to copy master_set before calling select(). Describe the two distinct bugs this introduces and explain how poll() avoids both.
Bug 1 — nfds = 3 Ignores All Client Sockets: select(nfds, ...) only scans descriptor numbers 0 through nfds - 1 (0, 1, 2). Since the client sockets are 4, 5, and 6, select() never monitors any of them (correct nfds = 6 + 1 = 7).
Bug 2 — Missing master_set Copy Loses Inactive FDs: select() mutates readfds in place to leave only the descriptors ready in the current wakeup. On iteration 2, every descriptor that did not send data during iteration 1 has been cleared from the set permanently! How poll() Avoids Both: poll() takes exact array length nfds (not max_fd + 1) and writes readiness into revents while leaving events untouched.
(1) Passing nfds = 3 checks only fds 0..2, completely ignoring 4, 5, 6 (needs max_fd + 1 = 7). (2) Passing master_set directly causes select() to clear idle sockets after the first wakeup. poll() avoids both because nfds is the array count and output revents is separate from input events.
How should a C network server respond to: (a) recv() == 0, (b) errno == EINTR, (c) errno == EAGAIN, and (d) errno == EPIPE? Also explain what it means if ss -ant shows hundreds of sockets stuck in CLOSE_WAIT on your server.
Handling Each Condition:
- (a)
recv() == 0: Peer closed its write side gracefully (FIN); close the local socket and free client resources. - (b)
errno == EINTR: A signal interrupted the syscall; retry the operation (unless a shutdown signal flag was set). - (c)
errno == EAGAIN: Non-blocking socket is temporarily empty/full; return to theselect()/poll()event loop and wait for the next readiness event. - (d)
errno == EPIPE: Peer closed its read end (SIGPIPEignored); close the local socket and clean up.
Diagnosing CLOSE_WAIT Accumulation: In TCP, CLOSE_WAIT means the remote peer already sent FIN, and the kernel acknowledged it, waiting for the local application to call close(fd). Hundreds of CLOSE_WAIT sockets indicate a file descriptor leak bug in your server (forgetting close(fd) when recv() == 0 or on error paths), which will eventually hit EMFILE (too many open files).
(a) recv() == 0: close fd cleanly; (b) EINTR: retry; (c) EAGAIN: return to event loop; (d) EPIPE: close fd cleanly. Hundreds of CLOSE_WAIT sockets mean the server received peer FINs but leaked descriptors by failing to call close().
A resilient TCP client uses connect_with_timeout() with poll() and exponential backoff across 3 attempts: Attempt 1 uses a 250 ms timeout, Attempt 2 doubles to 500 ms, and Attempt 3 doubles to 1000 ms.
- Calculate the worst-case total connect wait time (in milliseconds and seconds) if all 3 attempts time out.
- Explain why
getsockopt(fd, SOL_SOCKET, SO_ERROR, ...)is still required even ifpoll()reportsPOLLOUTbefore the timeout expires.
Part 1 — Total Worst-Case Timeout:
- Total timeout =
250 ms + 500 ms + 1000 ms = 1,750 ms(1.75 seconds).
Part 2 — Why SO_ERROR Is Required After POLLOUT: When a non-blocking connect() fails asynchronously (for instance, the server refuses the connection with TCP RST after 10 ms), the kernel marks the socket as both readable and writable (POLLOUT | POLLERR | POLLHUP) so the waiting process wakes up. Only getsockopt(fd, SOL_SOCKET, SO_ERROR, &err, &len) tells the application whether the handshake succeeded (err == 0) or failed (err == ECONNREFUSED).
- Worst-case wait time = 1,750 ms (1.75 s).
poll()reportsPOLLOUTfor both successful and failed non-blockingconnect()completions, sogetsockopt(..., SO_ERROR)is mandatory to check whetherso_err == 0or an error (ECONNREFUSED) occurred.