PadhaikoAll subjects

Socket Programming and Network APIs (part 2: 4.4–4.6)

Network and Systems Programming (ENCT 386)

Open in the Padhaiko app

4.4 Blocking versus non-blocking sockets

By default, every POSIX socket created with socket() or returned by accept() operates in blocking mode. Understanding when a socket blocks—and how non-blocking mode alters every system call—is fundamental to building responsive network servers.

Definition

Blocking vs. Non-Blocking Socket Operations

Enabling Non-Blocking Mode in C

Two standard POSIX mechanisms enable O_NONBLOCK:

  1. fcntl() on an existing socket:
Setting O_NONBLOCK via fcntl()C
int flags = fcntl(sockfd, F_GETFL, 0);
if (flags >= 0) {
    fcntl(sockfd, F_SETFL, flags | O_NONBLOCK);
}
  1. Atomic creation flags on Linux: socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0) and accept4(srv, addr, len, SOCK_NONBLOCK) avoid an extra fcntl() system call.
Behavior of Socket Calls in Blocking vs. Non-Blocking Mode
System CallBlocking Mode BehaviorNon-Blocking Mode (O_NONBLOCK) Behavior
accept()Sleeps until a client finishes the 3-way handshakeReturns -1 (EAGAIN / EWOULDBLOCK) if Accept queue is empty
recv() / read()Sleeps until at least 1 byte arrives or peer sends FIN (0)Returns available bytes immediately, 0 on EOF, or -1 (EAGAIN) if empty
send() / write()Sleeps until all bytes fit in kernel send buffer (or error)Copies as many bytes as fit; returns -1 (EAGAIN) if send buffer is full
connect()Sleeps until 3-way handshake finishes or failsInitiates handshake and returns -1 with errno == EINPROGRESS

4.5 I/O multiplexing (Select, poll basics)

I/O multiplexing enables a single thread to monitor dozens or thousands of file descriptors simultaneously and block in one kernel call until one or more of them are ready for reading, writing, or exception handling.

1. The select() System Call (<sys/select.h>)

select() Signature and fd_set MacrosC
int select(int nfds, fd_set *readfds, fd_set *writefds,
           fd_set *exceptfds, struct timeval *timeout);

FD_ZERO(&set);        /* clear all bits in set */
FD_SET(fd, &set);     /* add fd to set */
FD_CLR(fd, &set);     /* remove fd from set */
FD_ISSET(fd, &set);   /* test whether fd is ready after select() returns */

Critical select() Rules and Limitations

  1. nfds MUST be max_fd + 1: Passing the count of sockets instead of max_fd + 1 is a classic bug that causes select() to silently ignore higher-numbered file descriptors!
  2. In-Place Bitmask Mutation: select() overwrites readfds, writefds, and exceptfds to report which fds are ready. You must copy a persistent master_set into a temporary working_set before every select() call.
  3. FD_SETSIZE Hard Limit: Standard fd_set is a fixed-size bit array (typically 1024 bits = 128 bytes on glibc Linux). Monitoring any fd >= 1024 causes buffer overflow / undefined behavior.
  4. Linear Scanning (O(max_fd)): Every call copies the bitmasks user->kernel->user and forces user code to loop 0 .. max_fd checking FD_ISSET().

2. The poll() System Call (<poll.h>)

poll() fixes two major design flaws of select() (FD_SETSIZE cap and bitmask destruction) by passing an explicit array of struct pollfd:

poll() Structure and Event FlagsC
struct pollfd {
    int   fd;       /* file descriptor to watch (ignore entry if < 0) */
    short events;   /* requested events: POLLIN, POLLOUT */
    short revents;  /* returned events filled by kernel: POLLIN, POLLOUT, POLLERR, POLLHUP */
};

int poll(struct pollfd fds[], nfds_t nfds, int timeout_ms);
Aspectselect()poll()
Data RepresentationThree fixed bitmasks (fd_set)Caller-allocated array of struct pollfd
Max FD NumberHard-capped at FD_SETSIZE (1024) on LinuxNo fixed limit (bounded only by RLIMIT_NOFILE)
Re-initializationMust copy master_set before every call (mutated in place)Not needed (events preserved; kernel writes only revents)
Timeout Resolutionstruct timeval (microseconds; select may modify it)int timeout_ms (milliseconds; unmodified)
Scan ComplexityO(max_fd) even if only 2 high fds are openO(N) where N is number of entries in fds[]

4.6 Error handling and debugging techniques

Network programs fail constantly due to unreachable hosts, dropped links, interrupted syscalls, and half-closed peers. Robust error handling distinguishes recoverable transient conditions from fatal bugs.

Classifying Socket Errors

Essential Socket Error Codes and Recovery Actions
Condition / errnoCauseProper Server / Client Action
recv() == 0Remote peer closed connection cleanly (FIN)Not an error—close local socket and clean up client state
EINTRBlocking syscall interrupted by a caught signalRetry the syscall (or check shutdown flag)
EAGAIN / EWOULDBLOCKNon-blocking socket has no data / buffer fullStop reading/writing for now; wait via select/poll/epoll
EPIPE (after SIGPIPE ignored)Writing to a socket whose peer already closedClose local fd and remove client cleanly
ECONNRESETPeer crashed or sent TCP RSTLog, close local fd, and release resources
EADDRINUSEPort already bound (or lingering in TIME_WAIT)Enable SO_REUSEADDR or check conflicting process
EINPROGRESSNon-blocking connect() handshake in progressWait for writability (POLLOUT) in poll(), then inspect SO_ERROR

Diagnostic APIs and CLI Debugging Tools

  1. API Diagnostics:
    • perror("msg") / strerror(errno) for standard syscalls; gai_strerror(rc) for getaddrinfo() DNS failures (since getaddrinfo() returns its own error codes rather than setting errno).
    • getsockopt(fd, SOL_SOCKET, SO_ERROR, &err, &len) retrieves and clears pending asynchronous socket errors (essential after non-blocking connect()).
    • SO_RCVTIMEO and SO_SNDTIMEO attach kernel-enforced timeouts to blocking recv()/send() calls.
  2. Command-Line Network Debugging Tools:
    • ss -tulpn / netstat -antp: Lists listening ports, owning PIDs, and TCP states. A growing pile of sockets in CLOSE_WAIT indicates your server forgot to call close() after receiving peer FIN; many TIME_WAIT sockets are normal on the side that closed first.
    • strace -e trace=network ./server: Traces every socket, bind, connect, send, recv, and exact errno returned without recompiling.
    • lsof -p <pid>: Lists all open file descriptors and sockets held by a running process to spot fd leaks.
    • tcpdump / Wireshark: Captures live wire packets to inspect 3-way handshakes, retransmissions, RST flags, and malformed framing headers.

Worked Examples

Example

Comparing select() and poll() Scan Overhead and Memory Usage

A process opens a few files and then holds 250 active sockets whose highest file descriptor number is max_fd = 900.

  1. How many bytes are occupied by three fd_set structures (readfds, writefds, exceptfds) passed to select() when FD_SETSIZE = 1024? What value must be passed as nfds to select(), and how many fd positions must a for (fd = 0; fd < nfds; fd++) loop check per wakeup?
  2. How many bytes are occupied by struct pollfd fds[250] (assuming sizeof(struct pollfd) = 8 bytes: 4-byte int + two 2-byte shorts), and how many entries does poll() check?

Analyzing select() Overhead:

  • Each fd_set is a bitmask of FD_SETSIZE = 1024 bits = 1024 / 8 = 128 bytes.
  • Three sets (readfds, writefds, exceptfds) total 3 * 128 = 384 bytes copied per select() call.
  • nfds must be max_fd + 1 = 900 + 1 = 901, so both the kernel and user FD_ISSET() loop inspect 901 positions (0..900), even though only 250 sockets are active (wasting checks on 651 unused slots).

Analyzing poll() Overhead:

  • Array size = 250 * 8 bytes = 2,000 bytes.
  • Although 2,000 bytes > 384 bytes, poll() inspects only the 250 active entries (zero wasted scans over closed fd gaps) and never fails if max_fd grows past 1024.
Answer:
  1. select(): 384 bytes for 3 fd_sets; nfds = 901; scans 901 fd slots per iteration.
  2. poll(): 2,000 bytes for 250 pollfd structs; scans only 250 active sockets with no 1024 fd limit.
Example

Single-Threaded Multi-Client TCP Echo Server Using select()

Write a C TCP echo server that handles multiple simultaneous clients in a single thread using select(), maintaining a master fd_set and tracking max_fd accurately.

Design the select() Event Loop:

  1. Initialize master with FD_ZERO and add listen_fd; set max_fd = listen_fd.
  2. In each iteration, copy read_fds = master before calling select(max_fd + 1, &read_fds, NULL, NULL, NULL).
  3. When FD_ISSET(listen_fd, &read_fds) is true, call accept(), add cli_fd to master, and update max_fd.
  4. For every other ready fd, read data; on recv() <= 0, close fd and remove it with FD_CLR(fd, &master).

Complete C Implementation (select_echo.c):

Multi-Client TCP Echo Server with select()C
#include <arpa/inet.h>
#include <stdio.h>
#include <sys/select.h>
#include <sys/socket.h>
#include <unistd.h>

int main(void) {
    int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
    int opt = 1;
    setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));

    struct sockaddr_in addr = {0};
    addr.sin_family = AF_INET;
    addr.sin_addr.s_addr = htonl(INADDR_ANY);
    addr.sin_port = htons(9090);
    bind(listen_fd, (struct sockaddr *)&addr, sizeof(addr));
    listen(listen_fd, 32);

    fd_set master, read_fds;
    FD_ZERO(&master);
    FD_SET(listen_fd, &master);
    int max_fd = listen_fd;

    while (1) {
        read_fds = master; /* crucial copy: select() mutates read_fds */
        if (select(max_fd + 1, &read_fds, NULL, NULL, NULL) < 0) continue;

        for (int fd = 0; fd <= max_fd; fd++) {
            if (!FD_ISSET(fd, &read_fds)) continue;

            if (fd == listen_fd) {
                int cli = accept(listen_fd, NULL, NULL);
                if (cli >= 0 && cli < FD_SETSIZE) {
                    FD_SET(cli, &master);
                    if (cli > max_fd) max_fd = cli;
                } else if (cli >= FD_SETSIZE) {
                    close(cli); /* guard against fd_set overflow */
                }
            } else {
                char buf[1024];
                ssize_t n = recv(fd, buf, sizeof(buf), 0);
                if (n <= 0) {
                    close(fd);
                    FD_CLR(fd, &master);
                } else {
                    send(fd, buf, (size_t)n, 0);
                }
            }
        }
    }
    return 0;
}
Answer:

By copying master into read_fds before select(max_fd + 1, ...), checking FD_SETSIZE, adding newly accepted sockets, and removing disconnected clients with FD_CLR(), one thread serves many concurrent TCP clients without threads or busy waiting.

Example

Non-Blocking connect() with Timeout Using poll() and SO_ERROR

Write a C function connect_with_timeout(const char *ip, uint16_t port, int timeout_ms) that initiates a non-blocking TCP connect(), waits up to timeout_ms milliseconds using poll(), and properly verifies connection success using getsockopt(..., SO_ERROR, ...).

Why poll() Writability Alone Is Not Enough: When a non-blocking connect() finishes—whether successfully or with an error like ECONNREFUSED—poll() marks the socket as writable (POLLOUT). The application must inspect SO_ERROR via getsockopt() to tell success (err == 0) apart from failure (err != 0).

Implementation (connect_with_timeout):

Non-Blocking connect() with poll() Timeout and SO_ERROR CheckC
#include <arpa/inet.h>
#include <errno.h>
#include <fcntl.h>
#include <poll.h>
#include <sys/socket.h>
#include <unistd.h>

int connect_with_timeout(const char *ip, uint16_t port, int timeout_ms) {
    int fd = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
    if (fd < 0) return -1;

    struct sockaddr_in srv = {0};
    srv.sin_family = AF_INET;
    srv.sin_port = htons(port);
    inet_pton(AF_INET, ip, &srv.sin_addr);

    int rc = connect(fd, (struct sockaddr *)&srv, sizeof(srv));
    if (rc == 0) return fd; /* immediate local connect succeeded */
    if (errno != EINPROGRESS) { close(fd); return -1; }

    struct pollfd pfd = { .fd = fd, .events = POLLOUT, .revents = 0 };
    rc = poll(&pfd, 1, timeout_ms);
    if (rc <= 0) { close(fd); errno = (rc == 0) ? ETIMEDOUT : errno; return -1; }

    int so_err = 0;
    socklen_t len = sizeof(so_err);
    if (getsockopt(fd, SOL_SOCKET, SO_ERROR, &so_err, &len) < 0 || so_err != 0) {
        close(fd);
        errno = so_err ? so_err : errno;
        return -1;
    }
    return fd;
}
Answer:

Setting SOCK_NONBLOCK makes connect() return EINPROGRESS immediately; poll() waits for POLLOUT up to timeout_ms (returning ETIMEDOUT if rc == 0), and getsockopt(SOL_SOCKET, SO_ERROR) verifies whether the 3-way handshake succeeded (so_err == 0).


Exercises

Practice Exercises — Socket Programming (Part 2)

Question 1

Compare blocking and non-blocking sockets. Explain the meaning of EAGAIN/EWOULDBLOCK and EINPROGRESS, and why polling a non-blocking socket in a tight while loop without select() or poll() degrades server performance.

Blocking vs. Non-Blocking Behavior:

  • A blocking socket puts the calling thread to sleep in the kernel when an operation cannot finish immediately (empty receive buffer, full send buffer, empty accept queue).
  • A non-blocking (O_NONBLOCK) socket returns -1 immediately instead of sleeping: recv()/accept()/send() set errno = EAGAIN (or EWOULDBLOCK) when not ready, while connect() sets errno = EINPROGRESS because the TCP handshake proceeds asynchronously in the background.

Why Tight Busy-Polling Fails: Spinning in a tight loop (while (recv(...) < 0 && errno == EAGAIN)) forces the CPU core to remain at 100% utilization executing millions of wasted user-to-kernel mode switches per second, starving other runnable processes and wasting energy. Pairing O_NONBLOCK with select()/poll() lets the thread sleep cleanly until an I/O event actually occurs.

Answer:

Blocking sockets sleep when unready; non-blocking sockets return -1 immediately with EAGAIN/EWOULDBLOCK (or EINPROGRESS on connect()). Tight busy-polling burns 100% CPU on wasted syscalls, so non-blocking sockets must be driven by I/O multiplexing (select/poll/epoll).

Question 2
  1. On a system where FD_SETSIZE = 2048, how many bytes does a single fd_set bitmask occupy? Why can select() still fail or corrupt memory if a server opens 2,500 concurrent sockets?
  2. How much memory does a struct pollfd array (8 bytes per entry) take for 1,500 concurrent sockets, and what integer must be passed as nfds to poll() versus select()?

Part 1 — fd_set Size and Overflow Risk:

  • 2048 bits / 8 bits/byte = 256 bytes per fd_set.
  • Because FD_SET(fd, &set) writes bit fd into a fixed 2048-bit array, opening 2,500 sockets creates descriptors fd >= 2048, writing past the end of fd_set and causing stack/heap memory corruption unless explicitly guarded.

Part 2 — pollfd Array Size and nfds Meaning:

  • 1,500 * 8 bytes = 12,000 bytes (~11.7 KiB).
  • In poll(fds, nfds, timeout), nfds is simply the number of elements in the fds array (1500)—unlike select(), where nfds must be max_fd + 1 regardless of how many sockets are open.
Answer:
  1. One fd_set with FD_SETSIZE = 2048 takes 256 bytes; descriptors >= 2048 overflow the fixed bitmask.
  2. struct pollfd[1500] takes 12,000 bytes; poll() takes nfds = 1500 (array length), whereas select() takes nfds = max_fd + 1.
Question 3

A student writes a select() server with 3 open clients (fd = 4, 5, 6), passes nfds = 3 (the client count) instead of max_fd + 1, and forgets to copy master_set before calling select(). Describe the two distinct bugs this introduces and explain how poll() avoids both.

Bug 1 — nfds = 3 Ignores All Client Sockets: select(nfds, ...) only scans descriptor numbers 0 through nfds - 1 (0, 1, 2). Since the client sockets are 4, 5, and 6, select() never monitors any of them (correct nfds = 6 + 1 = 7).

Bug 2 — Missing master_set Copy Loses Inactive FDs: select() mutates readfds in place to leave only the descriptors ready in the current wakeup. On iteration 2, every descriptor that did not send data during iteration 1 has been cleared from the set permanently! How poll() Avoids Both: poll() takes exact array length nfds (not max_fd + 1) and writes readiness into revents while leaving events untouched.

Answer:

(1) Passing nfds = 3 checks only fds 0..2, completely ignoring 4, 5, 6 (needs max_fd + 1 = 7). (2) Passing master_set directly causes select() to clear idle sockets after the first wakeup. poll() avoids both because nfds is the array count and output revents is separate from input events.

Question 4

How should a C network server respond to: (a) recv() == 0, (b) errno == EINTR, (c) errno == EAGAIN, and (d) errno == EPIPE? Also explain what it means if ss -ant shows hundreds of sockets stuck in CLOSE_WAIT on your server.

Handling Each Condition:

  • (a) recv() == 0: Peer closed its write side gracefully (FIN); close the local socket and free client resources.
  • (b) errno == EINTR: A signal interrupted the syscall; retry the operation (unless a shutdown signal flag was set).
  • (c) errno == EAGAIN: Non-blocking socket is temporarily empty/full; return to the select()/poll() event loop and wait for the next readiness event.
  • (d) errno == EPIPE: Peer closed its read end (SIGPIPE ignored); close the local socket and clean up.

Diagnosing CLOSE_WAIT Accumulation: In TCP, CLOSE_WAIT means the remote peer already sent FIN, and the kernel acknowledged it, waiting for the local application to call close(fd). Hundreds of CLOSE_WAIT sockets indicate a file descriptor leak bug in your server (forgetting close(fd) when recv() == 0 or on error paths), which will eventually hit EMFILE (too many open files).

Answer:

(a) recv() == 0: close fd cleanly; (b) EINTR: retry; (c) EAGAIN: return to event loop; (d) EPIPE: close fd cleanly. Hundreds of CLOSE_WAIT sockets mean the server received peer FINs but leaked descriptors by failing to call close().

Question 5

A resilient TCP client uses connect_with_timeout() with poll() and exponential backoff across 3 attempts: Attempt 1 uses a 250 ms timeout, Attempt 2 doubles to 500 ms, and Attempt 3 doubles to 1000 ms.

  1. Calculate the worst-case total connect wait time (in milliseconds and seconds) if all 3 attempts time out.
  2. Explain why getsockopt(fd, SOL_SOCKET, SO_ERROR, ...) is still required even if poll() reports POLLOUT before the timeout expires.

Part 1 — Total Worst-Case Timeout:

  • Total timeout = 250 ms + 500 ms + 1000 ms = 1,750 ms (1.75 seconds).

Part 2 — Why SO_ERROR Is Required After POLLOUT: When a non-blocking connect() fails asynchronously (for instance, the server refuses the connection with TCP RST after 10 ms), the kernel marks the socket as both readable and writable (POLLOUT | POLLERR | POLLHUP) so the waiting process wakes up. Only getsockopt(fd, SOL_SOCKET, SO_ERROR, &err, &len) tells the application whether the handshake succeeded (err == 0) or failed (err == ECONNREFUSED).

Answer:
  1. Worst-case wait time = 1,750 ms (1.75 s).
  2. poll() reports POLLOUT for both successful and failed non-blocking connect() completions, so getsockopt(..., SO_ERROR) is mandatory to check whether so_err == 0 or an error (ECONNREFUSED) occurred.
← Socket Programming and Network APIs (part 1: 4.1–4.3)Advanced Network Programming →