Communication in Distributed Systems
Open in the Padhaiko app2.1 Distributed Object Concepts
In an object-oriented distributed system, state and behavior are encapsulated inside distributed objects whose methods can be invoked across process and machine boundaries over a network.
Distributed Object
A distributed object is an object whose state (instance variables) resides in one process (the server) while its methods can be invoked remotely by clients running in separate address spaces, using remote object references and remote interfaces.
Key Building Blocks
- Remote Interface: Specifies the method signatures, parameter types, and return types exposed to remote callers (often defined in an Interface Definition Language, IDL), cleanly separating interface from implementation.
- Remote Object Reference: A globally unique identifier (typically encoding the host IP, port, creation timestamp, object number, and interface type) that lets any process address a specific remote object without ambiguity.
- Proxy (Client-Side Stub) and Skeleton (Server-Side Stub):
- The proxy implements the same remote interface locally, marshalling method arguments into a network message and unmarshalling the reply.
- The skeleton unmarshals incoming requests, dispatches the call to the real server object, and marshals the return value or exception.
- Object Lifecycle & Distributed Garbage Collection: Remote objects are activated on demand, bound in a naming/registry service, and reclaimed via reference counting or lease-based garbage collection when clients disconnect or crash.
| Aspect | Local Object Invocation | Distributed Object Invocation |
|---|---|---|
| Address Space | Same process and memory heap | Separate processes, potentially on different machines |
| Reference | Direct memory pointer | Globally unique Remote Object Reference |
| Parameter Passing | Pass-by-value or pass-by-reference (pointer) | Pass-by-value (marshalled copy) or pass-by-remote-reference |
| Latency & Failure | Nanoseconds; fails only if entire process crashes | Milliseconds; partial network/server failures and timeouts |
2.2 Middleware System Concepts
Middleware
Middleware is the software layer sitting between the operating system/network stack and distributed applications. It masks hardware, OS, and language heterogeneity while providing uniform communication, naming, security, and transaction primitives.
Core Roles and Functions of Middleware
- Masking Heterogeneity: Translates between differing CPU byte orders (endianness), character encodings, and language type systems.
- Communication Abstractions: Replaces raw TCP/UDP sockets with higher-level primitives: RPC/RMI, REST/gRPC, message queues, and publish-subscribe.
- Common Distributed Services: Provides directory/naming services, authentication/TLS, distributed transactions (2PC), replication, and load balancing.
- Distribution Transparency: Hides physical location, migration, and transient network glitches behind consistent APIs.
Classification of Middleware
| Category | Interaction Pattern | Key Examples |
|---|---|---|
| RPC / Remote Invocation | Synchronous request-reply procedure or method calls | ONC RPC, Java RMI, gRPC |
| Object-Oriented Middleware (ORB) | Distributed object brokering via IDL | CORBA, Java EE / EJB |
| Message-Oriented Middleware (MOM) | Asynchronous message queuing and pub-sub | RabbitMQ (AMQP), Apache Kafka, MQTT |
| Web / API Middleware | Stateless HTTP/HTTP2 service endpoints | RESTful APIs, GraphQL, API Gateways |
| Data & Transaction Middleware | Distributed query routing and ACID coordination | ODBC/JDBC drivers, XA TP monitors |
2.3 Remote Method Invocation (RMI) and Remote Procedure Call (RPC)
2.3.1 Remote Procedure Call (RPC)
Introduced by Birrell and Nelson (1984), RPC allows a program to call a procedure located in another address space as if it were an ordinary local function call.
RPC Execution Sequence
Client Process Server Process
-------------- --------------
1. Calls local client stub
2. Stub marshals args into msg
3. OS sends msg over socket ---------> 4. OS receives msg, delivers to server stub
5. Server stub unmarshals args
6. Calls actual server procedure
7. Procedure executes & returns result
8. Server stub marshals reply
10. Client stub unmarshals <--------- 9. OS sends reply over socket
11. Returns result to callerParameter Passing and RPC Semantics
- Call-by-Value: Arguments are copied into the message; simple and safe across address spaces.
- Call-by-Copy/Restore: Used when pointers/arrays are passed—copied to the server, modified, and copied back on return (true call-by-reference is impossible across disjoint memories).
- IDL (Interface Definition Language): Language-neutral specification (e.g.,
.xfor ONC RPC,.protofor gRPC) compiled into matching client and server stubs.
| Semantic | Fault-Tolerance Mechanism | Execution Guarantee | Suitable For |
|---|---|---|---|
| Maybe | None (no retry, no ack) | Executed or time | Best-effort telemetry / heartbeats |
| At-Least-Once | Client retries until reply arrives | Executed time | Idempotent operations only (e.g., setBalance(x)) |
| At-Most-Once | Retries + server duplicate-request cache | Executed or time (never duplicated) | Non-idempotent operations (e.g., debit(100)) |
2.3.2 Remote Method Invocation (RMI)
RMI extends RPC to the object-oriented paradigm (notably Java RMI):
- Clients invoke methods on specific remote object instances rather than stateless global procedures.
- Supports object references as method parameters and return values, plus dynamic class/code downloading (in Java RMI) and remote exception propagation.
| Aspect | RPC | RMI |
|---|---|---|
| Programming Paradigm | Procedural (function-centric) | Object-oriented (object/method-centric) |
| Target Addressing | Service endpoint + procedure ID | Unique Remote Object Reference + method |
| Parameters | Primitive values and structs (pass-by-value) | Primitives, serializable objects (value), or remote object references |
| Polymorphism & Inheritance | Not supported | Supported via remote interfaces and subtypes |
2.4 Modern Remote Invocation: RESTful Web Services and gRPC
2.4.1 RESTful Web Services
REST (Representational State Transfer), introduced by Roy Fielding (2000), models distributed applications around resources identified by URIs and manipulated through standard HTTP methods.
Six REST Architectural Constraints
- Client-Server: Separates UI/client concerns from data storage/server concerns.
- Stateless: Every request contains all context (auth token, parameters) needed to process it; the server stores no client session state between requests.
- Cacheable: Responses declare cacheability (
Cache-Control,ETag) so intermediaries and clients can reuse responses. - Uniform Interface: Resources identified by URIs (
/users/42), manipulated via standard HTTP verbs (GET,POST,PUT,PATCH,DELETE), self-descriptive messages, and HATEOAS links. - Layered System: Clients cannot tell whether they connect directly to an origin server or through proxies, load balancers, or API gateways.
- Code-on-Demand (Optional): Servers may transfer executable scripts (e.g., JavaScript) to clients.
2.4.2 gRPC
Developed by Google, gRPC is a high-performance, open-source RPC framework built on HTTP/2 and Protocol Buffers (Protobuf):
- Contract-First IDL: Services and messages are declared in
.protofiles and compiled into strongly typed stubs across languages. - HTTP/2 Transport: Supports multiplexing multiple concurrent RPCs over one TCP connection, HPACK header compression, and binary framing.
- Four Communication Modes:
- Unary RPC: Single request, single response (classic RPC).
- Server-Streaming RPC: Single request, stream of responses (live stock feeds).
- Client-Streaming RPC: Stream of requests, single response (file upload / sensor batch).
- Bidirectional Streaming RPC: Independent simultaneous read/write streams (chat, real-time telemetry).
| Aspect | RESTful APIs (HTTP/1.1 + JSON) | gRPC (HTTP/2 + Protobuf) |
|---|---|---|
| Payload Format | Textual JSON (human-readable, larger) | Binary Protobuf (compact, fast to parse) |
| Transport & Multiplexing | Typically HTTP/1.1 (head-of-line blocking) | HTTP/2 multiplexed streams over single TCP |
| Streaming | Request-response (needs SSE/WebSockets) | Native unary, client, server, bidirectional streams |
| API Contract | Optional OpenAPI / Swagger | Strict .proto IDL with code-generated stubs |
| Primary Use Case | Public web APIs, browser clients | Internal microservice-to-microservice calls |
2.5 Data Serialization and Marshalling: XML, JSON, Protocol Buffers
Marshalling and Serialization
- Serialization: Converting an in-memory data structure or object into a flat sequence of bytes (or text) suitable for storage or network transmission.
- Marshalling: The broader process of packaging a procedure/method invocation—including target identifier, operation name, serialized parameters, and metadata (endianness, length headers)—into a wire message (and unmarshalling it at the receiver).
Comparing XML, JSON, and Protocol Buffers
- XML (eXtensible Markup Language): Verbose, self-describing tag-based text format supporting schemas (XSD) and namespaces; high parsing CPU and bandwidth overhead (used in SOAP and legacy enterprise systems).
- JSON (JavaScript Object Notation): Lightweight key-value text format (
{"id":42,"ok":true}); natively supported in browsers, human-readable, schema-optional, moderate payload size because field names repeat in every record. - Protocol Buffers (Protobuf): Binary schema-driven format using
.protodefinitions. Each field is encoded as a compact varint tag(field_number << 3) | wire_typefollowed by raw binary values—eliminating field-name strings from the wire payload. Supports forward/backward compatibility via stable field numbers.
<!-- XML (~102 bytes) -->
<User><id>1024</id><name>Asha</name><active>true</active></User>
// JSON (~41 bytes)
{"id":1024,"name":"Asha","active":true}
// Protocol Buffers .proto schema (~11 bytes on wire!)
message User {
int32 id = 1;
string name = 2;
bool active = 3;
}| Criterion | XML | JSON | Protocol Buffers |
|---|---|---|---|
| Encoding | Text (tags) | Text (key-value) | Binary (tag-length-value) |
| Wire Size | Largest (opening + closing tags) | Medium (repeated field names) | Smallest ( smaller than JSON/XML) |
| Parse / Encode Speed | Slowest | Moderate | Fastest (minimal string scanning) |
| Human Readability | Yes | Yes | No (binary; requires .proto schema) |
| Schema Evolution | XSD (rigid) | Ad-hoc / JSON Schema | Built-in via numeric field tags |
2.6 Message-Oriented Middleware: Message Queues and Publish-Subscribe Systems (RabbitMQ, MQTT)
Unlike synchronous RPC (where caller and callee must be online simultaneously), Message-Oriented Middleware (MOM) provides asynchronous, decoupled communication through persistent brokers.
2.6.1 Message Queues (Point-to-Point) vs. Publish-Subscribe
| Aspect | Message Queue (Point-to-Point) | Publish-Subscribe (Pub-Sub) |
|---|---|---|
| Delivery Cardinality | Each message is consumed by exactly one consumer (competing consumers) | Each message is delivered to all matching subscribers |
| Addressing | Producer sends to a named queue | Publisher sends to a topic / exchange; subscribers bind with filters |
| Use Case | Work distribution, background jobs, order processing | Event broadcasting, IoT telemetry, real-time notifications |
2.6.2 RabbitMQ (AMQP) and MQTT
- RabbitMQ (AMQP 0-9-1 Broker): Producers send messages to an Exchange, which routes copies to bound Queues based on routing rules:
- Direct Exchange: Matches exact
routing_key(task queues). - Fanout Exchange: Broadcasts to all bound queues (notifications).
- Topic Exchange: Pattern-matches wildcard keys such as
sensor.*.temp(*matches one word,#matches zero or more words). - Reliability: Supports persistent queues, publisher confirms, and consumer acknowledgments (
ACK/NACK) with Dead-Letter Queues (DLQ).
- Direct Exchange: Matches exact
- MQTT (Message Queuing Telemetry Transport): Ultra-lightweight binary publish-subscribe protocol (-byte minimum header) designed for constrained IoT devices and unreliable low-bandwidth links over TCP:
- Hierarchical topics (e.g.,
factory/floor2/temp), Last Will and Testament (LWT) on unexpected disconnect, and Retained Messages. - Three QoS Levels:
- QoS 0 (At most once): Fire-and-forget; no
ACK( packet). - QoS 1 (At least once):
PUBLISHPUBACKhandshake; duplicates possible on retry ( packets). - QoS 2 (Exactly once): 4-step handshake (
PUBLISHPUBRECPUBRELPUBCOMP); guaranteed single delivery, highest overhead ( packets).
- QoS 0 (At most once): Fire-and-forget; no
- Hierarchical topics (e.g.,
def topic_match(pattern: str, routing_key: str) -> bool:
p_parts, k_parts = pattern.split("."), routing_key.split(".")
# Exact match for single-word '*' wildcard
return len(p_parts) == len(k_parts) and all(
p == "*" or p == k for p, k in zip(p_parts, k_parts)
)
print(topic_match("iot.*.temp", "iot.kathmandu.temp")) # True
print(topic_match("iot.*.temp", "iot.kathmandu.hum")) # FalseWorked Out Examples
2.1
A telemetry service transfers a batch of sensor records over a () WAN link. Each record is when serialized in JSON and when serialized in Protocol Buffers. Ignoring TCP/HTTP framing overhead, calculate:
- The transmission time for the JSON batch and the Protobuf batch.
- The bandwidth savings factor and total bytes saved per batch.
Total Payload Sizes in Bits:
Transmission Times over a Link:
Bandwidth Savings:
- JSON transmission time = ; Protobuf transmission time = .
- Protobuf reduces payload size and transmission time by , saving () per batch.
2.2
A banking client starts with balance and calls a remote RPC deposit(200) whose reply packet is lost due to a transient network drop. The client's RPC runtime times out and retransmits the exact same request once, and the second reply arrives safely. Analyze the final account balance under:
- At-Least-Once semantics (no server duplicate filtering).
- At-Most-Once semantics (server caches request IDs and replies).
- How could the operation be redesigned to be idempotent?
1. At-Least-Once Semantics: The server received and executed the first deposit(200) (), sent a reply that was dropped, and then received the retry and executed deposit(200) a second time (). Because deposit is not idempotent, the customer is wrongly credited twice!
2. At-Most-Once Semantics: Each RPC carries a unique (client_id, seq_no) identifier. On the first arrival, the server executes deposit(200) () and stores the result 700 without re-executing the deposit.
3. Idempotent Redesign: Replace relative updates (deposit(200)) with a unique transaction key (applyTransaction(tx_id="TX-991", amount=200)) recorded atomically in the ledger, or pass the new state if appropriate. Any retry with "TX-991" finds the existing ledger entry and becomes a safe no-op.
- At-Least-Once: Balance becomes (corrupted duplicate execution).
- At-Most-Once: Balance becomes (duplicate suppressed via request ID cache).
- Attach a unique idempotency/transaction key checked atomically in the server's ledger.
2.3
Exam-Style Design: An IoT smart-farm monitors soil sensors sending reading per minute over a constrained cellular link, while a backend microservice pipeline archives readings and sends SMS drought alerts.
- Choose between gRPC and MQTT for the sensor-to-gateway link and specify the appropriate MQTT QoS level for (a) routine temperature logs and (b) critical pump-failure alarms.
- Compute the number of MQTT control/data packets per minute across all sensors if critical alarms use QoS 2 vs. QoS 1 (assuming every sensor emits message/minute).
1. Protocol & QoS Selection:
- Protocol: Choose MQTT over gRPC because MQTT has a -byte minimal header, tolerates intermittent connectivity (LWT, keepalive), and runs over low-power embedded stacks without HTTP/2 overhead.
- QoS 0 fits routine periodic temperature logs (losing an occasional sample is harmless and saves battery).
- QoS 1 (with idempotent alarm IDs) or QoS 2 fits critical pump-failure alarms where lost alerts are unacceptable.
2. Packet Count Comparison:
- Under QoS 1 ( packets per message:
PUBLISH+PUBACK):
- Under QoS 2 ( packets per message:
PUBLISH+PUBREC+PUBREL+PUBCOMP):
- Use MQTT (QoS 0 for routine logs; QoS 1/2 for alarms).
- QoS 1 generates , whereas QoS 2 doubles network traffic to .
Exercises
Practice Exercises — Communication in Distributed Systems
Explain distributed object concepts (remote object reference, proxy, skeleton) and list four core functions of middleware in a distributed system.
Distributed Object Components:
- Remote Object Reference: Globally unique handle identifying a remote object instance across the network.
- Proxy (Client Stub): Implements the remote interface on the client, marshalling calls into network messages.
- Skeleton (Server Stub): Unmarshals incoming messages, invokes the target object method, and returns marshalled results.
Four Functions of Middleware:
- Masks hardware, OS, and language heterogeneity (endianness, type conversion).
- Provides high-level communication abstractions (RPC, RMI, MOM, pub-sub).
- Supplies shared infrastructure services (naming, security, transactions).
- Offers distribution transparency (location and failure handling).
Distributed objects rely on remote object references, client proxies, and server skeletons to enable cross-process method calls. Middleware masks heterogeneity, provides communication abstractions, supplies naming/security/transaction services, and supports distribution transparency.
Trace the execution steps of a Remote Procedure Call (RPC) using client and server stubs. Compare RPC with RMI, and explain why At-Least-Once semantics require idempotent operations.
RPC Call Sequence: The client calls the local client stub stub marshals parameters into a message OS transmits over the network server OS delivers to server stub server stub unmarshals and invokes the real procedure result is marshalled back to client stub returned to caller.
RPC vs. RMI and Idempotency:
- RPC vs. RMI: RPC invokes stateless procedures in a procedural model; RMI invokes methods on specific remote object instances, passing object references and supporting polymorphism.
- Idempotency: Under At-Least-Once semantics, dropped replies cause client retries, so the server may run the operation multiple times. Only idempotent operations () stay correct under duplicate execution.
RPC routes calls through marshalling client/server stubs; RMI extends this to remote objects with object references and inheritance. At-Least-Once retries can re-execute requests, making idempotency mandatory to avoid duplicate side effects.
Compare RESTful web services and gRPC across transport protocol, payload format, API contract, streaming support, and typical use cases. List the six REST architectural constraints.
Six REST Constraints: Client-Server, Stateless, Cacheable, Uniform Interface, Layered System, and Code-on-Demand (optional).
REST vs. gRPC Comparison:
- REST: Typically HTTP/1.1 + text JSON, optional OpenAPI contract, request-response native, ideal for public browser/external APIs.
- gRPC: HTTP/2 (multiplexed streams + header compression) + binary Protobuf, strict
.protoIDL with generated stubs, native unary/client/server/bidirectional streaming, ideal for internal low-latency microservices.
REST enforces six constraints (Client-Server, Stateless, Cacheable, Uniform Interface, Layered System, optional Code-on-Demand) over HTTP/JSON for public APIs, whereas gRPC combines HTTP/2 streaming with binary Protobuf and .proto contracts for high-throughput internal services.
An event stream produces messages per second. Each message takes in XML, in JSON, and in Protocol Buffers.
- Calculate the required network bandwidth in () for each format.
- What percentage of bandwidth is saved by switching from XML to Protocol Buffers?
Bandwidth Calculation ():
Percentage Bandwidth Saved (XML Protobuf):
- XML: , JSON: , Protobuf: .
- Switching from XML to Protobuf saves of bandwidth.
Differentiate between Message Queues and Publish-Subscribe systems in Message-Oriented Middleware. Compare RabbitMQ (AMQP) and MQTT and explain the three MQTT QoS levels.
Message Queue vs. Publish-Subscribe:
- Message Queue (Point-to-Point): Each message is consumed by a single worker among competing consumers (ideal for task distribution).
- Publish-Subscribe: Each published message is broadcast to all subscribers matching a topic/filter (ideal for event fan-out).
RabbitMQ vs. MQTT & MQTT QoS Levels:
- RabbitMQ: Full-featured AMQP broker using Exchanges (direct, fanout, topic) routing to persistent queues with consumer ACKs; suited to backend enterprise workloads.
- MQTT: Lightweight binary pub-sub protocol (-byte header) tailored for constrained IoT sensors over unreliable networks:
- QoS 0: At most once (fire-and-forget, packet).
- QoS 1: At least once (
PUBLISH+PUBACK, packets, duplicates possible). - QoS 2: Exactly once (4-way handshake, packets, highest overhead).
Message queues deliver each message to one consumer, whereas pub-sub fans out messages to all matching subscribers. RabbitMQ offers rich AMQP exchange-to-queue routing for backend servers, while MQTT provides a minimal-header pub-sub protocol with QoS 0 (at most once), QoS 1 (at least once), and QoS 2 (exactly once) for IoT.