PadhaikoAll subjects

Communication in Distributed Systems

Distributed and Cloud Computing (ENCT 411)

Open in the Padhaiko app

2.1 Distributed Object Concepts

In an object-oriented distributed system, state and behavior are encapsulated inside distributed objects whose methods can be invoked across process and machine boundaries over a network.

Definition

Distributed Object

A distributed object is an object whose state (instance variables) resides in one process (the server) while its methods can be invoked remotely by clients running in separate address spaces, using remote object references and remote interfaces.

Key Building Blocks

  1. Remote Interface: Specifies the method signatures, parameter types, and return types exposed to remote callers (often defined in an Interface Definition Language, IDL), cleanly separating interface from implementation.
  2. Remote Object Reference: A globally unique identifier (typically encoding the host IP, port, creation timestamp, object number, and interface type) that lets any process address a specific remote object without ambiguity.
  3. Proxy (Client-Side Stub) and Skeleton (Server-Side Stub):
    • The proxy implements the same remote interface locally, marshalling method arguments into a network message and unmarshalling the reply.
    • The skeleton unmarshals incoming requests, dispatches the call to the real server object, and marshals the return value or exception.
  4. Object Lifecycle & Distributed Garbage Collection: Remote objects are activated on demand, bound in a naming/registry service, and reclaimed via reference counting or lease-based garbage collection when clients disconnect or crash.
Local Object vs. Distributed Object Invocation
AspectLocal Object InvocationDistributed Object Invocation
Address SpaceSame process and memory heapSeparate processes, potentially on different machines
ReferenceDirect memory pointerGlobally unique Remote Object Reference
Parameter PassingPass-by-value or pass-by-reference (pointer)Pass-by-value (marshalled copy) or pass-by-remote-reference
Latency & FailureNanoseconds; fails only if entire process crashesMilliseconds; partial network/server failures and timeouts

2.2 Middleware System Concepts

Definition

Middleware

Middleware is the software layer sitting between the operating system/network stack and distributed applications. It masks hardware, OS, and language heterogeneity while providing uniform communication, naming, security, and transaction primitives.

Core Roles and Functions of Middleware

  1. Masking Heterogeneity: Translates between differing CPU byte orders (endianness), character encodings, and language type systems.
  2. Communication Abstractions: Replaces raw TCP/UDP sockets with higher-level primitives: RPC/RMI, REST/gRPC, message queues, and publish-subscribe.
  3. Common Distributed Services: Provides directory/naming services, authentication/TLS, distributed transactions (2PC), replication, and load balancing.
  4. Distribution Transparency: Hides physical location, migration, and transient network glitches behind consistent APIs.

Classification of Middleware

Major Categories of Middleware
CategoryInteraction PatternKey Examples
RPC / Remote InvocationSynchronous request-reply procedure or method callsONC RPC, Java RMI, gRPC
Object-Oriented Middleware (ORB)Distributed object brokering via IDLCORBA, Java EE / EJB
Message-Oriented Middleware (MOM)Asynchronous message queuing and pub-subRabbitMQ (AMQP), Apache Kafka, MQTT
Web / API MiddlewareStateless HTTP/HTTP2 service endpointsRESTful APIs, GraphQL, API Gateways
Data & Transaction MiddlewareDistributed query routing and ACID coordinationODBC/JDBC drivers, XA TP monitors

2.3 Remote Method Invocation (RMI) and Remote Procedure Call (RPC)

2.3.1 Remote Procedure Call (RPC)

Introduced by Birrell and Nelson (1984), RPC allows a program to call a procedure located in another address space as if it were an ordinary local function call.

RPC Execution Sequence

RPC Client-Server Execution FlowText
Client Process                            Server Process
--------------                            --------------
1. Calls local client stub                
2. Stub marshals args into msg            
3. OS sends msg over socket  --------->   4. OS receives msg, delivers to server stub
                                          5. Server stub unmarshals args
                                          6. Calls actual server procedure
                                          7. Procedure executes & returns result
                                          8. Server stub marshals reply
10. Client stub unmarshals   <---------   9. OS sends reply over socket
11. Returns result to caller

Parameter Passing and RPC Semantics

RPC Invocation Semantics Under Failure
SemanticFault-Tolerance MechanismExecution GuaranteeSuitable For
MaybeNone (no retry, no ack)Executed or timeBest-effort telemetry / heartbeats
At-Least-OnceClient retries until reply arrivesExecuted timeIdempotent operations only (e.g., setBalance(x))
At-Most-OnceRetries + server duplicate-request cacheExecuted or time (never duplicated)Non-idempotent operations (e.g., debit(100))

2.3.2 Remote Method Invocation (RMI)

RMI extends RPC to the object-oriented paradigm (notably Java RMI):

AspectRPCRMI
Programming ParadigmProcedural (function-centric)Object-oriented (object/method-centric)
Target AddressingService endpoint + procedure IDUnique Remote Object Reference + method
ParametersPrimitive values and structs (pass-by-value)Primitives, serializable objects (value), or remote object references
Polymorphism & InheritanceNot supportedSupported via remote interfaces and subtypes

2.4 Modern Remote Invocation: RESTful Web Services and gRPC

2.4.1 RESTful Web Services

REST (Representational State Transfer), introduced by Roy Fielding (2000), models distributed applications around resources identified by URIs and manipulated through standard HTTP methods.

Six REST Architectural Constraints

  1. Client-Server: Separates UI/client concerns from data storage/server concerns.
  2. Stateless: Every request contains all context (auth token, parameters) needed to process it; the server stores no client session state between requests.
  3. Cacheable: Responses declare cacheability (Cache-Control, ETag) so intermediaries and clients can reuse responses.
  4. Uniform Interface: Resources identified by URIs (/users/42), manipulated via standard HTTP verbs (GET, POST, PUT, PATCH, DELETE), self-descriptive messages, and HATEOAS links.
  5. Layered System: Clients cannot tell whether they connect directly to an origin server or through proxies, load balancers, or API gateways.
  6. Code-on-Demand (Optional): Servers may transfer executable scripts (e.g., JavaScript) to clients.

2.4.2 gRPC

Developed by Google, gRPC is a high-performance, open-source RPC framework built on HTTP/2 and Protocol Buffers (Protobuf):

AspectRESTful APIs (HTTP/1.1 + JSON)gRPC (HTTP/2 + Protobuf)
Payload FormatTextual JSON (human-readable, larger)Binary Protobuf (compact, fast to parse)
Transport & MultiplexingTypically HTTP/1.1 (head-of-line blocking)HTTP/2 multiplexed streams over single TCP
StreamingRequest-response (needs SSE/WebSockets)Native unary, client, server, bidirectional streams
API ContractOptional OpenAPI / SwaggerStrict .proto IDL with code-generated stubs
Primary Use CasePublic web APIs, browser clientsInternal microservice-to-microservice calls

2.5 Data Serialization and Marshalling: XML, JSON, Protocol Buffers

Definition

Marshalling and Serialization

Comparing XML, JSON, and Protocol Buffers

  1. XML (eXtensible Markup Language): Verbose, self-describing tag-based text format supporting schemas (XSD) and namespaces; high parsing CPU and bandwidth overhead (used in SOAP and legacy enterprise systems).
  2. JSON (JavaScript Object Notation): Lightweight key-value text format ({"id":42,"ok":true}); natively supported in browsers, human-readable, schema-optional, moderate payload size because field names repeat in every record.
  3. Protocol Buffers (Protobuf): Binary schema-driven format using .proto definitions. Each field is encoded as a compact varint tag (field_number << 3) | wire_type followed by raw binary values—eliminating field-name strings from the wire payload. Supports forward/backward compatibility via stable field numbers.
Same Record in XML, JSON, and Protobuf SchemaText
<!-- XML (~102 bytes) -->
<User><id>1024</id><name>Asha</name><active>true</active></User>

// JSON (~41 bytes)
{"id":1024,"name":"Asha","active":true}

// Protocol Buffers .proto schema (~11 bytes on wire!)
message User {
  int32 id = 1;
  string name = 2;
  bool active = 3;
}
Serialization Format Trade-Offs
CriterionXMLJSONProtocol Buffers
EncodingText (tags)Text (key-value)Binary (tag-length-value)
Wire SizeLargest (opening + closing tags)Medium (repeated field names)Smallest ( smaller than JSON/XML)
Parse / Encode SpeedSlowestModerateFastest (minimal string scanning)
Human ReadabilityYesYesNo (binary; requires .proto schema)
Schema EvolutionXSD (rigid)Ad-hoc / JSON SchemaBuilt-in via numeric field tags

2.6 Message-Oriented Middleware: Message Queues and Publish-Subscribe Systems (RabbitMQ, MQTT)

Unlike synchronous RPC (where caller and callee must be online simultaneously), Message-Oriented Middleware (MOM) provides asynchronous, decoupled communication through persistent brokers.

2.6.1 Message Queues (Point-to-Point) vs. Publish-Subscribe

AspectMessage Queue (Point-to-Point)Publish-Subscribe (Pub-Sub)
Delivery CardinalityEach message is consumed by exactly one consumer (competing consumers)Each message is delivered to all matching subscribers
AddressingProducer sends to a named queuePublisher sends to a topic / exchange; subscribers bind with filters
Use CaseWork distribution, background jobs, order processingEvent broadcasting, IoT telemetry, real-time notifications

2.6.2 RabbitMQ (AMQP) and MQTT

Simulated RabbitMQ Topic Exchange Routing CheckPython
def topic_match(pattern: str, routing_key: str) -> bool:
    p_parts, k_parts = pattern.split("."), routing_key.split(".")
    # Exact match for single-word '*' wildcard
    return len(p_parts) == len(k_parts) and all(
        p == "*" or p == k for p, k in zip(p_parts, k_parts)
    )

print(topic_match("iot.*.temp", "iot.kathmandu.temp"))  # True
print(topic_match("iot.*.temp", "iot.kathmandu.hum"))   # False

Worked Out Examples

Example 2.1

2.1

A telemetry service transfers a batch of sensor records over a () WAN link. Each record is when serialized in JSON and when serialized in Protocol Buffers. Ignoring TCP/HTTP framing overhead, calculate:

  1. The transmission time for the JSON batch and the Protobuf batch.
  2. The bandwidth savings factor and total bytes saved per batch.

Total Payload Sizes in Bits:

Transmission Times over a Link:

Bandwidth Savings:

Answer:
  1. JSON transmission time = ; Protobuf transmission time = .
  2. Protobuf reduces payload size and transmission time by , saving () per batch.
Example 2.2

2.2

A banking client starts with balance and calls a remote RPC deposit(200) whose reply packet is lost due to a transient network drop. The client's RPC runtime times out and retransmits the exact same request once, and the second reply arrives safely. Analyze the final account balance under:

  1. At-Least-Once semantics (no server duplicate filtering).
  2. At-Most-Once semantics (server caches request IDs and replies).
  3. How could the operation be redesigned to be idempotent?

1. At-Least-Once Semantics: The server received and executed the first deposit(200) (), sent a reply that was dropped, and then received the retry and executed deposit(200) a second time (). Because deposit is not idempotent, the customer is wrongly credited twice!

2. At-Most-Once Semantics: Each RPC carries a unique (client_id, seq_no) identifier. On the first arrival, the server executes deposit(200) () and stores the result 700 without re-executing the deposit.

3. Idempotent Redesign: Replace relative updates (deposit(200)) with a unique transaction key (applyTransaction(tx_id="TX-991", amount=200)) recorded atomically in the ledger, or pass the new state if appropriate. Any retry with "TX-991" finds the existing ledger entry and becomes a safe no-op.

Answer:
  1. At-Least-Once: Balance becomes (corrupted duplicate execution).
  2. At-Most-Once: Balance becomes (duplicate suppressed via request ID cache).
  3. Attach a unique idempotency/transaction key checked atomically in the server's ledger.
Example 2.3

2.3

Exam-Style Design: An IoT smart-farm monitors soil sensors sending reading per minute over a constrained cellular link, while a backend microservice pipeline archives readings and sends SMS drought alerts.

  1. Choose between gRPC and MQTT for the sensor-to-gateway link and specify the appropriate MQTT QoS level for (a) routine temperature logs and (b) critical pump-failure alarms.
  2. Compute the number of MQTT control/data packets per minute across all sensors if critical alarms use QoS 2 vs. QoS 1 (assuming every sensor emits message/minute).

1. Protocol & QoS Selection:

  • Protocol: Choose MQTT over gRPC because MQTT has a -byte minimal header, tolerates intermittent connectivity (LWT, keepalive), and runs over low-power embedded stacks without HTTP/2 overhead.
  • QoS 0 fits routine periodic temperature logs (losing an occasional sample is harmless and saves battery).
  • QoS 1 (with idempotent alarm IDs) or QoS 2 fits critical pump-failure alarms where lost alerts are unacceptable.

2. Packet Count Comparison:

  • Under QoS 1 ( packets per message: PUBLISH + PUBACK):
  • Under QoS 2 ( packets per message: PUBLISH + PUBREC + PUBREL + PUBCOMP):
Answer:
  1. Use MQTT (QoS 0 for routine logs; QoS 1/2 for alarms).
  2. QoS 1 generates , whereas QoS 2 doubles network traffic to .

Exercises

Practice Exercises — Communication in Distributed Systems

Question 1

Explain distributed object concepts (remote object reference, proxy, skeleton) and list four core functions of middleware in a distributed system.

Distributed Object Components:

  • Remote Object Reference: Globally unique handle identifying a remote object instance across the network.
  • Proxy (Client Stub): Implements the remote interface on the client, marshalling calls into network messages.
  • Skeleton (Server Stub): Unmarshals incoming messages, invokes the target object method, and returns marshalled results.

Four Functions of Middleware:

  1. Masks hardware, OS, and language heterogeneity (endianness, type conversion).
  2. Provides high-level communication abstractions (RPC, RMI, MOM, pub-sub).
  3. Supplies shared infrastructure services (naming, security, transactions).
  4. Offers distribution transparency (location and failure handling).
Answer:

Distributed objects rely on remote object references, client proxies, and server skeletons to enable cross-process method calls. Middleware masks heterogeneity, provides communication abstractions, supplies naming/security/transaction services, and supports distribution transparency.

Question 2

Trace the execution steps of a Remote Procedure Call (RPC) using client and server stubs. Compare RPC with RMI, and explain why At-Least-Once semantics require idempotent operations.

RPC Call Sequence: The client calls the local client stub stub marshals parameters into a message OS transmits over the network server OS delivers to server stub server stub unmarshals and invokes the real procedure result is marshalled back to client stub returned to caller.

RPC vs. RMI and Idempotency:

  • RPC vs. RMI: RPC invokes stateless procedures in a procedural model; RMI invokes methods on specific remote object instances, passing object references and supporting polymorphism.
  • Idempotency: Under At-Least-Once semantics, dropped replies cause client retries, so the server may run the operation multiple times. Only idempotent operations () stay correct under duplicate execution.
Answer:

RPC routes calls through marshalling client/server stubs; RMI extends this to remote objects with object references and inheritance. At-Least-Once retries can re-execute requests, making idempotency mandatory to avoid duplicate side effects.

Question 3

Compare RESTful web services and gRPC across transport protocol, payload format, API contract, streaming support, and typical use cases. List the six REST architectural constraints.

Six REST Constraints: Client-Server, Stateless, Cacheable, Uniform Interface, Layered System, and Code-on-Demand (optional).

REST vs. gRPC Comparison:

  • REST: Typically HTTP/1.1 + text JSON, optional OpenAPI contract, request-response native, ideal for public browser/external APIs.
  • gRPC: HTTP/2 (multiplexed streams + header compression) + binary Protobuf, strict .proto IDL with generated stubs, native unary/client/server/bidirectional streaming, ideal for internal low-latency microservices.
Answer:

REST enforces six constraints (Client-Server, Stateless, Cacheable, Uniform Interface, Layered System, optional Code-on-Demand) over HTTP/JSON for public APIs, whereas gRPC combines HTTP/2 streaming with binary Protobuf and .proto contracts for high-throughput internal services.

Question 4

An event stream produces messages per second. Each message takes in XML, in JSON, and in Protocol Buffers.

  1. Calculate the required network bandwidth in () for each format.
  2. What percentage of bandwidth is saved by switching from XML to Protocol Buffers?

Bandwidth Calculation ():

Percentage Bandwidth Saved (XML Protobuf):

Answer:
  1. XML: , JSON: , Protobuf: .
  2. Switching from XML to Protobuf saves of bandwidth.
Question 5

Differentiate between Message Queues and Publish-Subscribe systems in Message-Oriented Middleware. Compare RabbitMQ (AMQP) and MQTT and explain the three MQTT QoS levels.

Message Queue vs. Publish-Subscribe:

  • Message Queue (Point-to-Point): Each message is consumed by a single worker among competing consumers (ideal for task distribution).
  • Publish-Subscribe: Each published message is broadcast to all subscribers matching a topic/filter (ideal for event fan-out).

RabbitMQ vs. MQTT & MQTT QoS Levels:

  • RabbitMQ: Full-featured AMQP broker using Exchanges (direct, fanout, topic) routing to persistent queues with consumer ACKs; suited to backend enterprise workloads.
  • MQTT: Lightweight binary pub-sub protocol (-byte header) tailored for constrained IoT sensors over unreliable networks:
    • QoS 0: At most once (fire-and-forget, packet).
    • QoS 1: At least once (PUBLISH + PUBACK, packets, duplicates possible).
    • QoS 2: Exactly once (4-way handshake, packets, highest overhead).
Answer:

Message queues deliver each message to one consumer, whereas pub-sub fans out messages to all matching subscribers. RabbitMQ offers rich AMQP exchange-to-queue routing for backend servers, while MQTT provides a minimal-header pub-sub protocol with QoS 0 (at most once), QoS 1 (at least once), and QoS 2 (exactly once) for IoT.

← Introduction to Distributed SystemsSynchronization and Coordination →