PadhaikoAll subjects

Emerging Trends

Artificial Intelligence (ENCT 351)

Open in the Padhaiko app

7.1 Sequence to sequence models

A Sequence-to-Sequence (Seq2Seq) model maps a variable-length input sequence to a variable-length output sequence —where and need not be equal (machine translation, text summarization, speech recognition, chatbots).

Encoder–Decoder Architecture

Encoder-Decoder Seq2Seq flowText
Input:  "How are you" ──► [ ENCODER (RNN/LSTM/Transformer) ] ──► context / K, V states
                                                                       │
Output: "कस्तो छ"      ◄── [ DECODER (auto-regressive) ] ◄───────────────┘
  1. Encoder: Reads the entire input sequence and produces contextual representations (originally a single final hidden state ).
  2. Decoder: Generates the output sequence one token at a time, conditioned on previously generated tokens and the encoder representation:
  1. Training vs. Inference: Trained with teacher forcing (feeding the ground-truth at each step); decoded at inference via greedy search or beam search (keeping top- partial hypotheses).

Attention Mechanism and Transformers

Compressing a long sentence into one fixed vector creates an information bottleneck. Attention (Bahdanau et al., 2015) lets the decoder look back at every encoder hidden state at each step via dynamic weights :

Modern Transformers (Vaswani et al., 2017) drop recurrence altogether and rely purely on Scaled Dot-Product Self-Attention:

This enables massive parallelization during training and captures long-range dependencies across thousands of tokens.

7.2 Federated learning

Traditional ML pools all user data onto a central server, creating privacy risks, regulatory barriers (GDPR/HIPAA), and high network bandwidth costs. Federated Learning (FL) trains a shared global model across many decentralized client devices (phones, hospitals, IoT nodes) without raw data ever leaving the device.

Definition

Federated Averaging (FedAvg)

In each communication round of FedAvg (McMahan et al., 2017):

  1. Broadcast: The central server sends the current global weights to a sampled subset of clients.
  2. Local training: Each client performs local SGD updates on its private dataset of size to obtain updated weights .
  3. Upload & Secure Aggregation: Clients upload only model updates/weights (not raw data). The server aggregates them weighted by dataset size ():

7.3 Edge AI

Edge AI executes AI inference (and sometimes lightweight adaptation) directly on local hardware—smartphones, cameras, drones, wearables, microcontrollers (TinyML)—rather than sending every sensor reading to cloud data centres.

AspectCloud AIEdge AI
Compute locationRemote GPU/TPU data centresOn-device (NPU, DSP, MCU, mobile GPU)
LatencyHigher (network round-trip + queuing)Ultra-low (real-time milliseconds)
Offline operationRequires internet connectivityWorks fully offline
PrivacyRaw audio/video leaves the deviceData stays local on the device
Compute & memoryVirtually unlimitedTightly constrained (power, RAM, storage)

Enabling Techniques for Edge AI

7.4 Ethics and AI: Responsible AI

As AI shapes hiring, healthcare, credit, policing, and public information, technical accuracy alone is insufficient. Responsible AI designs and governs AI systems to be fair, transparent, safe, and accountable.

Core Principles of Responsible AI
PrincipleMeaningPractical Engineering Measure
Fairness & Non-DiscriminationAvoid systematic harm across gender, caste, ethnicity, or ageAudit subgroup error rates; enforce demographic parity or equalized odds
Transparency & Explainability (XAI)Users understand when AI is used and why a decision was madeModel cards, feature attribution (SHAP, LIME), confidence disclosures
Privacy & Data GovernanceProtect personal data from misuse or reconstructionData minimization, anonymization, Differential Privacy, Federated Learning
Safety, Robustness & ReliabilityResist adversarial inputs, distribution shifts, and failuresRed-teaming, OOD detection, human-in-the-loop fallback
AccountabilityClear human/organizational responsibility for outcomesAudit logs, impact assessments, recourse mechanisms for affected people

Worked Out Examples

Example 7.1

7.1 (6 marks)

In a Seq2Seq Transformer with key dimension , a decoder query attends over two encoder keys and , with scalar values and . Compute the attention weights and the output context value .

Compute scaled dot products ():

  • .
  • .

Apply Softmax:

Weighted sum of values:

Answer:

Attention weights: , ; output context .

Example 7.2

7.2 (6 marks)

Three hospitals train a shared diagnostic parameter via FedAvg without sharing patient records. Their local sample counts and updated weights after round are: Hospital A (), Hospital B (), and Hospital C (). Compute the aggregated global weight . Compare this with unweighted averaging.

Total training samples across participating clients:

Weighted FedAvg aggregation:

Compare with unweighted mean: Unweighted averaging would give , unfairly letting the smallest hospital ( of data) distort the global model.

Answer:

FedAvg global weight (versus biased unweighted average ).

Example 7.3

7.3 (6 marks)

(a) An Edge AI model has parameters stored in 32-bit float (FP32, bytes/param). Compute its model size in MiB ( bytes) before and after 8-bit integer (INT8, byte/param) quantization. (b) A hiring model approves of applicants in Group A and of applicants in Group B. Compute the Demographic Parity Difference between the two groups.

Edge AI model size:

  • FP32 ( bytes/param): .
  • INT8 ( byte/param): ( smaller).

Demographic Parity Difference:

Answer:

(a) FP32 size , INT8 size ( reduction); (b) Demographic Parity Difference .

Exercises

Exercise: Seq2Seq, Federated Learning, Edge AI, and Responsible AI

Question 1

Explain the encoder–decoder architecture of a Seq2Seq model. Why does a single fixed context vector suffer on long sequences, and how does Attention solve this?

Encoder–Decoder: The encoder reads a variable-length input sequence into hidden representations; the decoder generates the output sequence one token at a time conditioned on prior outputs and encoder states.

Bottleneck & Attention: Compressing an entire long sentence into one fixed vector loses early details. Attention lets the decoder compute a fresh weighted sum of all encoder hidden states at every output step, focusing dynamically on relevant input words.

Answer:

Seq2Seq pairs an encoder and an autoregressive decoder; attention replaces the single fixed context vector bottleneck with dynamic step-by-step weighted combinations of all encoder states.

Question 2

What is Federated Learning and how does it preserve data privacy? Two devices participate in a FedAvg round with local datasets (weight ) and (weight ). Compute the new global weight.

Concept: Federated Learning trains a shared model across decentralized devices by sending model updates rather than raw user data to the central server.

FedAvg calculation ():

Answer:

Federated Learning keeps raw data on-device and aggregates parameter updates centrally. Global weight .

Question 3

Compare Edge AI and Cloud AI across latency, privacy, connectivity, and compute constraints. If an FP32 model is quantized from 32-bit float to 8-bit integer for an edge microcontroller, what is its new size (ignoring metadata)?

Comparison: Edge AI runs locally with millisecond latency, works offline, and keeps raw data on-device, but faces strict RAM/power limits. Cloud AI offers massive compute at the cost of network latency, internet dependence, and sending data off-device.

Quantized size: Reducing bit-width from bits ( bytes) to bits ( byte) compresses weights by a factor of :

Answer:

Edge AI trades cloud compute scale for low latency, offline reliability, and local privacy. INT8 quantization shrinks the model to .

Question 4

List the five core principles of Responsible AI and explain three sources of bias in AI systems. If a loan-approval system approves of urban applicants and of rural applicants, compute the Demographic Parity Difference.

Principles & bias: Core principles: Fairness, Transparency/Explainability, Privacy/Governance, Safety/Robustness, and Accountability. Bias sources: historical/sampling imbalance, biased labels/proxies, and feedback loops.

Demographic Parity Difference:

Answer:

Responsible AI rests on fairness, transparency, privacy, safety, and accountability; bias stems from sampling, label proxies, and feedback loops. Demographic Parity Difference .

Question 5

In a Transformer attention layer, two encoder positions produce equal scaled attention scores and with value scalars and . Compute the Softmax attention weights and the resulting context value .

Softmax weights: Since :

Context value:

Answer:

, , and .

← AI Applications