Emerging Trends
Open in the Padhaiko app7.1 Sequence to sequence models
A Sequence-to-Sequence (Seq2Seq) model maps a variable-length input sequence to a variable-length output sequence —where and need not be equal (machine translation, text summarization, speech recognition, chatbots).
Encoder–Decoder Architecture
Input: "How are you" ──► [ ENCODER (RNN/LSTM/Transformer) ] ──► context / K, V states
│
Output: "कस्तो छ" ◄── [ DECODER (auto-regressive) ] ◄───────────────┘- Encoder: Reads the entire input sequence and produces contextual representations (originally a single final hidden state ).
- Decoder: Generates the output sequence one token at a time, conditioned on previously generated tokens and the encoder representation:
- Training vs. Inference: Trained with teacher forcing (feeding the ground-truth at each step); decoded at inference via greedy search or beam search (keeping top- partial hypotheses).
Attention Mechanism and Transformers
Compressing a long sentence into one fixed vector creates an information bottleneck. Attention (Bahdanau et al., 2015) lets the decoder look back at every encoder hidden state at each step via dynamic weights :
Modern Transformers (Vaswani et al., 2017) drop recurrence altogether and rely purely on Scaled Dot-Product Self-Attention:
This enables massive parallelization during training and captures long-range dependencies across thousands of tokens.
7.2 Federated learning
Traditional ML pools all user data onto a central server, creating privacy risks, regulatory barriers (GDPR/HIPAA), and high network bandwidth costs. Federated Learning (FL) trains a shared global model across many decentralized client devices (phones, hospitals, IoT nodes) without raw data ever leaving the device.
Federated Averaging (FedAvg)
In each communication round of FedAvg (McMahan et al., 2017):
- Broadcast: The central server sends the current global weights to a sampled subset of clients.
- Local training: Each client performs local SGD updates on its private dataset of size to obtain updated weights .
- Upload & Secure Aggregation: Clients upload only model updates/weights (not raw data). The server aggregates them weighted by dataset size ():
- Challenges in FL: Non-IID data across clients (each user has different habits), communication bottlenecks over wireless links, stragglers / heterogeneous hardware, and privacy leaks from gradients (mitigated via Differential Privacy and Secure Aggregation).
7.3 Edge AI
Edge AI executes AI inference (and sometimes lightweight adaptation) directly on local hardware—smartphones, cameras, drones, wearables, microcontrollers (TinyML)—rather than sending every sensor reading to cloud data centres.
| Aspect | Cloud AI | Edge AI |
|---|---|---|
| Compute location | Remote GPU/TPU data centres | On-device (NPU, DSP, MCU, mobile GPU) |
| Latency | Higher (network round-trip + queuing) | Ultra-low (real-time milliseconds) |
| Offline operation | Requires internet connectivity | Works fully offline |
| Privacy | Raw audio/video leaves the device | Data stays local on the device |
| Compute & memory | Virtually unlimited | Tightly constrained (power, RAM, storage) |
Enabling Techniques for Edge AI
- Model compression: Quantization (e.g., 32-bit floats 8-bit integers cuts memory ), weight pruning, and knowledge distillation.
- Lightweight architectures: MobileNet (depthwise separable convolutions), EfficientNet, SqueezeNet.
- Specialized edge hardware & runtimes: Apple Neural Engine, Coral Edge TPU, NVIDIA Jetson, TensorFlow Lite / ONNX Runtime.
- Applications: Autonomous driving obstacle detection, wake-word detection ("Hey..."), smart agriculture sensors, and medical wearables.
7.4 Ethics and AI: Responsible AI
As AI shapes hiring, healthcare, credit, policing, and public information, technical accuracy alone is insufficient. Responsible AI designs and governs AI systems to be fair, transparent, safe, and accountable.
| Principle | Meaning | Practical Engineering Measure |
|---|---|---|
| Fairness & Non-Discrimination | Avoid systematic harm across gender, caste, ethnicity, or age | Audit subgroup error rates; enforce demographic parity or equalized odds |
| Transparency & Explainability (XAI) | Users understand when AI is used and why a decision was made | Model cards, feature attribution (SHAP, LIME), confidence disclosures |
| Privacy & Data Governance | Protect personal data from misuse or reconstruction | Data minimization, anonymization, Differential Privacy, Federated Learning |
| Safety, Robustness & Reliability | Resist adversarial inputs, distribution shifts, and failures | Red-teaming, OOD detection, human-in-the-loop fallback |
| Accountability | Clear human/organizational responsibility for outcomes | Audit logs, impact assessments, recourse mechanisms for affected people |
Worked Out Examples
7.1
In a Seq2Seq Transformer with key dimension , a decoder query attends over two encoder keys and , with scalar values and . Compute the attention weights and the output context value .
Compute scaled dot products ():
- .
- .
Apply Softmax:
Weighted sum of values:
Attention weights: , ; output context .
7.2
Three hospitals train a shared diagnostic parameter via FedAvg without sharing patient records. Their local sample counts and updated weights after round are: Hospital A (), Hospital B (), and Hospital C (). Compute the aggregated global weight . Compare this with unweighted averaging.
Total training samples across participating clients:
Weighted FedAvg aggregation:
Compare with unweighted mean: Unweighted averaging would give , unfairly letting the smallest hospital ( of data) distort the global model.
FedAvg global weight (versus biased unweighted average ).
7.3
(a) An Edge AI model has parameters stored in 32-bit float (FP32, bytes/param). Compute its model size in MiB ( bytes) before and after 8-bit integer (INT8, byte/param) quantization. (b) A hiring model approves of applicants in Group A and of applicants in Group B. Compute the Demographic Parity Difference between the two groups.
Edge AI model size:
- FP32 ( bytes/param): .
- INT8 ( byte/param): ( smaller).
Demographic Parity Difference:
(a) FP32 size , INT8 size ( reduction); (b) Demographic Parity Difference .
Exercises
Exercise: Seq2Seq, Federated Learning, Edge AI, and Responsible AI
Explain the encoder–decoder architecture of a Seq2Seq model. Why does a single fixed context vector suffer on long sequences, and how does Attention solve this?
Encoder–Decoder: The encoder reads a variable-length input sequence into hidden representations; the decoder generates the output sequence one token at a time conditioned on prior outputs and encoder states.
Bottleneck & Attention: Compressing an entire long sentence into one fixed vector loses early details. Attention lets the decoder compute a fresh weighted sum of all encoder hidden states at every output step, focusing dynamically on relevant input words.
Seq2Seq pairs an encoder and an autoregressive decoder; attention replaces the single fixed context vector bottleneck with dynamic step-by-step weighted combinations of all encoder states.
What is Federated Learning and how does it preserve data privacy? Two devices participate in a FedAvg round with local datasets (weight ) and (weight ). Compute the new global weight.
Concept: Federated Learning trains a shared model across decentralized devices by sending model updates rather than raw user data to the central server.
FedAvg calculation ():
Federated Learning keeps raw data on-device and aggregates parameter updates centrally. Global weight .
Compare Edge AI and Cloud AI across latency, privacy, connectivity, and compute constraints. If an FP32 model is quantized from 32-bit float to 8-bit integer for an edge microcontroller, what is its new size (ignoring metadata)?
Comparison: Edge AI runs locally with millisecond latency, works offline, and keeps raw data on-device, but faces strict RAM/power limits. Cloud AI offers massive compute at the cost of network latency, internet dependence, and sending data off-device.
Quantized size: Reducing bit-width from bits ( bytes) to bits ( byte) compresses weights by a factor of :
Edge AI trades cloud compute scale for low latency, offline reliability, and local privacy. INT8 quantization shrinks the model to .
List the five core principles of Responsible AI and explain three sources of bias in AI systems. If a loan-approval system approves of urban applicants and of rural applicants, compute the Demographic Parity Difference.
Principles & bias: Core principles: Fairness, Transparency/Explainability, Privacy/Governance, Safety/Robustness, and Accountability. Bias sources: historical/sampling imbalance, biased labels/proxies, and feedback loops.
Demographic Parity Difference:
Responsible AI rests on fairness, transparency, privacy, safety, and accountability; bias stems from sampling, label proxies, and feedback loops. Demographic Parity Difference .
In a Transformer attention layer, two encoder positions produce equal scaled attention scores and with value scalars and . Compute the Softmax attention weights and the resulting context value .
Softmax weights: Since :
Context value:
, , and .