PadhaikoAll subjects

Implementation and Testing

Project II (ENCT 461)

Open in the Padhaiko app

3.1 Implementation and deployment of the project

Implementation in Project II transforms your detailed architecture into a working, integrated, reproducible, and deployable system (Product Track) or a complete, repeatable experimental pipeline (Research Track). Unlike early seventh-semester scripts, eighth-semester implementation must handle real error paths, clean configuration, and consistent environment packaging.

3.1.1 Incremental Implementation Sequence

Rather than coding isolated UI pages or standalone notebooks until the last week, build vertically in thin end-to-end slices:

  1. Core Data & Environment Setup: Initialize the database migrations, dataset loaders, or hardware firmware skeleton with pinned dependency manifests (requirements.txt, package.json, platformio.ini) and .env.example templates (never commit API keys or database passwords to Git).
  2. Core Business Logic / Model Pipeline: Implement domain services, REST/WebSocket handlers, or training/evaluation scripts with structured error handling, meaningful status codes, and rotating log files.
  3. Interface & Hardware Integration: Connect frontend screens or physical sensors/actuators incrementally to real backend endpoints, replacing static mock JSON early.
  4. Hardening & Packaging: Add authentication guards, input sanitization, graceful offline/timeout fallbacks, and one-command startup scripts (docker compose up or single-command build/flash).

3.1.2 Deployment Strategies Across Project Tracks

"Deployment" means placing the completed system into an operational environment where target users or evaluators can actually run and verify it.

Deployment Patterns for B.E. Computer Engineering Projects
Project TypePractical Deployment TargetKey Deployment Checklist
Web / SaaS PlatformDocker Compose on VPS/cloud VM or local campus server behind Nginx reverse proxyHTTPS/TLS, environment variables, database backup script, health-check endpoint (/health)
Mobile ApplicationSigned Android APK installed on physical test devices + hosted backend APIOffline state handling, API base-URL configuration, permission prompts, crash logging
ML / Research PipelineReproducible CLI + lightweight demo web UI (Gradio/FastAPI) or documented inference packageFixed random seeds, saved model checkpoints, eval.py script reproducing thesis tables
Embedded / IoT HybridFlashed microcontroller in protective enclosure + local gateway or cloud telemetry serverWatchdog timer enabled, brownout recovery, Wi-Fi/MQTT auto-reconnect, calibration constants

3.2 Testing and validation of the developed system (unit testing, integration testing, and user testing)

Testing proves that your implementation works as specified; validation proves that the system actually solves the real-world problem for its intended users or research hypothesis. ENCT 461 explicitly requires three complementary testing layers: unit testing, integration testing, and user testing.

Definition

Unit, Integration, and User Testing

3.2.1 Designing a Rigorous Test Case Matrix

For Chapter 5 of your final report, examiners expect structured test evidence rather than a single sentence claiming "everything worked." Document test IDs, preconditions, inputs, expected outputs, actual results, and pass/fail status across all three layers.

Test Layers, Scope, Tools, and Evidence Artifacts
Test LayerWhat Is VerifiedTypical Tools / MethodsEvidence for Final Report
Unit TestingPure functions, schema validators, auth helpers, feature extractors, sensor parserspytest, unittest, Jest, JUnit, Unity (C/embedded)Automated test runner output + pass count and edge cases
Integration TestingAuth + DB CRUD flows, MQTT-to-DB pipeline, edge-to-cloud sync, inference APITestClient / Supertest, Postman/Newman, Docker test DB, hardware-in-the-loopIntegration test matrix with HTTP codes, payloads, and failure recovery checks
User Testing (UAT)Task completion rate, error recovery by non-authors, UI clarity, field usabilityGuided task sheet (5–15 target users) + System Usability Scale (SUS) surveyTask-success table, SUS score summary, qualitative feedback & UI fixes made
Example Unit and Integration Test Structure (pytest)Python
def test_claim_amount_rejects_negative_and_zero():
    """Unit test: boundary check on validation helper."""
    assert validate_dosage_ml(0.5) is True
    assert validate_dosage_ml(0.0) is False
    assert validate_dosage_ml(-2.0) is False

def test_telemetry_ingest_persists_to_db(test_client, db_session):
    """Integration test: API endpoint validates, stores, and retrieves sensor packet."""
    payload = {"device_id": "node-01", "temp_c": 4.2, "humidity": 61.5}
    res = test_client.post("/api/v1/telemetry", json=payload)
    assert res.status_code == 201
    fetched = test_client.get("/api/v1/telemetry/node-01/latest")
    assert fetched.status_code == 200
    assert fetched.json()["temp_c"] == 4.2

3.2.2 Conducting Credible User Testing (UAT)

Even a strong technical backend fails user validation if real operators find the workflow confusing:

  1. Recruit 8 to 20 representative users outside your three-member project group (students, clinicians, farmers, teachers, or company staff matching your target domain).
  2. Define 4 to 6 concrete tasks (without coaching users step-by-step on where to click) and record completion success, time taken, and points of confusion.
  3. Collect structured feedback using a brief 5-point Likert questionnaire or the standard 10-item System Usability Scale (SUS), then fix the top usability friction points and log the before/after improvement.

3.3 Performance evaluation and improvement

The ENCT 461 syllabus emphasizes systematic performance evaluation, critical comparison against existing solutions or baseline methods, and iterative improvement based on findings.

3.3.1 Choosing Appropriate Evaluation Metrics

Select quantitative metrics that genuinely reflect your system's goal—never cherry-pick a misleading metric (such as raw accuracy on an imbalanced 95:5 dataset).

AspectProduct-Oriented Evaluation MetricsResearch-Oriented Evaluation Metrics
Primary Quality MetricsTask success rate, uptime/reliability, error rate, SUS usability scoreMacro-F1, Precision/Recall, PR-AUC / ROC-AUC, RMSE/MAE, BLEU/WER, IoU/mAP
Speed & Resource MetricsAPI latency (p50, p95, p99 in ms), throughput (req/s), memory/CPU, battery lifeInference latency (ms/sample), FLOPs/parameter count, training time, VRAM footprint
Comparison StandardLegacy manual workflow, existing open-source/commercial tool, or unoptimized v1 buildPublished baseline models, standard heuristics, and ablation variants of your own method
Failure / Error AnalysisLoad-test saturation point, offline-sync recovery time, security scan findingsConfusion matrix, worst-performing subclasses, noise/domain-shift sensitivity

3.3.2 Baseline Comparison, Ablation, and Iterative Improvement Loop

  1. Benchmark a clean baseline first: Measure your initial version or standard reference baseline on the exact same test split, hardware, or workload.
  2. Profile to locate the true bottleneck: Never guess what is slow or inaccurate. Use database EXPLAIN ANALYZE, Python cProfile, browser DevTools Network/Performance tabs, or per-class confusion matrices to find the dominant bottleneck.
  3. Apply targeted improvements one variable at a time:
    • Database/Backend: Add composite indexes, eliminate N+1 queries, introduce Redis/in-memory caching, or move heavy exports to async background jobs.
    • ML/Edge Inference: Apply class-weighted loss or focal loss, domain-specific data augmentation, hyperparameter tuning, or ONNX / INT8 post-training quantization.
    • Embedded/Hardware: Replace blocking delay() calls with timer interrupts, add digital low-pass/median filtering for sensor spikes, or optimize deep-sleep duty cycles.
  4. Re-measure and document the delta: Report the exact before-and-after numbers under identical test conditions (for example, p95 query latency reduced from 480 ms to 32 ms, or macro-F1 improved from 0.71 to 0.84 after adding temporal features).
Example 3.1

Designing a Three-Layer Test Matrix for a Product-Track Web & Mobile System

A group has implemented a Clinic Appointment & Prescription Triage Web/Mobile System with a React/Android frontend, FastAPI backend, and PostgreSQL database. Design a structured test plan covering unit testing, integration testing, and user testing (UAT) that produces defensible Chapter 5 evidence.

Define Unit Tests (isolated logic and boundary validation):

  • Test slot-generation logic for overlapping doctor schedules, lunch breaks, and invalid past timestamps.
  • Test role-permission guards (ensuring a patient token cannot query another patient's prescription ID) and password-hashing/JWT expiry helpers.

Define Integration Tests (end-to-end service and DB contracts):

  • Run an ephemeral PostgreSQL test container with pytest: verify POST /appointments creates a booking inside an ACID transaction and prevents double-booking the same slot under two concurrent requests.
  • Verify offline-queued clinic sync: submit two offline prescriptions from the Android client, restore connectivity, and confirm idempotency keys prevent duplicate records.

Design User Acceptance Testing (UAT) with real non-team participants:

  • Recruit 10 volunteers (6 acting as patients, 4 acting as reception/clinic staff) to execute 5 scripted tasks (register, book next available slot, reschedule, view prescription QR, cancel).
  • Measure task completion rate, average time per task, and SUS questionnaire score; record that 3 of 6 patients initially missed the "Reschedule" button, then move it to the primary appointment card and re-verify.
Answer:

The group pairs automated pytest unit tests (slot rules, RBAC), containerized transactional integration tests (concurrency and offline sync idempotency), and a 10-user task-based UAT with SUS scoring and documented UI fixes.

Example 3.2

Conducting Baseline Comparison, Ablation, and Performance Improvement (Research Track)

A Research-Track group built an Acoustic Cough & Breathing Classifier for Early Respiratory Screening on an imbalanced dataset (70% healthy, 20% mild, 10% severe). Their initial CNN baseline on raw MFCCs reached 79% overall accuracy, yet examiners note that overall accuracy hides poor detection of the minority severe class. Outline a systematic performance evaluation and improvement workflow.

Replace misleading overall accuracy with class-sensitive evaluation metrics: Compute macro-averaged F1, per-class Precision and Recall, a 3x3 confusion matrix, and one-vs-rest PR-AUC under 5-fold subject-independent cross-validation (ensuring recordings from the same patient never split across train and test folds).

Run error analysis and diagnose the bottleneck: Inspecting the confusion matrix reveals that the baseline achieves 0.91 recall on Healthy but only 0.42 recall on Severe because severe samples represent just 10% of the training batches and short silence segments dilute MFCC averages.

Apply controlled improvements via an ablation study:

  1. Baseline: Raw MFCC + 2D-CNN + standard cross-entropy.
  2. Ablation A (+Class-Weighted Focal Loss): Penalizes missed minority Severe cases.
  3. Ablation B (+Log-Mel Spectrogram + Delta Features + SpecAugment): Preserves fine spectral/temporal dynamics.
  4. Ablation C (+Pretrained Audio Backbone Fine-Tuning / Lightweight Separable CNN): Tests representation capacity and measures CPU/edge inference latency (ms per clip).

Document exact macro-F1, Severe-class recall, and inference latency across all four rows on identical test folds.

Answer:

The group switches from raw accuracy to subject-independent macro-F1 and per-class recall, diagnoses minority-class collapse via a confusion matrix, and executes a step-by-step ablation study isolating focal loss, log-Mel/SpecAugment features, and lightweight architecture trade-offs.

Viva-Style Questions: Implementation and Testing

Question 1

What is the difference between unit testing, integration testing, and user testing, and why are all three needed in Project II?

Roles of the three testing layers:

  • Unit testing checks individual functions or components in isolation (fast, catches logic/edge-case bugs early).
  • Integration testing checks data flow and contracts between connected modules (DB queries, APIs, sensor buses, model pipelines).
  • User testing (UAT) checks whether real target users can complete actual tasks comfortably without confusion or workflow breakdowns.

Why none can replace the others: Passing all unit tests does not guarantee two modules agree on JSON field names or database transactions (integration bug); passing integration tests does not guarantee real users understand the interface or trust the system's workflow (usability/validation gap).

Answer:

Unit tests verify isolated functions, integration tests verify inter-module contracts and data pipelines, and user tests validate real-world usability and workflow fit; all three together prove both technical correctness and practical usefulness.

Question 2

What does "reproducible implementation" mean for both a Product-Track software project and a Research-Track ML project?

For Product Track: Anyone (teammates or evaluators) can clone the Git repository, follow a documented setup using pinned dependency files or Docker Compose, seed test data, and run the exact same working application without hidden local files or manual database tweaks.

For Research Track: Dataset split indices, preprocessing scripts, hyperparameters, and random seeds are fixed and versioned so running the evaluation script reproduces the reported metrics, confusion matrices, and comparison tables consistently.

Answer:

Reproducibility means pinned environments, versioned configuration/seeds, and documented setup/evaluation scripts so a clean machine reproduces the exact running system (Product) or reported benchmark tables (Research).

Question 3

Why is reporting only "accuracy" or "average response time" often criticized during ENCT 461 performance evaluation?

Hidden failure modes in averages:

  • On imbalanced datasets (e.g., 92% normal, 8% fault), a trivial classifier predicting only "normal" gets 92% accuracy while failing completely on the critical minority class.
  • On backend APIs, a low average latency (e.g., 45 ms) can hide severe tail latency (p95/p99 spikes of 3,000 ms under concurrent load or cache misses).

Better evaluation practice: Pair summary metrics with macro-F1 / per-class recall / confusion matrices for ML tasks, and p50/p95/p99 latency plus concurrent throughput for software systems.

Answer:

Single averages mask minority-class failure on imbalanced data and tail-latency spikes under load; pairing macro-F1/per-class recall or p95/p99 latency reveals true system behavior.

Question 4

What is an ablation study, and how does it strengthen Topic 3.3 (Performance evaluation and improvement)?

Definition: An ablation study systematically removes, swaps, or modifies one component at a time (such as a preprocessing step, loss term, feature group, or caching layer) while holding all other settings constant, and measures the resulting change in performance.

Value in Project II: It provides scientific proof of why your proposed improvement works—separating genuine contributions from accidental complexity—and directly answers examiner questions such as "Did module X really help, or would a simpler baseline perform just as well?"

Answer:

An ablation study isolates one component at a time under identical conditions to prove quantitatively which design choices actually drive performance gains versus added complexity.

Question 5

How should a team handle negative or disappointing test results discovered during Topic 3.3 evaluation (for instance, a model performing worse under low light or an endpoint slowing down under 100 concurrent users)?

Diagnose and iterate honestly: Document the failure honestly as a finding, profile or inspect error cases to identify the root cause (e.g., lack of low-light training samples or missing DB connection pooling), apply a targeted engineering fix, and re-measure on the same benchmark.

Report remaining limits transparently: If a constraint remains (for example, hardware memory caps batch size or extreme darkness still degrades accuracy below threshold), state it clearly in Scope/Limitations rather than hiding or fabricating numbers—evaluators reward rigorous failure analysis far more than suspiciously perfect claims.

Answer:

Log the failure honestly, profile the root cause, apply and benchmark a targeted improvement, and document any remaining boundary conditions transparently in Limitations rather than hiding unfavorable cases.

← Design, Development and Project ManagementProgress Report Submission and Presentation II →