Types of Data, Preprocessing Needs & Data Collection
Open in the Padhaiko app3.1 Types of Data: Structured, Unstructured, Semi-Structured Data
In modern computational data pipelines, datasets arrive from heterogeneous sources including relational databases, sensor arrays, web logs, video streams, and natural language documents. To design appropriate storage architectures, ingestion pipelines, and machine learning models, data scientists categorize data according to its degree of internal organization and schema enforcement into three fundamental classes: structured, semi-structured, and unstructured data.
Taxonomy of Data: Structured, Semi-Structured, and Unstructured
Data representations are formally partitioned into three structural paradigms:
- Structured Data: Information whose individual elements are addressable, strongly typed, and organized according to a strict predefined conceptual schema (known as schema-on-write). It is canonically represented as two-dimensional tabular relations consisting of rows (records, observations, or entities) and columns (attributes, fields, or features).
- Examples: Relational database tables (PostgreSQL, MySQL), structured CSV files, enterprise resource planning (ERP) records, student academic registries.
- Semi-Structured Data: Information that does not adhere to the rigid two-dimensional tabular constraints of relational databases, but contains internal semantic tags, markers, keys, or hierarchical tree structures that segregate data elements into identifiable fields (known as schema-on-read). The schema is self-describing and embedded directly alongside the values.
- Examples: JSON payloads, XML documents, YAML configurations, NoSQL document-store collections (MongoDB), server access logs.
- Unstructured Data: Information that possesses no predefined data model, formal schema, or conceptual structure. Its constituent parts cannot be indexed or addressed through standard relational algebraic operations without undergoing feature extraction or semantic parsing.
- Examples: Natural language text documents (PDFs, Word files, emails), audio recordings, satellite imagery, video feeds, raw sensor telemetry.
| Structural Dimension | Structured Data | Semi-Structured Data | Unstructured Data |
|---|---|---|---|
| Schema Paradigm | Schema-on-write (rigid, predefined) | Schema-on-read (self-describing, flexible) | No schema (raw binary or character streams) |
| Underlying Data Model | Relational tables, relational algebra | Hierarchies, trees, key-value pairs, graphs | Unordered character sequences, multi-channel arrays |
| Storage Technology | Relational DBMS (RDBMS), SQL warehouses | NoSQL document/key-value stores, JSON/XML files | Data lakes, object storage (Amazon S3, HDFS) |
| Query Mechanism | Structured Query Language (SQL) | JSONPath, XPath, NoSQL query APIs | Full-text indexing, deep neural feature extractors |
| Schema Evolution | Rigorous: requires database migrations (ALTER TABLE) | Agile: new attributes added per record dynamically | Arbitrary: no schema constraints to enforce |
| Volume in Enterprise | of global data | of global data | of global data |
Converting Unstructured and Semi-Structured Data to Structured Form
Because statistical estimators, gradient-based optimizers, and linear algebra routines require numeric matrices (), unstructured and semi-structured assets must undergo feature extraction and schema flattening before downstream modeling:
- Semi-Structured (JSON / XML) Flattening:
- Path extraction: Nested keys (such as
user.address.city) are extracted into standalone column attributes (address_city). - Array unnesting (Exploding): One-to-many nested lists are normalized into relational records with foreign key linkages.
- Path extraction: Nested keys (such as
- Unstructured Text Parsing:
- Text corpus tokens are converted into vector spaces via Tokenization, Stopword Filtering, Lemmatization, Term Frequency-Inverse Document Frequency (), or dense transformer-based embeddings (e.g., Word2Vec, BERT).
- Unstructured Audio and Image Processing:
- Images are decoded into numerical tensors of shape (height, width, color channels) or transformed into convolutional spatial embeddings.
- Audio waveforms are converted via Discrete Fourier Transforms into frequency-time spectrograms or Mel-Frequency Cepstral Coefficients (MFCCs).
Extracting Relational Attributes from Semi-Structured Document
Transform the semi-structured session object into a flattened relational schema and extract relevant feature attributes.
Decompose nested dictionary hierarchies into first-normal-form attributes: We map each nested key path to a distinct top-level feature name using dot-notation flattening:
sessionIdsession_id(String categorical identifier)client.deviceclient_device(Categorical feature:"mobile")client.osclient_os(Categorical feature:"Android")metrics.durationSecduration_seconds(Continuous numerical feature: )totalValuetotal_spend(Continuous numerical feature: )
Feature-engineer nested array entities: The nested array attribute metrics.actions = ["click", "search", "purchase"] represents sequential events. In a flattened relational table, arrays cannot reside in a scalar cell without violating first normal form (). We extract structured summary indicators:
action_count- Binary flags (one-hot encoding):
has_purchase,has_search,has_click.
Tabular representation: The final structured row is represented as:
| session_id | client_device | client_os | duration_seconds | action_count | has_purchase | total_spend |
|---|---|---|---|---|---|---|
| s-8109 | mobile | Android | 420 | 3 | 1 | 1250.50 |
The semi-structured JSON document is transformed into a flat relational schema by converting nested hierarchy paths into explicit column headers (client_device, client_os, duration_seconds, total_spend) and unnesting arrays into structured quantitative metrics (action_count = 3, has_purchase = 1).
3.2 Why Preprocess Data?
In machine learning and data science, raw observational data collected from real-world sensors, web scraping, surveys, or database logs is inherently imperfect. It is frequently contaminated with missing values, recording noise, extreme outliers, duplicate entries, typographical errors, and inconsistent measurement units.
Data Preprocessing and the GIGO Principle
Data Preprocessing is the systematic discipline of identifying, resolving, and transforming raw, imperfect data into a clean, coherent, normalized, and analytically sound format before model training.
The foundational imperative for data preprocessing is the Garbage In, Garbage Out (GIGO) theorem: no machine learning algorithm—regardless of its complexity, parameter count, or computational capacity—can extract accurate, generalizable insights from corrupt, contaminated, or improperly structured input data.
Core Objectives of Data Preprocessing
- Resolving Incomplete and Missing Data: Many predictive algorithms (such as linear regression, logistic regression, support vector machines, and feedforward neural networks) cannot mathematically evaluate equations containing missing values (
NaNornull). Imputation or controlled deletion guarantees numerical completeness. - Eliminating Noise and Outliers: Sensor glitches, transmission bugs, or manual data entry errors produce extreme anomalies that distort sample parameters (e.g., sample mean and sample variance ), pulling gradient descent away from true optima.
- Harmonizing Disparate Scales: Features recorded in wildly different measurement ranges (e.g., customer income spanning to versus customer age spanning to ) cause gradient-based algorithms to oscillate inefficiently and distance-based classifiers (-NN, -Means) to become dominated solely by large-magnitude features.
- Encoding Categorical Signals: Mathematical models execute linear combinations of vectors (). Categorical attributes (e.g.,
"City" \in \{\text{"Kathmandu"}, \text{"Pokhara"}, \text{"Biratnagar"}\}) must be converted into numerical indicators via one-hot or target encoding. - Mitigating Algorithmic Bias and Leakage: Unchecked imbalances, duplicate observations, or historical reporting skews cause models to internalize spurious correlations, producing discriminatory or unstable predictions.
Consequences of Omitting Preprocessing
Skipping preprocessing in a data science project triggers compounding operational failure modes:
- Catastrophic Model Crashes: Numerical linear algebra libraries throw unhandled exceptions when encountering non-numeric tokens or missing elements during matrix inversions.
- Distorted Optimization Landscapes: Loss surfaces with unscaled features form elongated, poorly conditioned ellipses where gradient vectors point nearly orthogonal to the minimum, resulting in slow convergence or gradient explosion.
- Spurious Generalization: Models memorize duplicate records or artifactual noise, achieving artificially high training performance while failing completely on unseen test distributions.
- Wasted Computational Budget: Feeding high-cardinality, uncompressed, or redundant features inflates memory footprint and prolongs GPU/CPU compute cycles unnecessarily.
Quantifying the Impact of Feature Preprocessing on Distance Metrics
Compute the raw Euclidean distance and the normalized Euclidean distance across Salary and Age. Demonstrate how failure to scale features causes one attribute to completely drown out another.
Compute raw Euclidean distance: The standard Euclidean distance metric between vectors and is:
Substitute the raw values:
The contribution of the age difference () accounts for a negligible fraction of the squared distance:
The raw distance metric is entirely dominated by salary, rendering the 25-year age difference practically invisible to distance-based algorithms such as -nearest neighbors or -means clustering.
Compute min-max normalized coordinates: The bounds for Salary are and , with range . The bounds for Age are and , with range .
Using min-max normalization :
- Customer :
- Customer :
Calculate normalized Euclidean distance: Substitute the scaled coordinates into the metric:
Here, the salary squared difference is and the age squared difference is . Both features now contribute meaningfully and proportionately to the distance metric.
The raw Euclidean distance is , where the age attribute accounts for less than of the squared difference. After min-max normalization to , the distance is , restoring balanced sensitivity across both feature dimensions.
3.3 Data Collection Strategies
Data collection is the purposeful gathering and measuring of targeted variables from identified repositories or environments. Data science practitioners classify collection methods into two foundational modalities: primary data collection and secondary data collection.
Data Collection Paradigms
- Primary Data Collection: The first-hand acquisition of original, bespoke data gathered directly by the investigator specifically for the empirical investigation at hand.
- Mechanisms: Direct physical instrumentation, specialized surveys, controlled lab experiments, live user interviews, field telemetry.
- Secondary Data Collection: The retrieval, ingestion, and repurposing of existing data originally compiled, curated, or archived by an outside entity or for an earlier independent purpose.
- Mechanisms: Public government portals, academic open-source repositories, published census datasets, financial market ticker archives, commercial data brokers.
| Aspect | Primary Data Collection | Secondary Data Collection |
|---|---|---|
| Origin & Provenance | Collected first-hand directly from the primary source | Gathered second-hand from existing published repositories |
| Specificity & Alignment | Tailored precisely to the research question and hypothesis | May not align perfectly with the target operational variables |
| Cost & Time Expenditure | High expenditure: requires custom infrastructure and surveys | Low cost and rapid retrieval via APIs or file downloads |
| Data Recency | Real-time or highly contemporary | Historical; subject to temporal obsolescence |
| Control over Quality | Direct control over measurement precision and sampling | Dependent on the documentation and rigor of third parties |
| Canonical Examples | Custom IoT sensor readings, active clinical trial logs | World Bank reports, Kaggle archives, statutory filings |
Five Ubiquitous Sources of Data in Data Science
- Transactional Enterprise Databases: Relational SQL systems (PostgreSQL, Oracle) storing continuous business operations (sales orders, inventory updates, payroll, ledger logs).
- Web Scraping and Web Services (REST APIs): Structured web services (JSON/REST APIs) and automated web crawlers extracting market competitor pricing, social media trends, or news sentiment.
- Sensors and Internet of Things (IoT) Telemetry: Physical hardware endpoints recording spatial, environmental, or mechanical variables (accelerometers, GPS trackers, temperature probes, industrial SCADA systems).
- Surveys, Questionnaires, and Human Feedback: Structured instruments collecting customer sentiment, patient medical histories, user preference feedback (RLHF in generative models), and demographic records.
- Open Data Repositories and Public Archives: Government, academic, and non-profit open-access portals (data.gov, TU institutional repositories, WHO global health datasets, Kaggle open datasets).
Sampling Strategies in Data Collection
When collecting primary data or selecting representative subsets from big-data repositories, extracting the entire population is often computationally or economically infeasible. Practitioners utilize rigorous statistical sampling:
- Simple Random Sampling (SRS): Every item in population has an identical probability of selection.
- Stratified Sampling: The population is segregated into mutually exclusive, homogeneous subgroups (strata) based on a key characteristic (e.g., gender, geographic region, academic department), and samples are drawn proportionally from each stratum:
where is stratum size, is total population, and is target sample size.
- Systematic Sampling: Selecting every -th item from an ordered list after a random start between and .
- Cluster Sampling: Dividing the population into heterogeneous clusters (e.g., distinct colleges), randomly selecting a subset of clusters, and surveying all elements within chosen clusters.
Exercises: Data Types & Collection
Define structured, semi-structured, and unstructured data. Provide one concrete real-world engineering or software example of each.
Define Structured Data with example: Structured data is highly organized information that conforms to a rigid, predefined data model and schema (schema-on-write). Every record resides in fixed columns with explicitly enforced data types (integers, floats, dates, strings), enabling efficient relational algebraic querying via SQL. Example: A university student examination table in an RDBMS containing columns (student_id INT, full_name VARCHAR, credits_completed INT, gpa FLOAT).
Define Semi-Structured Data with example: Semi-structured data lacks the rigid tabular format of relational databases but contains internal organizational tags, keys, or hierarchical markers (schema-on-read) that segregate fields. Different records may contain different optional fields without breaking global schema integrity. Example: A web service API returning a customer order record in JSON format with nested key-value objects and variable-length item arrays.
Define Unstructured Data with example: Unstructured data has no predefined conceptual schema, tabular structure, or semantic markup. It consists of raw, continuous sequences of characters, pixels, or waveforms that cannot be queried using relational operations without explicit feature extraction. Example: A collection of scanned PDF textbook pages, video surveillance footage, or free-form doctor clinical diagnosis notes.
- Structured Data: Information adhering to a strict tabular schema with typed columns (e.g., student registry in a MySQL database).
- Semi-Structured Data: Self-describing data with internal keys or tags but no rigid table constraints (e.g., JSON response payload from a REST API).
- Unstructured Data: Data lacking any conceptual data model or schema (e.g., audio recording of a lecture or raw satellite imagery).
Explain how unstructured data (such as raw text or image files) can be converted into structured tabular form for machine learning analysis.
Core conceptual paradigm: Converting unstructured data into structured tabular form is termed feature extraction or representation learning. The objective is to map each high-dimensional, irregular raw object into a fixed-length numeric feature vector , forming an design matrix where rows represent samples and columns represent features.
Techniques for Unstructured Text Data:
- Lexical preprocessing: Tokenize character strings into words, lowercase all characters, remove punctuation and stopwords (e.g., "the", "is"), and apply lemmatization or stemming to reduce words to root forms.
- Vector space modeling:
- Bag-of-Words (BoW): Construct a global vocabulary of size ; represent each document as a vector where each coordinate counts word occurrences.
- TF-IDF: Scale counts by Inverse Document Frequency to downweight ubiquitous terms:
- Dense Embeddings: Map sentences to low-dimensional latent vectors using neural language encoders (Word2Vec, BERT).
Techniques for Unstructured Image Data:
- Grid rasterization: Convert image pixels into a numeric tensor of shape .
- Feature extraction: Compute handcrafted spatial descriptors (Histogram of Oriented Gradients [HOG], SIFT) or pass the tensor through a pre-trained Convolutional Neural Network (CNN) backbone to extract the penultimate dense layer vector .
- Tabular assembly: Place the extracted 512 embedding coefficients into tabular columns labeled , with the target label in column .
Unstructured data is converted to structured form through feature extraction pipelines. Text is tokenized, filtered, and transformed into numeric vectors via TF-IDF or transformer embeddings; images are rasterized and converted to dense feature vectors via convolutional networks. The resulting fixed-length vectors form the columns of a standard two-dimensional design matrix.
Explain the fundamental importance of data preprocessing before training machine learning models.
Algorithmic and mathematical necessity: Machine learning models are mathematical optimization algorithms that compute matrix multiplications, inner products, and distance metrics. If raw data contains missing entries (NaN), unencoded text strings, or extreme recording corruptions, standard optimization functions cannot compute gradients or loss values, resulting in training crashes or invalid numerical states.
Guaranteed performance and convergence:
- Handling missing values: Imputation or controlled deletion prevents computational execution errors and avoids throwing away valuable complete features.
- Scale equalization: Feature normalization ensures optimization surfaces are isotropic (spherical) rather than elongated, allowing gradient descent to converge rapidly without oscillation.
- Noise and outlier dampening: Eliminating measurement artifacts prevents the optimizer from fitting spurious extreme errors, improving model generalization on unseen data.
- Encoding categorical signals: Transforming categorical attributes into numeric indicators enables linear and non-linear estimators to incorporate non-numerical domain attributes.
Prevention of Garbage In, Garbage Out (GIGO): A predictive model trained on corrupted, biased, or inconsistent data simply learns the patterns of corruption. High-quality preprocessing ensures that the model captures authentic statistical signals rather than data-entry anomalies.
Data preprocessing is essential because machine learning algorithms require clean, consistently formatted numerical matrices. Preprocessing prevents computational crashes from missing data, ensures isotropic optimization landscapes for fast gradient descent convergence, eliminates distortion from recording noise and outliers, and enforces the GIGO principle so models learn genuine underlying relationships.
Identify three common data quality issues that preprocessing addresses. Describe the consequences of skipping preprocessing in a data science project.
Three pervasive data quality issues:
- Missing Data: Incomplete records resulting from sensor power loss, survey non-responses, or transmission packet drops (e.g., blank medical test records).
- Noise and Outliers: Extreme corruptions or recording artifacts that do not reflect true phenomena (e.g., human age recorded as years or temperature sensor spiking to ).
- Inconsistent Formatting and Schema Heterogeneity: Divergent date representations (
"2026-11-07"vs"07/11/2026"), mixed casing ("Kathmandu","KATHMANDU","ktm"), and conflicting units (meters vs feet).
Consequences of skipping preprocessing:
- Algorithmic Failure: Libraries such as Scikit-Learn or PyTorch throw runtime exceptions when encountering
NaNvalues or non-numerical strings. - Severe Prediction Degradation: Outliers excessively inflate quadratic loss functions (), dragging regression lines toward corrupted anomalies.
- Distorted Convergence: Gradient descent oscillates wildly across poorly scaled loss surfaces, requiring excessive training epochs or failing to converge.
- Flawed Business Decisions: Duplicate and misaligned records produce biased aggregations, leading stakeholders to make decisions based on false analytical conclusions.
Three common data quality issues are missing values, recording noise/outliers, and inconsistent formatting/duplicates. Skipping preprocessing leads to runtime algorithm crashes, slow or diverging optimization, severe model prediction error due to outlier leverage, and flawed analytical conclusions.
Provide a concrete real-world scenario demonstrating how preprocessing improves model performance. List five common sources of data used in data science.
Scenario: Automated Real Estate Valuation Model: Suppose an engineering team trains a gradient-boosted regression model to predict residential house prices based on raw municipal listings:
- Raw Issue 1: Property area is recorded inconsistently (some listings in square feet, others in square meters). Preprocessing converts all measurements into standard square meters.
- Raw Issue 2: The bedroom count column has missing values where agents left the field empty. Preprocessing imputes missing entries with the neighborhood median.
- Raw Issue 3: Living area ranges from to , while bathroom count ranges from to . Min-max scaling normalizes both to , preventing the living area from overshadowing the bathroom count in distance calculations.
- Raw Issue 4: An anomalous listing records a sale price of (foreclosure transfer). Outlier removal filters this non-market record, preventing significant distortion of the regression line.
Following preprocessing, the model's Mean Absolute Error (MAE) drops significantly, and gradient descent converges in fewer iterations.
Five common sources of data in data science:
- Transactional Relational Databases (SQL databases of customer sales and billing).
- Web APIs and Web Scraping (REST endpoints, e-commerce catalog scraping).
- Physical Sensors and IoT Telemetry (smart grid meters, environmental monitoring probes).
- Surveys and Questionnaires (customer satisfaction forms, public opinion polls).
- Public and Open Government Portals (census archives, TU open research repositories).
In a real estate pricing pipeline, preprocessing resolves mixed measurement units (sq ft vs sq m), imputes missing bedroom counts, scales living area and bathroom features, and removes zero-dollar foreclosure outliers, yielding stable convergence and reduced prediction error. Five common sources are transactional databases, web APIs/scraping, IoT sensors, surveys, and public open data repositories.
Differentiate between primary and secondary data collection across at least four distinct comparative dimensions.
Comparative analysis: We evaluate primary and secondary data collection across operational dimensions:
| Dimension | Primary Data Collection | Secondary Data Collection |
|---|---|---|
| Data Nature | Original, bespoke data collected first-hand for the specific active study | Pre-existing data collected earlier by another party for a different purpose |
| Collection Cost & Time | High monetary cost; labor-intensive and slow collection cycle | Economical; rapid ingestion via downloads, databases, or public APIs |
| Relevance & Precision | Perfectly tailored to the project's exact variable definitions and constraints | May contain irrelevant variables or lack key domain attributes |
| Quality Control | Complete ownership over sampling design, instrumentation, and measurement accuracy | Dependent on documentation, integrity, and potential hidden biases of the publisher |
| Recency | Real-time, up-to-date observations | Historical; potential temporal drift or obsolescence |
Primary data is gathered first-hand specifically for the study (custom surveys, sensor probes), offering perfect domain relevance and direct quality control at the expense of high cost and time. Secondary data is acquired from existing repositories (government censuses, research databases), providing low-cost, immediate access but risking misaligned variables and unknown collection errors.
A data science team needs to collect a representative sample of students from an engineering college population of to evaluate mobile study app performance. The population is partitioned into three distinct academic programs:
- Computer Engineering: students
- Electronics & Information Engineering: students
- Civil Engineering: students
Using proportional stratified sampling, compute the exact sample size allocation for each stratum.
Formulate proportional allocation formula: In proportional stratified sampling, the sample size allocated to stratum is strictly proportional to the fraction of the total population represented by that stratum:
where:
- Total population
- Total desired sample size
Compute stratum allocations:
- Computer Engineering stratum ():
- Electronics Engineering stratum ():
- Civil Engineering stratum ():
Verify total sample constraint: Sum the stratum allocations:
The allocation exactly matches the required sample size of without rounding distortion.
Under proportional stratified sampling, the allocated sample sizes are:
satisfying .