Data Cleaning Fundamentals For Interviews: Missing Values, Outliers And Duplicates

The aforesaid income dashboard tin show 2 wholly different stories earlier and aft cleaning: 1 shows a blockbuster month, the different reveals copy orders, blank pincodes and 1 intolerable transaction doing each the drama. Data cleaning is the quiet measurement that decides whether study becomes penetration aliases confidently presented nonsense.

Data cleaning intends detecting and fixing inaccurate, incomplete, copy aliases inconsistent records earlier analysis.
For missing values, first inquire why they are missing: random absence, process failure, aliases meaningful non-response.
For outliers, ne'er delete blindly. Classify them arsenic information error, uncommon event, aliases valid business signal.
For duplicates, separate nonstop copy rows from fuzzy copy entities for illustration “R. Sharma” and “Rahul Sharma”.
The safest cleaning series is: floor plan information - diagnose issues - dainty defects - validate results - archive decisions.
Use metrics for illustration completeness, uniqueness, validity, consistency, copy complaint and outlier reappraisal complaint to beryllium cleaning quality.
The interview-winning line: “I do not cleanable mechanically; I sphere business meaning while improving information reliability.”

The Big Picture: Cleaning Is Decision-Making, Not Housekeeping

Think of information cleaning arsenic a controlled conversion of earthy operational information into analysis-ready data. The extremity is not to make the dataset look neat; the extremity is to make judge each translator is defensible, reproducible and aligned pinch the business question.

Data cleaning workflow A five-stage workflow from earthy information to documented analysis-ready data. Raw Data messy input Profile scan defects Diagnose find cause Treat fix safely Validate and Document prove what changed iterate if validation fails Good cleaning is simply a loop: hole the data, past beryllium the hole did not harm the business meaning.

The Three Data Defects You Must Recognise

Most cleaning conversations trim to 3 defects: missing values, outliers and duplicates. Each has a different risk. Missing values tin bias your sample, outliers tin distort averages and models, and duplicates tin inflate counts, revenue, customers aliases conversions.

Bad cleaning versus bully cleaning A two-sided comparison showing mechanical cleaning versus business-aware cleaning. Bad Cleaning Good Cleaning Deletes messy rows Uses 1 norm for all Ignores guidelines cause No audit trail Diagnoses first Chooses by context Protects signal Documents changes The quality is judgment. The champion analysts do not conscionable region dirt; they protect awesome while reducing noise.

A Five-Step Cleaning Process You Can Apply successful Any Dataset

Use this building erstwhile you person a earthy dataset, whether it is simply a customer file, run report, indebtedness dataset aliases operations dump.

Missing Values: Diagnose Before You Impute

A missing worth is not automatically a mistake. A blank section tin mean “not applicable”, “customer refused”, “system failed”, “not captured yet” aliases “unknown”. Those meanings require different treatments.

There are 3 useful missingness patterns:

Missing wholly astatine random: the absence is unrelated to observed aliases unobserved data. Example: a random upload failure.
Missing astatine random: missingness is related to observed variables. Example: income is much often missing for self-employed applicants.
Missing not astatine random: missingness is related to the missing worth itself. Example: high-income customers whitethorn debar revealing income.

In speedy commerce, missing pincodes, incomplete landmarks aliases inconsistent flat names tin straight impact transportation duty and ETA accuracy. The superior driver of amended cleaning is standardising reside fields astatine capture; supporting drivers see pincode validation, GPS checks, rider feedback and customer prompts. The strategical truthful what: cleaning is not back-office hygiene; it improves operational reliability and customer experience.

Common treatments: delete rows only erstwhile the missing stock is mini and unbiased; usage mean aliases median imputation for elemental numeric fields; usage mode imputation for categorical fields; adhd a missingness emblem erstwhile absence itself whitethorn transportation information; aliases cod the information again if the section is business-critical.

Outliers: Separate Error, Rare Event and Business Signal

An outlier is an study that lies unusually acold from the remainder of the data. But “unusual” is not the aforesaid arsenic “wrong”. A ₹0 bid worth whitethorn beryllium a strategy error; a very ample bid whitethorn beryllium a firm buyer; a abrupt transportation hold whitethorn uncover a way disruption.

Outlier determination framework A determination character for classifying outliers arsenic error, uncommon arena aliases business signal. Extreme Value Found Can it beryllium verified? Data Error correct aliases remove Rare Event retain and explain Business Signal segment aliases model An outlier curen is correct only aft you cognize whether the outlier is false, uncommon aliases meaningful.

Worked illustration - IQR method: Suppose bid values are ₹100, ₹120, ₹130, ₹140, ₹150, ₹160, ₹180 and ₹1,500. Q1 is ₹125, Q3 is ₹170, truthful IQR = ₹45. The precocious obstruction is Q3 + 1.5 × IQR = ₹170 + ₹67.5 = ₹237.5. The ₹1,500 bid is statistically flagged, but you still verify it earlier removing it. It whitethorn beryllium a bulk order, not an error.

Duplicates: Entity Resolution, Not Just Deleting Rows

A copy tin beryllium elemental aliases amazingly hard. Exact duplicates person the aforesaid values crossed rows. Fuzzy duplicates mention to the aforesaid real-world entity but look differently: “Amit K.”, “Amit Kumar”, 2 telephone numbers, 1 email typo, aliases 2 merchandise listings for the aforesaid item.

Good copy handling requires a survivorship rule: if 2 records mention to the aforesaid entity, which worth survives? The latest reside whitethorn beryllium amended for delivery, but the verified PAN whitethorn beryllium amended for KYC. This is wherever business discourse matters.

Data Quality Metrics to Track

If you declare the dataset is clean, beryllium it pinch metrics. These measures are particularly useful erstwhile explaining your cleaning activity to a business stakeholder aliases interviewer.

Definitions You Should Be Able to Say Clearly

Data cleaning: Detecting and fixing inaccurate, incomplete, copy aliases inconsistent records truthful study reflects the existent process.
Missing value: A section pinch nary usable recorded worth for an study wherever a worth whitethorn beryllium expected.
Outlier: An study unusually distant from different observations successful the aforesaid adaptable aliases business context.
Duplicate: Two aliases much records that correspond the aforesaid transaction, entity aliases event.
Imputation: Replacing a missing worth pinch an estimated worth based connected rules, statistic aliases models.
Data profiling: Summarising structure, distributions, missingness, characteristic and anomalies earlier cleaning decisions.

Meesho: Cleaning Marketplace Data Before It Reaches Search, Recommendations and Decisions

Meesho shows why cleaning seller-generated marketplace information is simply a business capability, not conscionable a method step.

Situation: Meesho operates successful Indian e-commerce pinch a ample guidelines of sellers, value-conscious buyers and highly varied merchandise catalogues crossed fashion, location and manner categories. In specified a marketplace, earthy information tin go messy quickly: merchandise titles whitethorn beryllium inconsistent, sizes whitethorn beryllium missing, copy listings whitethorn appear, and customer addresses whitethorn alteration crossed languages, pincodes and section naming conventions.

The move: The superior cleaning driver is catalogue and information governance astatine ingestion: standardising merchandise attributes earlier they travel into search, recommendations, pricing and operations dashboards. Supporting drivers see mandatory category-specific fields, fuzzy matching to place copy listings, validation rules for pincodes and merchandise attributes, quality reappraisal for ambiguous cases, and feedback loops from returns, ratings and hunt behaviour.

Outcome aliases lesson: Cleaner information improves discoverability, reduces misleading analytics and makes downstream instrumentality learning much reliable. The important instruction for interviews is that information cleaning is not 1 bid successful Python; it is simply a strategy of rules, validation and business ownership.

Marketplace information becomes messy astatine the constituent wherever existent sellers, products and customer addresses participate the system.

Takeaway: A shallow reply says “remove nulls and duplicates.” A beardown reply says “build rules that sphere the existent marketplace awesome while preventing bad information from polluting decisions.”

How AI Changes Data Cleaning Fundamentals

AI is making information cleaning faster, but not judgment-free. In 2026, the champion analysts usage AI to observe patterns and propose rules, while humans still determine what the information means.

AI-assisted anomaly detection: Machine learning tin emblem different transactions, transportation times, claims, clicks aliases customer behaviour that elemental thresholds whitethorn miss. The expert still classifies each shape arsenic error, uncommon arena aliases signal.
Entity solution astatine scale: AI helps lucifer fuzzy duplicates crossed names, addresses, telephone formats and merchandise titles. This is particularly useful successful customer maestro data, marketplace catalogues and banking KYC workflows.
LLM-generated cleaning rules: Tools tin publication file descriptions and propose checks for illustration day consistency, allowed categories, pincode magnitude aliases negative-value warnings. The consequence is hallucinated rules, truthful each norm needs business validation.

Load a sample dataset dictionary, your cleaning notes and the institution discourse into ChatGPT aliases Claude. Ask: “Profile apt data-quality risks, propose validation rules for each column, and database questions I should inquire earlier imputing missing values.” Then manually verify each proposal against the business problem.

Interview Relevance

“You person a customer transactions dataset pinch missing values, utmost transaction amounts and copy customer IDs. How will you cleanable it earlier analysis?”

Use 1 business building successful your answer: “I would not region thing until I cognize whether it is bad information aliases valuable signal.” That condemnation signals maturity.

The mistake: blindly deleting rows pinch nulls, removing each outliers and moving `drop_duplicates()` without business context. Why it costs candidates: it shows instrumentality knowledge but anemic analytical judgment. The fix: diagnose the origin first, take a curen second, and validate the business effect earlier last analysis.

What to Revise Next

Once you understand cleaning, move to the first-pass study that comes instantly aft it. Then revise the mistakes that make different smart analysts suffer credibility.