Data cleaning

Why this is judgment, not a checklist

Data cleaning sounds mechanical and is not. Every real dataset arrives with problems — missing values, duplicated records, outliers, inconsistent category labels, numbers stored as text — and for each one there is a real decision to make rather than a single correct procedure to apply. Getting this stage wrong is one of the most common ways a project fails quietly: the model trains without error, produces plausible-looking predictions, and is simply wrong in ways nobody traces back to a cleaning decision made weeks earlier.

Missing values: the central judgment call

When a value is missing, three options exist, and the right one depends on why it is missing, not just how often.

Drop it. Removing rows with missing values is simplest, and it is safe only when missingness is essentially random with respect to the problem. If people who skip a sensitive survey question tend to have unusually high or low values for exactly the thing the question asked about, dropping those rows does not produce a smaller random sample — it produces a biased one, systematically missing a specific kind of case.

Impute it. Filling in a plausible value — the column’s average, its most common value, or a value predicted from the other columns — keeps every row, at the cost of inventing data that was never actually observed. This is reasonable when missingness looks essentially random and the imputed value is unlikely to meaningfully mislead the model.

Flag it. Adding a separate indicator column recording that a value was missing, alongside an imputed value filling the gap, preserves information that straightforward imputation discards. This matters whenever the fact of something being missing is itself predictive — a customer who never answers an income question may differ systematically from one who does, regardless of what number eventually gets imputed in that field.

What to watch for

The video goes straight to the judgment call that actually matters here — when to drop a missing value and when to impute one — with the tradeoffs stated plainly rather than treated as a rote rule. That is the one part of data cleaning that genuinely requires thought rather than a mechanical procedure, which is why it is the centre of this lesson.

The quieter problems

Missing values get the most attention because they are the most visible. Several other problems are just as corrosive and considerably easier to overlook.

Duplicate rows inflate the apparent importance of whatever they duplicate, and a model can end up effectively over-weighting one case simply because it appears in the data three times rather than once.

Inconsistent category labels — "NY", "New York" and "ny" recorded as though they were three different categories — silently fragment what should be a single group, diluting the signal a model could otherwise extract from it.

Wrong data types — a date stored as plain text, a numeric column read in as a string because one row contains a stray comma — can cause a value to be treated as categorical when it is really continuous, or sorted alphabetically when it should be sorted numerically, producing a model that has quietly learned the wrong kind of relationship entirely.

None of these announce themselves the way a clearly missing value does. Catching them is a large part of what the exploratory analysis immediately preceding this stage is actually for.