Dotwave Understanding nulls and duplicates
Two data-quality issues show up in almost every real-world file an analyst receives: missing values and duplicate rows. Dotwave measures both during profiling and surfaces them in Step 1, but the numbers only help if you know exactly what they count. This article explains how Dotwave defines and reports nulls and duplicates, how the per-column figures relate to the totals, and why neither problem should be fixed reflexively.
What null_pct means
A null is a missing value — a cell with no data at all, which is different from a cell containing a zero or an empty string. For each column, Dotwave reports null_pct, the percentage of rows in that column that are missing a value. It is calculated simply: the column's null count divided by the total number of rows, expressed as a percentage. A null_pct of 4% on a 10,000-row dataset means 400 rows have no value in that column.
The percentage matters more than the raw count because it scales with the dataset. Two hundred missing values is trivial in a million-row file and catastrophic in a three-hundred-row one. Dotwave uses null_pct to drive its highlighting and its suggested actions, which is why the same amber-at-10%, red-at-50% thresholds appear throughout the pipeline: they are proportions, not fixed counts, so they mean the same thing regardless of file size.
Per-column null counts versus total_nulls
Dotwave reports nulls at two levels, and it is worth keeping them straight. Each column has its own null count — the number of missing cells in that one field. The dataset-level total_nulls tile is the sum of every column's null count across the whole table. It is a single figure that answers "how much is missing overall," while the per-column counts answer the far more actionable question of "where is it missing."
A large total_nulls can be entirely benign if it comes from one optional field that is legitimately blank most of the time, or genuinely serious if it is spread thinly across every column in a way that will drop most rows if you filter on completeness. Never plan a fix from the total alone — open the column table, see how the total decomposes, and treat each column on its own terms.
A missing value and a zero are not the same thing. A null means "no data was recorded"; a zero means "the recorded value was zero." Filling nulls with zero on a numeric column can silently distort every average and sum you compute later, so choose a fill that matches what the missing data actually represents.
What counts as an exact duplicate row
Dotwave counts exact duplicate rows — rows where every single column matches another row value-for-value. A row that agrees with another on nine columns but differs on the tenth is not a duplicate by this definition; it is a distinct record that happens to be similar. This strict, whole-row rule is deliberate. It means Dotwave will never quietly collapse two rows that differ in some meaningful field, so the duplicate count you see is conservative and trustworthy: every row it flags really is a byte-for-byte repeat of another.
Near-matches — the same customer entered twice with a slightly different spelling, or London versus london — are a separate problem that whole-row matching will not catch, because the rows are not actually identical. Those are handled by fuzzy deduplication in the guided cleaning dialog, not by the exact-duplicate count here.
Previewing duplicates before you act
Before removing anything, preview the duplicate rows Dotwave found. The preview shows you the actual repeated records side by side so you can confirm they are noise rather than signal. This step is not a formality — it is where you decide whether the duplicates are an artifact of a bad export or a real feature of the data. Looking at the rows themselves, rather than trusting the count, is what keeps you from deleting legitimate records.
Why duplicates may be intentional
It is tempting to treat every duplicate as an error, but in real datasets they are often entirely correct. A customer who places the same order twice in a month produces two identical order rows, and both are real transactions you would be wrong to merge. A sensor logging the same reading at two timestamps, a survey where two respondents genuinely gave identical answers, or a ledger with repeated standard charges can all produce legitimate exact-duplicate rows. The question is never "are there duplicates" but "should these particular records be unique." Answer that from the data's meaning, not from the count.
A high duplicate count (>20% of rows) is unusual. Check your data source before removing — they may be legitimate records.