Dotwave Detecting and handling outliers
Outliers are the values that sit far outside the normal range of a column — a $2 million order in a dataset where the median is $80, or an age of 350 in a customer table. Left unexamined, they distort averages, inflate standard deviations, and skew the models and charts you build downstream. Step 3 of the Dotwave pipeline, Preprocess, gives every numeric column a dedicated outlier review so you can decide what to do before the values leak into your analysis. This article explains the detection maths, how to read the box plot, and the three handling options Dotwave offers.
How Dotwave detects outliers
Dotwave uses the interquartile range (IQR) method, the same distribution-based rule most analysts reach for by hand. It is robust because it depends on percentiles rather than the mean, so a single extreme value cannot move the boundaries the way it would move an average. The calculation runs per numeric column:
- Q1 is the 25th percentile — a quarter of the values fall below it.
- Q3 is the 75th percentile — three quarters of the values fall below it.
- IQR is the spread of the middle half of the data, calculated as
Q3 − Q1.
From those three numbers Dotwave derives the fences. Any value below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR is flagged as an outlier. The 1.5 multiplier is the conventional Tukey threshold: wide enough to ignore ordinary variation, tight enough to catch the genuinely extreme. Because the method reads the shape of your actual column, a tightly clustered column produces narrow fences while a naturally spread-out column produces wide ones — the definition of "extreme" adapts to each field.
What the box plot shows
For every numeric column Dotwave draws a box plot so you can see the distribution rather than trusting a single count. Reading it left to right (or bottom to top), the plot encodes six things:
- Minimum — the lowest non-outlier value, at the end of the lower whisker.
- Q1 — the near edge of the box.
- Median — the line inside the box, the middle value of the column.
- Q3 — the far edge of the box.
- Maximum — the highest non-outlier value, at the end of the upper whisker.
- Outlier points — individual dots plotted beyond the whiskers, one per flagged value.
The box itself is the IQR, so a long box means a widely spread middle half and a short box means tight clustering. Scanning the dots beyond the whiskers tells you at a glance whether you are dealing with one stray typo or a whole tail of extreme readings — a distinction that should shape which option you pick next.
Your options
Once a column's outliers are on screen, Dotwave gives you three handling choices. Each is applied as a step in your cleaning recipe, so the decision is recorded and repeatable:
Replace every value that falls outside the fences with the boundary value — anything above the upper fence becomes the upper fence, anything below the lower fence becomes the lower fence. All rows are kept; only the extreme magnitudes are pulled in. This is often called winsorizing.
Delete the rows that contain an outlier in this column entirely. Use this when an outlier means the whole record is invalid — for example a corrupted import row — rather than one unusual-but-real measurement.
Make no change. Choose this when the extreme values are legitimate and meaningful — a genuine high-value customer or a real spike you want your analysis to reflect.
Removing rows is permanent within the recipe. Cap outliers is usually safer — it preserves all rows and just limits extreme values.
Choosing between them
Reach for Cap when the extreme values are plausible but you do not want them dominating an average or a chart axis. Reach for Remove rows only when the record itself is untrustworthy, and remember it shrinks your dataset. Reach for Keep as-is whenever the outliers carry signal you care about. Because each choice lives in your recipe, you can revisit it, and the audit trail records exactly which columns you treated and how.