Inspect and clean a dataset

Real data is messy, and cleaning often takes most of a project. Look for:

• Missing values - blank, NA, -1, 999. Missing isn’t zero: it may mean unknown, not recorded or not applicable.
• Duplicates - the same observation recorded twice.
• Inconsistent categories - Bus, bus, BUS, bicycle vs bike.
• Units and types - minutes mixed with seconds, numbers stored as text, dates in two formats.
• Impossible or extreme values - a 9,000-minute trip, a negative age.

For each problem, choose a rule, apply it in code (never by hand in a spreadsheet), and write it down - your conclusions depend on these choices.

In pandas, the common fixes are one line each:

1trips["minutes"] = trips["minutes"].replace(-1, pd.NA)        # sentinel → missing
2trips = trips.drop_duplicates(subset="trip_id")                # keep the first copy
3trips["mode"] = trips["mode"].str.strip().str.lower()          # " Bus" → "bus"
4trips["mode"] = trips["mode"].replace({"bicycle": "bike"})     # merge aliases
5trips["start"] = pd.to_datetime(trips["start"], format="mixed") # one date type

John Tukey’s rule flags values more than 1.5 × IQR below the first quartile (Q1) or above the third (Q3), where the interquartile range IQR = Q3 − Q1 is the spread of the middle half of the data. A flagged value is a question, not a verdict: it may be an error to fix, or the most interesting observation in the dataset. Never delete outliers just to make results look tidier.