Filtering Data

If there is one Pandas pattern you will use more than any other, it is the boolean mask. Almost every interesting question you ask of a dataset starts with "first, give me the rows where..."

The basic shape

A boolean mask is a Series of True/False values, one per row. Apply it to a DataFrame and you get back only the rows where the mask is True.

Two-step or one-step — pick your style:

# Two-step (more readable for complex conditions)
high_earners = df["salary"] > 100
df[high_earners]

# One-step (more compact)
df[df["salary"] > 100]

Comparison operators

All the comparisons return a boolean mask:

Combining conditions with &, |, ~

You must use & (and), | (or), ~ (not) — not the Python keywords and, or, not. And always wrap each condition in parentheses, because & has higher precedence than >.

If you forget the parentheses you will get a cryptic error about ambiguous truth values. The fix is always the same: add parentheses.

Useful helpers

A handful of methods produce boolean masks directly:

isin — membership

between — ranges

str.contains — substring matching

isna / notna — missing checks

query — a string-based DSL

Pandas has an alternative filter syntax that some people find more readable, especially for multi-condition filters:

Whether you prefer query or boolean masks is mostly a style choice. Most codebases use masks; query shows up in notebooks.

A real-world filter

You will write filters like this all day, every day, as an analyst. Cosmetic note: breaking each condition onto its own line and aligning the &s makes complex filters readable.

Filter, then assign

A common pattern: filter to a subset, then add or modify a column on just that subset. Use .loc for the assignment:

A filter challenge

Check your understanding