Pandas Rescue

A diagnostic-first approach to broken or slow pandas code. Identify the failure mode before reaching for syntax.

When to use

• The user has a DataFrame, runs an operation, gets the wrong answer
• A pandas pipeline got slow after recent edits and they don't know why
• A merge or groupby produced a row count they didn't expect
• A warning fires that they don't understand

Skip for: "teach me pandas from scratch" (recommend a tutorial); "convert to polars" (separate decision); brand-new data science learners (different skill).

Workflow

1. Always start with df.info() and df.head(). Before reading any code, dump the DataFrame's shape, dtypes, non-null counts. 60% of pandas confusion is "the column I thought was int64 is object" or "I have 12k rows and 6k nulls in the join key." .info() answers both. Run .describe(include='all') if you need string stats too.

2. Diagnose the failure mode (one of seven):

   Symptom	Likely cause	First thing to try
   SettingWithCopyWarning	View-vs-copy ambiguity	df = df.copy() after a chained selection
   Merge changed row count	One side has duplicates in the key	df[key].duplicated().sum() on both sides
   NaN where you expected 0	Aggregation skipped a group entirely	.fillna(0) after the agg OR groupby(..., dropna=False)
   KeyError after groupby	MultiIndex; the column you want is in .index	.reset_index() after the agg
   Wrong dtype	Read CSV inferred wrong; or NaN promoted int→float	dtype= in read_csv OR astype('Int64') (capital I, nullable)
   Slow operation	.iterrows() or .apply() row-wise	Vectorize: column-level ops, np.where, pd.merge
   Date math broken	Strings parsed as object, not datetime64	pd.to_datetime(..., utc=True) early in the pipeline

3. Fix one thing at a time. Apply the most likely fix, re-run df.info() + the failing operation. Don't bundle three fixes into one diff — you won't know which one mattered, and one of them probably broke something else.

4. Pin the input. If the bug is non-deterministic ("it worked last week"), check the upstream data: row count, dtypes, NaN counts. Often the real bug is upstream and pandas is faithfully reflecting it.

5. Performance: vectorize, then chunk, then leave pandas.

   • If it's a row-wise .apply() doing simple math → replace with column-level vectorization (df['x'] + df['y'], np.where).
   • If the DataFrame is >2GB in RAM → either chunksize= in read_csv OR switch to polars / DuckDB for that step.
   • Don't pd.concat in a loop — accumulate rows in a list, build the DataFrame once.

Output shape

For a fix: a corrected code snippet + 1–2 sentences explaining the failure mode (not just the syntax). Without the failure-mode explanation, the user will hit the same shape again next week.

For a performance question: the slow operation's profile (%timeit or cProfile numbers), the bottleneck, the rewrite, and the new numbers.

Definition of done

• Failure mode named (one of the seven, or a clear new category)
• One-line explanation of WHY the original was wrong
• The rewrite runs to completion in a fresh REPL with the user's data
• For perf fixes: before/after numbers

Gotchas

1. Don't suggest .copy() blindly. It usually shuts up the warning but masks a real bug — the user IS modifying a view they thought was independent. Diagnose first; copy second.

2. Int64 vs int64. Nullable integer is capital-I Int64. Regular int64 can't hold NaN — assignment will silently upcast to float64. If the user has integer columns with NaN, recommend the nullable dtype.

3. groupby default is dropna=True. Surprise null group disappears silently. Recommend explicit dropna=False when null is a meaningful category.

4. .merge how='left' doesn't guarantee no row growth. A left join with duplicates on the right side still multiplies rows. Always check df.shape[0] before and after.

5. Beware inplace=True. Returns None, breaks chaining, no perf benefit on modern pandas. Read the doc; recommend the non-inplace form.

6. Sometimes the answer is "use polars." If the user is fighting pandas for a 10GB CSV or a complex string-heavy pipeline, polars or DuckDB are often 10x faster with cleaner ergonomics. Say so when it fits — don't force pandas where it shouldn't be.