Pandas

Pandas is a table with named columns that you can filter, group and summarise in one line. It is where almost every AI project starts, because real data arrives as a table.

By Pranay Mahendrakar
14 min read
3 reading levels
Published 20 Aug 2026

Read these first

On this page 7

Why you should care
Why it exists
How it works
A real example you have seen
An honest word
Remember this
What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Pandas is a tool for working with tables — rows, named columns, and messy real-world data.

Think of the register your class teacher carried. One row for each student. One column for each subject. Attendance on the side, names down the left, and someone's tea stain in the corner.

Now imagine an assistant who knows that register perfectly. You ask "who scored above eighty in science?" and the answer comes back before you finish the sentence. You ask "what is the average per subject?" and it is already written down. Pandas is that assistant.

Why you should care

Real data almost never arrives as a neat pile of numbers. It arrives as a table with names, dates, blank cells, and text that should have been a number. A sales export. A hospital record. A survey.

Before a model can learn, somebody must load that table. Then look at it, fix the holes, and reshape it. That work is most of an AI project. Practitioners routinely report it eats more time than the modelling does. Pandas is the tool for it.

Why it exists

NumPy is excellent at grids of numbers that are all the same kind. Real tables are not like that. One column holds city names, the next holds cups sold, the next holds a price with decimals.

NumPy also has no idea what a column is called. You would have to remember that column two is the price, and remembering that is how mistakes happen.

Wes McKinney built pandas in 2008 at a financial firm for exactly this reason. He needed named columns, mixed types, dates, and missing values, with NumPy's speed underneath.

How it works

NumPy array                 Pandas DataFrame
-----------                 ----------------
one type for everything     every column has its own type
no names                    every column has a name
positions only              rows can be labelled by date, name or id
holes not allowed           blanks are expected and handled

You do the work in whole columns, never row by row:

"add a revenue column"   →   cups column x price column   →   new column, all rows at once
"only rows above 200"    →   a true/false test per row    →   a smaller table
"total per city"         →   group the rows by city       →   one line per city

A real example you have seen

Open your bank's monthly statement. Rows are transactions. Columns are date, description, amount, balance. Now you want your total spend on food last month.

You would filter the rows by description, then add up one column. That is exactly a pandas filter followed by a sum. The tool is a written version of what you already do by eye.

An honest word

Pandas has more than one way to do almost everything. Some of those ways behave differently, and that surprises experienced people too. There is a famous warning message about copies that confuses nearly everyone the first time.

If pandas feels bigger and stranger than NumPy, that is a correct reading. Learn a small core well: load, look, filter, group, join. That core covers most days.

Remember this

Pandas is a table with named columns, built on top of NumPy.
You operate on whole columns at once, not one row at a time.
Cleaning and reshaping data is most of real AI work, and this is the tool for it.

What to learn next

Matplotlib — turning your table into a picture.
NumPy — the fast array layer underneath pandas.
Python for machine learning — moving from a table to a trained model.

Developer — Code and libraries.

Setup

bash

pip install pandas

That pulls in NumPy as a dependency. Every example below uses an inline table, so nothing is downloaded and everything runs on a CPU instantly.

Building and inspecting a table

look.py

import pandas as pd

# A tiny sales book. In real work this line would be pd.read_csv("sales.csv").
sales = pd.DataFrame({
    "city":  ["Pune", "Pune", "Nagpur", "Nagpur", "Pune", "Nagpur"],
    "item":  ["chai", "coffee", "chai", "coffee", "chai", "chai"],
    "cups":  [40, 12, 25, 9, 33, 18],
    "price": [10.0, 25.0, 10.0, 25.0, 10.0, 10.0],
})

print(sales)
print()
print("shape:", sales.shape)
print(sales.dtypes)

Output

     city    item  cups  price
0    Pune    chai    40   10.0
1    Pune  coffee    12   25.0
2  Nagpur    chai    25   10.0
3  Nagpur  coffee     9   25.0
4    Pune    chai    33   10.0
5  Nagpur    chai    18   10.0

shape: (6, 4)
city      object
item      object
cups       int64
price    float64
dtype: object

The leftmost column with 0 1 2 3 4 5 is the index, the row label. It is not a data column. object is the dtype pandas uses for Python strings.

Filter, derive, group, sort

analyse.py

import pandas as pd

sales = pd.DataFrame({
    "city":  ["Pune", "Pune", "Nagpur", "Nagpur", "Pune", "Nagpur"],
    "item":  ["chai", "coffee", "chai", "coffee", "chai", "chai"],
    "cups":  [40, 12, 25, 9, 33, 18],
    "price": [10.0, 25.0, 10.0, 25.0, 10.0, 10.0],
})

# A new column is built from whole columns at once, never row by row.
sales["revenue"] = sales["cups"] * sales["price"]

big = sales[sales["revenue"] > 200]
print("rows above 200:")
print(big)
print()

print("revenue by city and item:")
print(sales.groupby(["city", "item"])["revenue"].sum())
print()

print("sorted by revenue:")
print(sales.sort_values("revenue", ascending=False).head(3))

Output

rows above 200:
     city    item  cups  price  revenue
0    Pune    chai    40   10.0    400.0
1    Pune  coffee    12   25.0    300.0
2  Nagpur    chai    25   10.0    250.0
3  Nagpur  coffee     9   25.0    225.0
4    Pune    chai    33   10.0    330.0

revenue by city and item:
city    item  
Nagpur  chai      430.0
        coffee    225.0
Pune    chai      730.0
        coffee    300.0
Name: revenue, dtype: float64

sorted by revenue:
   city    item  cups  price  revenue
0  Pune    chai    40   10.0    400.0
4  Pune    chai    33   10.0    330.0
1  Pune  coffee    12   25.0    300.0

Notice the row labels in the sorted output: 0, 4, 1. Sorting moved the rows but kept each row's original index. That is deliberate, and it is how pandas keeps a row identifiable after reordering.

Named aggregations and joins

report.py

import pandas as pd

sales = pd.DataFrame({
    "city":  ["Pune", "Pune", "Nagpur", "Nagpur", "Pune", "Nagpur"],
    "item":  ["chai", "coffee", "chai", "coffee", "chai", "chai"],
    "cups":  [40, 12, 25, 9, 33, 18],
    "price": [10.0, 25.0, 10.0, 25.0, 10.0, 10.0],
})
sales["revenue"] = sales["cups"] * sales["price"]

print("summary per city:")
print(sales.groupby("city").agg(
    orders=("cups", "size"),
    total_cups=("cups", "sum"),
    avg_revenue=("revenue", "mean"),
))
print()

owners = pd.DataFrame({
    "city":  ["Pune", "Nagpur"],
    "owner": ["Asha", "Ravi"],
})
print("after merge:")
print(sales.merge(owners, on="city").head(3))
print()
print("value counts:")
print(sales["item"].value_counts())

Output

summary per city:
        orders  total_cups  avg_revenue
city                                   
Nagpur       3          52   218.333333
Pune         3          85   343.333333

after merge:
     city    item  cups  price  revenue owner
0    Pune    chai    40   10.0    400.0  Asha
1    Pune  coffee    12   25.0    300.0  Asha
2  Nagpur    chai    25   10.0    250.0  Ravi

value counts:
item
chai      4
coffee    2
Name: count, dtype: int64

Named aggregation — the orders=("cups", "size") form — is worth learning early. It names the output column and states the source column and the function in one readable line. The older agg({"cups": ["sum", "size"]}) form produces a two-level column header that is awkward to work with.

merge is a database join. on="city" matches rows by that column. The default is an inner join. Cities present in one table but not the other are dropped without a warning. Pass how="left" and check the row count afterwards when that matters.

Missing values

holes.py

import numpy as np
import pandas as pd

readings = pd.DataFrame({
    "sensor": ["a", "b", "c", "d", "e"],
    "temp":   [31.5, np.nan, 34.0, np.nan, 30.0],
})

print(readings)
print()
print("missing per column:")
print(readings.isna().sum())
print()
print("filled with the column mean:")
readings["temp_filled"] = readings["temp"].fillna(readings["temp"].mean())
print(readings)
print()
print("dropping the bad rows instead:")
print(readings.dropna(subset=["temp"]))

Output

  sensor  temp
0      a  31.5
1      b   NaN
2      c  34.0
3      d   NaN
4      e  30.0

missing per column:
sensor    0
temp      2
dtype: int64

filled with the column mean:
  sensor  temp  temp_filled
0      a  31.5    31.500000
1      b   NaN    31.833333
2      c  34.0    34.000000
3      d   NaN    31.833333
4      e  30.0    30.000000

dropping the bad rows instead:
  sensor  temp  temp_filled
0      a  31.5         31.5
2      c  34.0         34.0
4      e  30.0         30.0

NaN means "not a number" and is how pandas marks a blank in a numeric column. Note that readings["temp"].mean() skips the blanks rather than treating them as zero. Filling with the mean is convenient. It also shrinks the apparent variation in your data, so treat it as a decision rather than a default.

Common mistakes

1. Chained assignment.

python

import pandas as pd

sales = pd.DataFrame({"city": ["Pune", "Nagpur"], "revenue": [400.0, 250.0]})
sales[sales["city"] == "Pune"]["revenue"] = 0
print(sales)

Output

SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

     city  revenue
0    Pune    400.0
1  Nagpur    250.0

Pune's revenue is still 400.0. The first bracket produced a copy, so the assignment landed on a temporary object that was thrown away. Under Copy-on-Write the warning disappears and the no-op stays. Fix: one indexer, not two — sales.loc[sales["city"] == "Pune", "revenue"] = 0.

2. Mixing and with element-wise conditions.

python

import pandas as pd

sales = pd.DataFrame({"cups": [40, 12, 25], "price": [10.0, 25.0, 10.0]})
print(sales[(sales["cups"] > 20) and (sales["price"] < 20)])

Output

Traceback (most recent call last):
  File "filter.py", line 4, in <module>
    print(sales[(sales["cups"] > 20) and (sales["price"] < 20)])
ValueError: The truth value of a Series is ambiguous.
Use a.empty, a.bool(), a.item(), a.any() or a.all().

Python's and wants a single true-or-false value. Each side here is a whole column of them, so pandas refuses to guess what you meant. Fix: use & and keep every condition in its own brackets — (a > 20) & (b < 20).

3. Looping with iterrows. It is readable and it is slow, because each row is rebuilt as a Series object. On a hundred rows nobody notices. On a million rows it is the difference between one second and twenty minutes. Fix: express the work as column arithmetic, or use df.apply on a whole column, or drop to .to_numpy().

4. Trusting an inner join silently. merge drops non-matching rows without complaint. A typo in one city name quietly deletes that city's revenue from your report. Fix: print len(df) before and after every merge, or pass indicator=True and inspect the _merge column.

Try it yourself

Add a "discount" column that gives ten percent off any row where cups is above thirty, and nothing otherwise. Use np.where(sales["cups"] > 30, 0.1, 0.0) rather than a loop.

Then compute revenue after discount and rerun the per-city summary. Confirm that only Pune's numbers move, and think about why.

What to learn next

Matplotlib — plotting straight from a DataFrame.
Python for machine learning — going from a clean table to a model.
Feature engineering — turning table columns into model inputs.

Researcher — Mathematics and papers.

Storage model

A DataFrame is not a two-dimensional array. It is a collection of one-dimensional arrays with a shared row index, managed by an internal BlockManager.

The manager consolidates: columns sharing a dtype are gathered into a single two-dimensional NumPy block. A frame with 30 float columns and 5 object columns typically holds two blocks, not 35 arrays.

Consequences that matter in practice:

Column-wise operations are contiguous NumPy operations and run at NumPy speed.
Row-wise access crosses block boundaries and involves boxing each value into a Python object. iterrows is slow for this structural reason, not because of a missing optimisation.
Inserting k columns one at a time can trigger repeated consolidation, giving O(k**2) copying. Build a dict of columns and construct once, or pd.concat along axis=1 in a single call.
df.values and df.to_numpy() on a mixed-dtype frame must upcast to a common dtype, often object. That silently destroys the performance you came for.

Copy semantics and Copy-on-Write

The historical ambiguity — does a selection return a view or a copy? — was resolved by Copy-on-Write (CoW). Under CoW, every indexing operation behaves as if it returned a copy. The underlying buffer stays shared until a write occurs. Only then is the data duplicated.

CoW is opt-in from pandas 2.0 via pd.options.mode.copy_on_write = True and becomes the default behaviour in pandas 3.0. Check pd.__version__ before assuming which regime you are in. Under CoW, chained assignment never propagates, and SettingWithCopyWarning is retired because the ambiguity it warned about no longer exists.

Cost of the core operations

Let n be the number of rows, g the number of distinct groups, and k the number of columns.

Operation	Typical cost	Notes
Column arithmetic	`O(n)`	a NumPy ufunc over one contiguous block
Boolean mask selection	`O(n)`	builds a mask, then a gather; allocates a copy
`groupby(...).sum()`	`O(n)` average	hash table over key columns, then a segmented reduction
`sort_values`	`O(n log n)`	quicksort by default; pass `kind="mergesort"` for stability
`merge` on a key	`O(n + m)` average	hash join; falls back to sort-merge when both sides are sorted
`set_index` then `.loc`	`O(1)` to `O(log n)`	hash index for unique keys, binary search for a sorted `MultiIndex`
`iterrows`	`O(n)` with a large constant	one Series object constructed per row

The groupby path uses a factorisation step. Key columns are mapped to dense integer codes. The aggregation then becomes a scatter-add over those codes, in compiled Cython. This is why grouping a million rows is fast while looping over a thousand is not.

Memory, and the cost of strings

Output

memory (bytes per column):
Index      128
city       372
item       370
cups        48
price       48
revenue     48
dtype: int64

That is memory_usage(deep=True) on the six-row sales table. Six floats cost 48 bytes. Six short city names cost 372. The exact string figure shifts a little between CPython versions, but the ratio does not.

The reason: an object column stores 8-byte pointers to individual Python str objects. Each of those carries a header of roughly 49 bytes, plus its characters. Two remedies:

Categorical dtype. df["city"] = df["city"].astype("category") stores small integer codes plus one dictionary of distinct values. For low-cardinality columns this is often a 10x to 100x reduction. groupby also gets faster, because the factorisation is already done.
Arrow-backed strings. From pandas 2.0, dtype="string[pyarrow]" or pd.read_csv(..., dtype_backend="pyarrow") stores one contiguous UTF-8 buffer plus an offsets array. This removes the per-value Python object entirely and gives vectorised string kernels.

Where pandas is the wrong tool

Pandas materialises everything in RAM and generally needs several times the dataset size as working room during joins and sorts. A useful rule of thumb is that a frame above roughly a third of available memory will start to hurt.

At that point the alternatives are genuinely better, not only different:

Polars — Rust engine, Arrow memory, multi-threaded by default, with a lazy optimiser that fuses operations and pushes filters down.
DuckDB — an embedded columnar SQL engine that queries Parquet and pandas frames directly, with out-of-core execution.
Dask or Ray — partition a pandas-like API across cores or machines. Reach for these when the API is what you want to keep.

There is no shame in the handoff. Pandas remains the best interactive tool for data that fits in memory.

References

McKinney, W. "Data Structures for Statistical Computing in Python." Proceedings of the 9th Python in Science Conference (SciPy), 51–56, 2010. The original design paper.
McKinney, W. Python for Data Analysis, 3rd ed., O'Reilly, 2022.
pandas development team. "Copy-on-Write" and "PDEP-6" in the pandas developer documentation, 2023 onward.
Apache Arrow project. Columnar Format Specification, version 1.0, 2020. The layout behind string[pyarrow].
Abadi, D., Boncz, P., Harizopoulos, S., et al. "The Design and Implementation of Modern Column-Oriented Database Systems." Foundations and Trends in Databases 5(3), 2013. Background for why columnar storage wins on analytics.

What to learn next

NumPy — strides, dtypes, and the memory model underneath.
Feature engineering — encoding, leakage, and train-time transforms.
Statistics — what a group mean does and does not tell you.

🟢 Beginner — No maths. Plain English.

Why you should care#

Why it exists#

How it works#

A real example you have seen#

An honest word#

Remember this#

What to learn next#

🟡 Developer — Code and libraries.

Setup#

Building and inspecting a table#

Filter, derive, group, sort#

Named aggregations and joins#

Missing values#

Common mistakes#

Try it yourself#

What to learn next#

🔴 Researcher — Mathematics and papers.

Storage model#

Copy semantics and Copy-on-Write#

Cost of the core operations#

Memory, and the cost of strings#

Where pandas is the wrong tool#

References#

What to learn next#

Beginner — No maths. Plain English.

Why you should care

Why it exists

How it works

A real example you have seen

An honest word

Remember this

What to learn next

Developer — Code and libraries.

Setup

Building and inspecting a table

Filter, derive, group, sort

Named aggregations and joins

Missing values

Common mistakes

Try it yourself

What to learn next

Researcher — Mathematics and papers.

Storage model

Copy semantics and Copy-on-Write

Cost of the core operations

Memory, and the cost of strings

Where pandas is the wrong tool

References

What to learn next