Learning from data that does not fit in memory

Setup#

pip install scikit-learn numpy

Outputs verified with scikit-learn 1.7.2.

partial_fit: fit, but resumable#

In sklearn, out-of-core models expose partial_fit — the incremental sibling of fit. The chunk generator below fakes reading from a huge file, so the example runs anywhere; in real code the generator would read the next slice of a CSV or a database cursor.

import numpy as np
from sklearn.linear_model import SGDClassifier

rng = np.random.default_rng(0)

def chunk_stream(n_chunks=50, rows=200):
    """Pretend each chunk was read from a huge file on disk."""
    for _ in range(n_chunks):
        X = rng.normal(size=(rows, 20))
        y = (X[:, :3].sum(axis=1) + rng.normal(0, 0.5, rows) > 0).astype(int)
        yield X, y

model = SGDClassifier(loss="log_loss", random_state=0)
for i, (X, y) in enumerate(chunk_stream()):
    model.partial_fit(X, y, classes=[0, 1])   # classes needed on the first call
    if i % 10 == 0:
        print(f"after chunk {i:2d}: accuracy on this chunk = {model.score(X, y):.3f}")

X_test, y_test = next(chunk_stream(1))
print(f"final accuracy on unseen data: {model.score(X_test, y_test):.3f}")

after chunk  0: accuracy on this chunk = 0.870
after chunk 10: accuracy on this chunk = 0.875
after chunk 20: accuracy on this chunk = 0.920
after chunk 30: accuracy on this chunk = 0.875
after chunk 40: accuracy on this chunk = 0.920
final accuracy on unseen data: 0.905

Chunk-by-chunk accuracy wobbles with each chunk's luck; the exact decimals may shift slightly across sklearn versions, though the rising-then-stable shape is what matters.

The walkthrough#

classes=[0, 1] on the first call. A chunk might contain only one class — imagine 200 rows with no fraud in them. The model must know the full set of labels up front, since no chunk can be trusted to show them all. Forgetting this argument is the most common partial_fit error.

SGDClassifier with loss="log_loss" is logistic regression trained by stochastic gradient descent — the "small correction steps" learner. The same class does SVM-style learning with loss="hinge". For regression there is SGDRegressor; for counting-based models, MultinomialNB.partial_fit; for clustering, MiniBatchKMeans.partial_fit.

The generator is the architecture. chunk_stream yields one piece at a time and holds nothing else. Swap its body for pd.read_csv("huge.csv", chunksize=200) and the same loop trains on a file of any size.

Text pairs perfectly with the hashing trick. A normal vectorizer must see all documents to build its vocabulary — which is a full pass you cannot afford. HashingVectorizer needs no fit at all, so each text chunk can be vectorised and fed onward independently.

Common mistakes#

Scaling with a scaler fitted on... what, exactly? StandardScaler().fit() needs the full column to compute a mean — the one thing you do not have. Either use StandardScaler.partial_fit chunk by chunk in a first pass, or compute running statistics, or use scale-free preprocessing like hashing. Fitting the scaler on chunk 1 alone and hoping is the quiet version of this bug.

Feeding chunks in a meaningful order. If the file is sorted by date or by label, early chunks teach a world the late chunks contradict, and the model's final state overweights whatever came last. Shuffle the source once if you can, or interleave reads. Sorted-by-label input can even crash the first partial_fit with a single-class chunk if classes was omitted.

Assuming one pass is enough. One sweep through the data equals one epoch. If the stream is finite and accuracy still climbs, loop over the file again — a second and third pass often helps, exactly as extra epochs do in deep learning.

Reaching for out-of-core when the data fits. Ten million rows of 20 floats is around 1.6 GB — that fits in memory, and plain fit is faster and less fiddly. Measure first; out-of-core is the tool for actually oversized data.

For the PyTorch version of this problem — image folders bigger than RAM, streaming datasets — see datasets too big for RAM.

Try it yourself#

Break it on purpose: sort all chunks so every label-0 row comes before every label-1 row, and retrain. Watch the final accuracy fall, then fix it by shuffling within a buffer of five chunks.

What to learn next#

• The hashing trick — the vectoriser that needs no vocabulary pass.
• Streaming data — the infrastructure that delivers the chunks.
• Datasets too big for RAM — the same problem in PyTorch.