RunTimeIndex

MemoryError, what it means and how to fix it

MemoryError is raised when Python cannot allocate memory for an operation. Two very different situations produce it: data that is legitimately enormous, and a bug that grows memory without bound (an infinite loop appending to a list is the textbook case). Distinguishing them matters, because the first is solved by streaming and data types, and the second by finding the leak-like loop that no amount of RAM would satisfy.

What it looks like

seen = []
while True:                 # nothing ever stops this
    seen.append(load_next_batch())

Traceback (most recent call last):
  File "main.py", line 3, in <module>
MemoryError

# the honest fix is a stopping condition or streaming,
# not more RAM: process each batch and let it go

The fix, in order

1. Decide which kind it is: watch memory while it runs. Steady growth to the ceiling means unbounded accumulation (find what grows). A single spike at one line means one enormous operation (find which).
2. Stream instead of loading. Read files line by line or in chunks instead of read() on the whole file; use generators so results are produced and discarded one at a time instead of collected into a list.
3. Hunt accidental copies. Slicing giant lists, list * n on big data, joining enormous strings in a loop, and keeping every intermediate result in a “history” list are quiet multipliers. Most “huge data” programs are holding the data three times over.
4. Shrink the representation. For numeric data, arrays or NumPy dtypes use a fraction of the memory of Python objects (a list of ints costs several times the raw numbers). For text, process and discard rather than storing everything.
5. Check the ceiling you actually have. 32-bit Python caps near 2 GB regardless of machine RAM; containers and services often have explicit limits far below the machine's memory. A 64-bit interpreter and a raised container limit are configuration fixes, distinct from code fixes.

If it still happens

If a program's memory grows in proportion to uptime rather than input size, something accumulates per operation (caches without eviction, registries of completed tasks, logging buffers). Profile by counting: print the lengths of the long-lived collections periodically and the growing one names itself.

Common questions

Related errors