UnicodeDecodeError, what it means and how to fix it

Text files are bytes; decoding turns bytes into characters using an encoding. Python opens text files as UTF-8 by default on most systems, and this error means the bytes are not valid UTF-8: the file was actually saved in an older encoding (Windows-1252 is the usual suspect, especially for files from Excel or older Windows tools), in UTF-16, or it is not text at all. The message names the byte and the exact position where decoding failed.

What it looks like

open("names.csv").read()     # the file is really Windows-1252

Traceback (most recent call last):
  File "main.py", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9
in position 103: invalid continuation byte

# fixed: decode with the encoding the file actually uses
open("names.csv", encoding="cp1252").read()

The fix, in order

  1. Ask where the file came from. CSVs exported by Excel on Windows are typically Windows-1252 (or UTF-16 in some exports), not UTF-8. Files from older systems and some locales carry their legacy encoding. Provenance predicts the encoding better than guessing.
  2. Open with the actual encoding: open(path, encoding="cp1252") for Western European Windows files; encoding="utf-8-sig" for UTF-8 files that begin with a byte-order mark (the ones whose first character comes out as \ufeff); encoding="utf-16" for files where every other byte looks empty in a hex view.
  3. Let the data tell you: the failing byte and position in the message point at the first non-UTF-8 byte. Accented characters, smart quotes, and currency symbols in the source file are the usual bytes involved.
  4. Convert at the boundary, consistently. Read with the legacy encoding, then work in Unicode internally and write UTF-8 out. Mixed pipelines that decode differently at different stages corrupt text quietly instead of failing loudly.
  5. Use error handlers only as a deliberate trade: errors="replace" or errors="ignore" keep the program running but destroy the undecodable characters. Acceptable for logs and previews; a data loss bug when used on data you must keep.

If it still happens

If the failing position is 0 and the first bytes look like a file signature (a “PK” for zip-based formats like xlsx and docx, or image headers), the file is binary and should never be decoded as text at all; use the library for that format instead of open().

Common questions

Why does the same code work on my machine but fail on another? The default encoding for open() depends on the system locale. A Windows machine may default to cp1252 and silently succeed where a Linux machine defaulting to UTF-8 fails (or the reverse). Passing encoding= explicitly makes the behavior identical everywhere, which is why explicit is the fix and not just a workaround.

What does 'invalid continuation byte' versus 'invalid start byte' mean? UTF-8 multi-byte characters have a start byte announcing the sequence length and continuation bytes following a fixed pattern. A continuation error means a start byte was followed by bytes that do not continue it; a start error means a byte appeared that cannot begin any character. Both mean the bytes are not UTF-8; the distinction rarely changes the remedy.

Related errors

FileNotFoundError · JSONDecodeError · OSError