Streaming CSV Imports Without Blowing Up Memory
To import a huge CSV safely, stream the file through a parser so memory stays constant, validate each row as it arrives, enqueue work in small chunks for concurrency-safe upserts, and record per-row status so failed rows can be retried without redoing the whole job.
Loading a large CSV into memory and looping over it works until the file is big enough to crash the process. The fix is to never hold the whole file at once.
Stream the parse
A streaming parser hands you one row at a time. Memory usage stays flat whether the file is a thousand rows or a million.
Validate per row
Run each row through a schema as it arrives and record precise, actionable errors — row number, column, and reason — rather than failing the whole import on the first bad cell.
Queue in chunks
Batch rows into small groups and process them through a queue with bounded concurrency. Upserts must be safe to run in parallel and idempotent on retry.
Make it resumable
Persist the status of every row. If the job dies halfway, you can retry only the rows that failed instead of starting over — and you can hand the user a downloadable error report.
Frequently asked questions
- How do I import a CSV without running out of memory?
- Use a streaming parser that yields one row at a time so memory stays constant, and process rows in small queued batches rather than loading the entire file.