Converting multi-gigabyte CSVs to Parquet in bounded memory

The service is an internal tool that converts CSV files to Parquet, a compressed columnar format that analytics tools read far faster. Large files, several gigabytes, are the main case.

That rules out the usual shortcuts. According to the project's design notes, ASP.NET's multipart IFormFile buffers the whole upload before application code runs and has a 128 MB default limit, and loading a CSV into memory to convert it cannot scale to multi-gigabyte inputs.

• Inputs of several gigabytes, with memory that stays bounded however large the file.
• One build must run under IIS in-process (uploads up to about 4 GB, a hard IIS limit) or as a Windows Service on Kestrel with no size cap.
• DuckDB has no async API, so a conversion blocks its thread for the whole run.
• No bytes flow while DuckDB converts, so clients, proxies and load balancers need long timeouts.
• Conversions share one CPU and one disk, so concurrency has to be capped and excess load shed cleanly.

Three limits apply from the outside in. Request admission rejects excess load before its upload touches the disk. Conversion slots cap how many DuckDB statements run at once, while uploads and downloads are never capped, so a queued request keeps streaming its body while the ones ahead of it convert. DuckDB's thread and memory settings are totals for the whole service, because every conversion shares one DuckDB instance.

When a client disconnects, its conversion is interrupted and its slot goes to a request that is still waiting. DuckDB stops once the row group it is writing is done.

• Synchronous conversion keeps the API trivial, but the client holds one connection open for the whole conversion, possibly minutes. Clients must stream in both directions and disable short timeouts.
• PreserveRowOrder stays on by default. Turning it off is the single biggest speed-up (about 20% off one conversion, per the README) but reorders rows, so it is an opt-in.
• zstd compression is the default. Snappy is faster but writes a noticeably larger file.
• Type detection samples roughly the first 20,000 rows, so a column whose type changes later makes DuckDB reject the file with a 422.
• IIS caps uploads at about 4 GB. Larger files need Windows Service mode.
• The free-space check runs per request, so requests that arrive together can all pass it against the same free space. The project's plan lists this as an open item.

The suite has 41 xUnit tests across four layers:

• In-memory host: a valid CSV returns typed Parquet, an empty body returns 400, a CSV DuckDB rejects returns 422, and the OpenAPI document and Swagger UI are served.
• Real Kestrel: a CSV of about 100 MB streams in and out with a matching row count, and an upload over MaxUploadBytes returns 413.
• Parallel requests: eight concurrent requests each get their own data back, excess requests get 503 with Retry-After, a queue timeout returns 503, and a client that disconnects frees its slot and leaves no temp files.
• Options and converter: every unusable setting fails at startup naming the setting, including a codec crafted to inject SQL, and a conversion cancelled at any moment throws OperationCanceledException within a second.

Run for this write-up: dotnet test with .NET SDK 10.0.302 on macOS completed with 41 passed, 0 failed, 0 skipped, including the large-file test.

Reported in the repository's README, comparing the parallel-request design against the previous behaviour (a fresh DuckDB instance per request, COPY on a thread-pool thread, two conversions at once):

The last two rows are the point: converting no longer holds thread-pool threads, so the rest of the service stays responsive while the CPU is busy. Raw throughput of eight simultaneous conversions on four cores is unchanged (2,511 ms against 2,589 ms, within run-to-run noise) because at that point the CPU is the limit. The plan also records interrupt latency over a 100 MB CSV at default settings: 57 ms typical and 617 ms worst.

The repository publishes no peak-memory or large-file figures, and bounded memory is the central claim, so this was measured for this write-up with the same endpoint and a 3.2 GB input:

By default DuckDB may use about 80% of RAM, so the 1.8 to 2.8 GB peak is not a bound. With an explicit 1 GB limit the process stayed under 0.9 GB for the same input at similar speed and the same output size, so the configured limit, not the file size, sets the ceiling. The idle service sits at about 127 MB, and the work folder was empty after a clean shutdown. The CSV compressed 5.3 times, which depends on the data and should not be generalised.

Method: a seeded synthetic CSV (7 columns: integer id, account string, date, decimal amount, currency, status and a three-word note) uploaded with curl -T over loopback to the Release build on an Apple M4 (10 cores, 16 GB, macOS 27.0.1, .NET SDK 10.0.302). Wall time is curl's total, covering upload to disk, conversion and download. Memory is the process's resident set sampled every 100 ms, so very short peaks could be missed. This is a single machine and a handful of runs: indicative, not a controlled benchmark. The generator script is not part of the repository.

• A cancel that lands while DuckDB is still binding the statement surfaces as an INTERRUPT error, not OperationCanceledException, so an aborted client briefly received a 422. The converter now treats any failure after its token fires as cancellation, and because Cancel() only interrupts a statement DuckDB has started, the interrupt repeats every 50 ms until the COPY returns.
• A review of the parallel-requests work raised nine findings. Each was checked against DuckDB before acting: seven were real and fixed, one was a clean-up, and one is left open (the per-request disk check).
• The headline result of the parallelism work is responsiveness, not throughput, which stayed CPU-bound.
• Not measured: peak memory under several concurrent multi-gigabyte uploads (the measurement above is one request at a time), the cost of higher compression levels, and behaviour behind a real reverse proxy.