aaronjyoakum commented on PR #11858: URL: https://github.com/apache/arrow-rs/pull/11858#issuecomment-6375312971
> 🤔 how much do we really save by lazily initialization here? For a writer with many files open at once and many columns, it makes a big difference Against our benchmarks: > A fresh `ZSTD_DCtx` is about 8.6 KB resident and a fresh `ZSTD_CCtx` with a level set is about 5.1 KB (libzstd 1.5.5), so roughly 13.7 KB per leaf column per open `ArrowWriter` before any data is written. With a ~300-column schema that is ~4 MiB per open writer. The DCtx part is never used on the write path. > Peak RSS (`VmHWM`), arrow-rs 59.3.0 vs 59.3.0 + this PR, jemalloc: | workload | released | with this PR | |---|---|---| | 5,300 writers open concurrently, 1 to 2 rows each, 300 leaf columns | 39.0 GiB | **15.4 GiB** | | 5 writers, ~1.2 M rows (steady state) | 1.1 GiB | 1.1 GiB | > Steady-state throughput was unchanged within noise over four alternating runs. The hot path only gains an `Option` check, because a normal column creates its compressor on its first page anyway. > With lazy init a tiny file's compressor appears once at close and is dropped with the writer, so contexts no longer coexist across thousands of open writers. That lifetime effect is most of the drop beyond the DCtx itself. Thanks for the PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
