aptro opened a new pull request, #12294:
URL: https://github.com/apache/arrow-rs/pull/12294

   # Which issue does this PR close?
   
   - Closes #12250.
   
   # Rationale for this change
   
   `create_codec` builds one `ZSTDCodec` per column chunk, and each codec owns 
its
   zstd contexts. After #11858 they are created lazily, but a reader still 
creates
   (and frees) one decompression context per column chunk it reads, and a writer
   holds one compression context per column. For a query engine reading many 
small
   column chunks, context creation is a measurable share of CPU: in our service
   (DataFusion over zstd Parquet), a uprobe counted ~582k `ZSTD_createDCtx` 
calls in
   110 s on 59.3.2, about 11% of query CPU.
   
   # What changes are included in this PR?
   
   As proposed in #12250: one decompressor per thread and one compressor per
   (thread, level), created on first use; `ZSTDCodec` keeps only its level. A 
zstd
   bulk call resets its session, so a reused context produces the same bytes as 
a
   fresh one. A re-entrant call (not possible today) falls back to a fresh 
context
   rather than panicking.
   
   # Are these changes tested?
   
   New test `test_codec_zstd_shared_contexts_match_fresh_contexts`: four 
threads,
   codecs at levels 1/3/9/19 interleaved, inputs from 0 B to 3 MiB; compressed 
bytes
   must equal a fresh `bulk::Compressor`'s, and round-trip. (Negative control: 
making
   the lookup ignore the level fails it.)
   
   Measured on 59.3.2 + this patch, same machine, alternated rounds:
   - `ZSTD_createDCtx`: 581,990 -> 852 per 110 s run; `ZSTD_createCCtx`: 
583,418 -> 2,603.
   - Query CPU per poll, small reads (8 columns): 9.63 -> 7.88 ms (-18%); large 
reads: 81 -> 57 ms (-30%). Latency p50 down in every row.
   
   # Are there any significant user-facing changes?
   
   No API change. Contexts now live for the thread's lifetime (bounded by 
threads x
   levels used, not by columns).
   
   # AI usage disclosure
   
   The change, the test and this description were drafted with Claude Code (an 
AI coding assistant), following the fix proposed in #12250; the measurements 
above were run with its help on our own deployment.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to