dwsmith1983 commented on PR #5565:
URL:
https://github.com/apache/datafusion-comet/pull/5565#issuecomment-5535481011
Thanks for the careful pass. All four are in d21b24eb9:
1. EOF now releases the decoder. Every store of an input batch goes through
one `receive` point on the scan input state, and `InputBatch::EOF` resets the
decode context there, so a finished input no longer keeps its zstd workspace
while the task continues. Pinned by `eof_releases_retained_decoder_workspace`.
2. Batch and decoder live under a single `Arc<Mutex<ScanInputState>>`. The
separate decoder mutex is gone, and the `batch` field is no longer `pub`
(nothing outside the module used it). Sharing across `execute()` clones is
unchanged.
3. The decode benchmark now covers {400, 8192} rows x {zstd 1, zstd 3, lz4,
none}, each with a context that outlives iterations, one locked per frame
behind the same `Arc<Mutex<_>>` shape the operator uses, one created per reader
lifetime of 64 frames and dropped (so startup and cleanup are measured), and a
fresh context per frame. On this machine the per-frame context create and drop
is about 2 percent of decode time at 400 rows and within noise at 8192 rows,
and the lock is not measurable. That matches the earlier framing: the decode
side is about retention bounds, not throughput. A benchmark through
`ShuffleScanExec` itself is not possible without a JVM, since its decode path
runs inside `with_env` and fetches blocks through JNI; the locked variant is
the closest stand-in and is labeled as such.
4. The reuse test now builds a fresh `ShuffleDecodeContext` for every
reference decode and asserts the reused side created exactly one context across
all frames, so it compares what it says it compares.
The 32 KiB per-frame output buffer in the zstd writer is noted as a separate
follow-up; I did not touch it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
