sunchao commented on code in PR #6098: URL: https://github.com/apache/datafusion-comet/pull/6098#discussion_r4088570760
########## native/shuffle/src/ipc.rs: ########## @@ -46,11 +46,25 @@ const CONTINUATION_MARKER: [u8; 4] = [0xff; 4]; /// from several shuffles, and a single entry would thrash. const SCHEMA_CACHE_CAPACITY: usize = 4; +/// Maximum estimated serialized-plus-parsed size of all cached schemas on a thread, excluding +/// allocator overhead. A shared 4 MiB budget keeps retention modest while allowing wide schemas +/// to use space left by other entries: 8,000 short-named Int32 fields need about 1.4 MiB with +/// Arrow 59 on a 64-bit target. Both field count and names/metadata contribute to this estimate. +const SCHEMA_CACHE_RETAIN_LIMIT: usize = 4 << 20; Review Comment: Raised the shared budget to **16 MiB** in [28bbe25c3](https://github.com/apache/datafusion-comet/pull/6098/commits/28bbe25c3c0517dd7845e36a3f5e3750795bfa67), keeping the four-entry cap. The docs now explain the headroom for interleaved wide schemas and the separate 1 MiB scratch limit. Added a regression with four distinct 8,000-column schemas decoded round robin for ten rounds, across all four codecs and both decoder entry points. It ends at **36 hits / 4 misses** for every combination. I first ran it with the old 4 MiB budget: round two failed with 0 hits / 8 misses instead of 4 / 4. With 16 MiB, all **157 shuffle tests** pass, including the resized boundary/eviction tests; all-target Clippy with `-D warnings`, Rustfmt, and `git diff --check` also pass. ########## native/shuffle/benches/shuffle_reader.rs: ########## @@ -132,6 +132,12 @@ fn criterion_benchmark(c: &mut Criterion) { } } + // Wide schemas exceed 1 MiB when both serialized and parsed metadata are counted. Review Comment: Updated in [28bbe25c3](https://github.com/apache/datafusion-comet/pull/6098/commits/28bbe25c3c0517dd7845e36a3f5e3750795bfa67): the comment now says that this 8,000-column schema uses about **1.4 MiB of the serialized-plus-parsed cache budget**. It explains that the warm/cold comparison guards against cache-admission cutoffs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
