JonSnow1807 opened a new pull request, #12122: URL: https://github.com/apache/arrow-rs/pull/12122
# Which issue does this PR close? Closes #12115. # Rationale for this change The Parquet Arrow writer counts all rows when deciding whether to create a dictionary cache, even though only selected non-null values are encoded. This can allocate a cache for a batch containing mostly or entirely null values. Nulls can come from dictionary keys, dictionary values, or a parent array masking a child value. # What changes are included in this PR? Pass the complete selection’s non-null value count through the internal write_gather interface and use it for the dictionary-cache decision. The count covers the enclosing write batch, preserving profitable caching when the writer splits it into smaller mini-batches. Existing caches remain available for reuse across sparse and all-null writes. # Are these changes tested? Regression tests cover sliced inputs, null keys, null dictionary values, parent-masked values, and partially or entirely null selections. A dense control checks that mini-batching does not disable a profitable cache. A lifecycle test checks reuse across dense, sparse, empty, and all-null writes, followed by replacement with another profitable dictionary. The memory regression fails against the original implementation and passes with this change. Measurements used an Intel Core i5-14500 on Linux with Rust 1.99.1 and default compiler flags. Inputs were prepared outside the timed section. Four balanced before/after pairs used 50 Criterion samples, a one-second warmup, and a three-second measurement period. These figures are retained ArrowWriter::memory_size() bytes after one write to a fresh writer: | Rows | Dictionary values | Input | Before | After | |---:|---:|---|---:|---:| | 8,192 | 4,096 | Dense | 372,778 | 372,778 | | 8,192 | 4,096 | 50% null | 340,010 | 340,010 | | 8,192 | 4,096 | Approximately 99% null | 38,890 | 6,122 | | 8,192 | 4,096 | All null | 32,768 | 0 | | 131,072 | 65,536 | Approximately 99% null | 620,983 | 96,695 | | 131,072 | 65,536 | All null | 524,456 | 168 | The affected cases avoid 32 KiB or 512 KiB of new cache storage. These are retained writer-memory estimates, not process RSS. The geometric centers of the paired timing ratios were approximately 10% faster for the large 99%-null case and 52% faster for the large all-null case. The dense control was 1.85% slower overall, with individual pairs ranging from 0.89% to 2.51% slower. The 50%-null control was 0.98% slower overall. The benchmark additions were a separate local overlay and are not included in this implementation diff. # Are there any significant user-facing changes? No public API or intended result changes. AI assistance was used for implementation, regression tests, independent review, and validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
