schenksj commented on issue #11147: URL: https://github.com/apache/arrow-rs/issues/11147#issuecomment-6562789424
+1 from another downstream reader. We serve rows from Parquet files beside a search index, and for low-cardinality columns we'd like to evaluate filters on `RLE_DICTIONARY` pages at the run level: match dictionary IDs, then intersect runs of matching rows. That avoids materializing every value. Today the only route is the `experimental` feature (`encodings::rle::RleDecoder`), which has no stability guarantee, as with the other uses described here. We'd adopt it behind an exact version pin and an adapter, but a supported seam would be much better. Of the two options in the issue, option 2 (a public, validated value-section dictionary-index decoder with an explicit count contract) would cover our use. A run-oriented variant that yields `(dictionary_id, run_length)` pairs, not one ID per value, would cover it even better. Validation matters to us too. A supported decoder that returns errors rather than panicking on corrupt run headers or out-of-range indices (#11516, #11689) would let us call it without a panic guard. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
