djouallah opened a new issue, #11659: URL: https://github.com/apache/arrow-rs/issues/11659
**Describe the bug** Since parquet 59.0.0, `ArrowWriter` records each column chunk's `encoding_stats` with the dictionary page **last**, after the data pages. Up to 58.x the dictionary page came first. The file layout itself is unchanged (the dictionary page is still physically first), only the order of the `encoding_stats` entries in the footer changed. Readers depend on that order. Arrow C++ decides whether a column chunk is fully dictionary encoded, and so whether it can expose dictionary indices instead of decoding values (`ColumnWithExposeEncoding` / `RecordReaderWithExposeEncoding`), with [`IsColumnChunkFullyDictionaryEncoded`](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/parquet/file_reader.cc#L63-L88): ```cpp // The 1st page should be the dictionary page. if (encoding_stats[0].page_type != PageType::DICTIONARY_PAGE || ...) { return false; } ``` With the 59.x ordering this returns false for every column chunk parquet-rs writes, so those readers lose the dictionary fast path on files that are, in fact, fully dictionary encoded. **To Reproduce** ```rust use std::sync::Arc; use arrow_array::{RecordBatch, StringArray}; use parquet::arrow::ArrowWriter; fn main() { // 100k rows of a low-cardinality string column: one dictionary page, several data pages. let values: Vec<String> = (0..100_000).map(|i| format!("unit_{}", i % 650)).collect(); let batch = RecordBatch::try_from_iter([("s", Arc::new(StringArray::from(values)) as _)]).unwrap(); let props = parquet::file::properties::WriterProperties::builder() .set_data_page_size_limit(1024) .build(); let mut buf = Vec::new(); let mut w = ArrowWriter::try_new(&mut buf, batch.schema(), Some(props)).unwrap(); w.write(&batch).unwrap(); w.close().unwrap(); std::fs::write(std::env::args().nth(1).unwrap(), &buf).unwrap(); } ``` Run with `parquet` + `arrow-array` pinned to each version, then read the footer's `ColumnChunk.meta_data.encoding_stats` in file order: | parquet | `encoding_stats` (file order) | |---|---| | 58.3.0 | `DICTIONARY_PAGE PLAIN x1`, `DATA_PAGE RLE_DICTIONARY x98` | | 59.3.0 | `DATA_PAGE RLE_DICTIONARY x98`, `DICTIONARY_PAGE PLAIN x1` | `dictionary_page_offset` (4) and `data_page_offset` (7711) are the same in both, i.e. the dictionary page is still written first. **Expected behavior** The dictionary page's entry comes first in `encoding_stats`, as it did before 59.0.0 and as the pages are laid out in the file. **Additional context** The change came with the page-spilling writer in #10020 (data pages are produced before the dictionary page and the dictionary is moved to the front of the file afterwards, but the stats keep production order). #10062 mentions "the `page_encoding_stats` ordering difference" as spec-valid, and #10051, which added a test asserting the dictionary entry comes first, was closed. The spec does not mandate an order, but Arrow C++ relies on it, so this is a behaviour regression for files written by 59.x. There is no writer property to restore the old order (`defers_dictionary_ordering()` is always true on the Arrow path). A fix would be to move the dictionary page's entry to the front when the dictionary location is updated at column close. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
