DarkWanderer opened a new pull request, #10420:
URL: https://github.com/apache/arrow-rs/pull/10420
# Which issue does this PR close?
- Closes #9010.
Supersedes #9011 (closed as stale) with a reimplementation against current
main.
# Rationale for this change
Some Parquet writers (e.g. Grafana Tempo) use a low-cardinality string
column's dictionary page as a makeshift row-group index: if a required
value is not present in the dictionary, the whole row group can be
skipped without reading any data pages. Readers need a way to decode a
dictionary page on its own, independent of the row-by-row array reader,
to make use of this.
# What changes are included in this PR?
- `parquet::file::metadata::dictionary::decode_dictionary_page`: decodes
a BYTE_ARRAY dictionary page (Thrift header parse, decompress, PLAIN
decode) into a `Utf8`/`Binary` Arrow array.
- `ParquetMetaDataReader::read_column_dictionary` (sync) and
`read_column_dictionary_async` (async) to fetch and decode a given
row group/column's dictionary page from a `ParquetMetaData`, returning
`Ok(None)` if the chunk has no dictionary page.
- `ParquetRecordBatchStreamBuilder::get_row_group_column_dictionary`
convenience method mirroring `get_row_group_column_bloom_filter`.
# Are these changes tested?
Yes: a round-trip unit test for a dictionary-encoded string column, a
non-BYTE_ARRAY rejection test, and sync + async reader tests that
decode a real dictionary page written through `ArrowWriter`.
# Are there any user-facing changes?
Yes, purely additive: three new public APIs (see above). No changes to
any existing API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
