Mihaly Szjatinya has uploaded a new patch set (#8). (
http://gerrit.cloudera.org:8080/24579 )
Change subject: IMPALA-15139: Support DELTA_LENGTH_BYTE_ARRAY Parquet encoding
......................................................................
IMPALA-15139: Support DELTA_LENGTH_BYTE_ARRAY Parquet encoding
Add read support for the DELTA_LENGTH_BYTE_ARRAY Parquet encoding
(encoding enum value 6) for BYTE_ARRAY columns
(STRING, VARCHAR, BINARY).
The new ParquetDeltaLengthByteArrayDecoder class decodes the lengths
section using the existing ParquetDeltaDecoder<int32_t> infrastructure,
then slices bytes zero-copy from the page buffer into StringValues.
All string lengths are decoded eagerly on NewPage() so that subsequent
NextValue() / NextValues() / SkipValues() calls need only index into
the pre-decoded lengths vector.
Changes:
- be/src/exec/parquet/parquet-delta-length-byte-array-decoder.h/.cc:
New decoder class with NewPage, NextValue, NextValues, SkipValues,
and GetTotalValueCount.
- be/src/exec/parquet/parquet-metadata-utils.cc: Whitelist
DELTA_LENGTH_BYTE_ARRAY and DELTA_BINARY_PACKED in
IsEncodingSupported().
DELTA_BINARY_PACKED is listed in column chunk metadata by some writers
when encoding BYTE_ARRAY columns with DELTA_LENGTH_BYTE_ARRAY.
- be/src/exec/parquet/parquet-column-readers.h/.cc: Wire the new decoder
into InitDataDecoder (StringValue/BYTE_ARRAY specialization),
ReadValue, MaterializeValueBatch, SkipEncodedValuesInPage,
DecodeValue, and DecodeValues dispatch paths.
- be/src/exec/parquet/parquet-delta-decoder.cc: Add missing
miniblocks_in_block_ == 0 guard before the modulo check to avoid
undefined behaviour on corrupt input.
- be/src/exec/parquet/CMakeLists.txt: Register new source and
test target.
- testdata/parquet_delta_length_byte_array_encoding/: Pre-generated test
Parquet file and pyarrow generator script.
- tests/query_test/test_parquet_delta_length_byte_array_encoding.py:
E2E test suite.
- testdata/bin/minicluster_trino/: Remove session-property-config.json
and session-property-config.properties (they disabled
DELTA_LENGTH_BYTE_ARRAY in Trino as a workaround); add .gitignore for
the generated *-site.xml build-time inputs.
- tests/custom_cluster/test_trino_interop.py: Add
test_impala_reads_trino_parquet: Trino writes a Parquet table with
VARCHAR columns (Trino 482 uses DELTA_LENGTH_BYTE_ARRAY by default)
and Impala reads it back.
Testing:
Unit tests (13/13 passed):
parquet-delta-length-byte-array-decoder-test
E2E tests (10/10 passed, with/without codegen):
tests/query_test/test_parquet_delta_length_byte_array_encoding.py
Parquet regression tests (315 passed, 14 expected skips):
tests/query_test/test_scanners.py::TestParquet
tests/query_test/test_scanners.py::TestParquetV2
tests/query_test/test_parquet_stats.py
Trino interop tests (3/3 passed):
tests/custom_cluster/test_trino_interop.py
Iceberg-Trino interop tests (3/3 passed):
tests/custom_cluster/test_iceberg_trino_interop.py
Change-Id: I9e4395fa30cd7fb2c89ef5a2b16b043d4de60b52
Assisted-by: Claude Sonnet 4.6 (GitHub Copilot)
---
M be/src/exec/parquet/CMakeLists.txt
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-delta-decoder.cc
M be/src/exec/parquet/parquet-delta-decoder.h
A be/src/exec/parquet/parquet-delta-length-byte-array-decoder-test.cc
A be/src/exec/parquet/parquet-delta-length-byte-array-decoder.cc
A be/src/exec/parquet/parquet-delta-length-byte-array-decoder.h
M be/src/exec/parquet/parquet-metadata-utils.cc
A testdata/bin/minicluster_trino/.gitignore
M testdata/bin/minicluster_trino/Dockerfile
D testdata/bin/minicluster_trino/session-property-config.json
D testdata/bin/minicluster_trino/session-property-config.properties
A testdata/parquet_delta_length_byte_array_encoding/README
A
testdata/parquet_delta_length_byte_array_encoding/mixed_types_delta_length_byte_array.parquet
A testdata/parquet_delta_length_byte_array_encoding/parquet_files_generator.py
M tests/custom_cluster/test_trino_interop.py
A tests/query_test/test_parquet_delta_length_byte_array_encoding.py
17 files changed, 842 insertions(+), 43 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/24579/8
--
To view, visit http://gerrit.cloudera.org:8080/24579
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9e4395fa30cd7fb2c89ef5a2b16b043d4de60b52
Gerrit-Change-Number: 24579
Gerrit-PatchSet: 8
Gerrit-Owner: Mihaly Szjatinya <[email protected]>
Gerrit-Reviewer: Balazs Hevele <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Mihaly Szjatinya <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
