L. C. Hsieh created SPARK-59853:
-----------------------------------
Summary: Accept a short last bit-packed run in the vectorized RLE
reader
Key: SPARK-59853
URL: https://issues.apache.org/jira/browse/SPARK-59853
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
The Parquet spec requires bit-packed runs in the RLE/bit-packing hybrid
encoding to be padded to a multiple of 8 values, but some non-compliant writers
write a last bit-packed run that is shorter than its padding, e.g. Polars
before https://github.com/pola-rs/polars/pull/13883 (see
https://github.com/pola-rs/polars/issues/13818, "Spark can't read parquet files
written by polars").
The vectorized reader (VectorizedRleValuesReader) rejects such a run, even when
its bytes cover every value that is read, while the non-vectorized reader reads
it: parquet-java's RunLengthBitPackingHybridDecoder reads min(numGroups *
bitWidth, available) bytes. Arrow C++ also accepts it since
https://github.com/apache/arrow/pull/47992.
The vectorized reader could accept it the way Arrow does: only for the last run
of the stream, and only the values fully covered by the remaining bytes, so
reading past them still fails.
This came up in the review of SPARK-59832
(https://github.com/apache/spark/pull/59117), which keeps the existing strict
behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
