Roman Karlstetter created ARROW-13024:
-----------------------------------------
Summary: Decoding byte stream split encoded parquet columns fails
when file has nulls
Key: ARROW-13024
URL: https://issues.apache.org/jira/browse/ARROW-13024
Project: Apache Arrow
Issue Type: Bug
Components: C++, Parquet
Affects Versions: 4.0.0, 3.0.0, 2.0.0
Reporter: Roman Karlstetter
Reading from a parquet file fails with the following error
{{Data size too small for number of values (corrupted file?)}}.
This happens for the case when there is a {{BYTE_STREAM_SPLIT}}-encoded column
which has less values stored than number of rows, which is the case when the
column has null values (definition levels are present).
The problematic part is the condition checked in
{{ByteStreamSplitDecoder<DType>::SetData}}, which raises the error if the
number of values does not match the size of the data array.
I'm unsure whether I have enough experience with the internals of the
encoding/decoding part of this implementation to fix this issue, but my
suggestion would be to initialize {{num_values_in_buffer_}} with
{{len/static_cast<int64_t>(sizeof(T))}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
