weijietong opened a new issue, #3712:
URL: https://github.com/apache/parquet-java/issues/3712
### Describe the bug, including details regarding any error messages,
version, and platform.
Summary
A corrupt Parquet file produced by a Spark writer (parquet-mr based,
parquet 1.15.x, zstd-jni 1.5.2-1) contains data pages whose page header
uncompressed_page_size is recorded 2 bytes larger than the bytes the
decompressor actually produces. Reading
those pages throws EOFException at page decompression / level decoding
time.
Environment
- Writer: Apache Spark (parquet-mr 1.15.2 vendored dependencies), ZSTD
compression, dictionary encoding enabled (default).
- zstd-jni: 1.5.2-1 at write time (Spark). parquet 1.15.2 declares
zstd-jni.version = 1.5.6-6.
- Reader: paimon vectorized reader (VectorizedColumnReader /
VectorizedRleValuesReader) using parquet-mr 1.15.2, zstd-jni 1.5.7-6.
- Schema column: ARRAY<ARRAY<BIGINT>> → leaf optional int64 element at
path *.list.element.list.element, maxRepetitionLevel=2, maxDefinitionLevel=5,
dictionary-encoded (PLAIN_DICTIONARY), V1 data pages (writer version
PARQUET_1_0).
Error messages
Reading the affected page fails with:
org.apache.parquet.io.ParquetDecodingException: could not decompress page
at
org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader$1.visit(ColumnChunkPageReadStore.java:212)
...
Caused by: java.io.EOFException
at java.base/java.io.DataInputStream.readFully(DataInputStream.java:202)
at
org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:399)
at org.apache.parquet.bytes.BytesInput.copy(BytesInput.java:205)
at
org.apache.parquet.hadoop.CodecFactory$HeapBytesDecompressor.decompress(CodecFactory.java:178)
at
org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader$1.visit(ColumnChunkPageReadStore.java:178)
After patching the header's uncompressed_page_size (3127 → 3125) so
decompression succeeds, reading still fails, but inside level decoding:
java.io.EOFException
at
org.apache.parquet.bytes.SingleBufferInputStream.sliceBuffers(SingleBufferInputStream.java:134)
at
org.apache.parquet.bytes.ByteBufferInputStream.sliceStream(ByteBufferInputStream.java:116)
at
org.apache.paimon.format.parquet.reader.VectorizedRleValuesReader.initFromPage(VectorizedRleValuesReader.java:108)
at org.apache.parquet.column.page.DataPageV1.accept(DataPageV1.java:134)
Reproducer / measurements
On the corrupt file (178 MB, 6 row groups, 1431 leaf columns), scanning
every page by independently decoding the thrift PageHeader and decompressing
the page body with zstd-jni directly (bypassing the reader), only 2 pages are
affected, both identical
in shape:
- row group 4, column 139 (55.list.element.list.element), data page 5
- row group 4, column 141 (57.list.element.list.element), data page 1
Both report:
header uncompressed_page_size = 3127
header compressed_page_size = 1877
valueCount = 2898
rlEnc=RLE dlEnc=RLE valEnc=PLAIN_DICTIONARY
zstd actual decompressed bytes = 3125 (delta = +2)
Decompressing with zstd-jni 1.5.2-1 and 1.5.7-6 both yield 3125, so this
is not a zstd-jni version artifact — the writer fed 3125 bytes to zstd but
recorded 3127 in the header.
When I decompressed according to 3125 and read the page content, I found
that the declared data length for the RLE part was 658, but the actual content
length was 656. It seems that the RLE part output 2 bytes less
### Component(s)
Core
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
