edgarRd commented on issue #3712:
URL: https://github.com/apache/parquet-java/issues/3712#issuecomment-5735627459
We hit what looks like the same defect on an independent writer stack, with
a much larger loss than 2 bytes. Adding it as a second data point.
## Environment
- Writer: Apache Spark 3.1.1 on Java 8 (1.8.0_504), writing through Apache
Iceberg's Parquet writer (an Iceberg 1.6.1-based build). Footer created_by =
parquet-mr version 1.15.1 (build `c7257b8faff5699e13bbc781679dc03f48c1102a`).
- Compression ZSTD, V1 data pages, PARQUET_1_0 writer version, default heap
ByteBufferAllocator, page write checksums enabled.
- zstd-jni on the executor classpath was 1.4.8-1 (Spark 3.1's pinned
version; the parquet ZSTD codec ran against it).
- Readers that fail on the file: parquet-mr 1.13.1 (could not decompress
page → `EOFException` in `DataInputStream.readFully` from
`CodecFactory$HeapBytesDecompressor.decompress`), DuckDB (ZSTD Decompression
failure), Trino (out-of-bounds while decoding levels).
## The file
- `485,647,873 bytes, 46,000 rows, 8 row groups, 4,602 leaf columns`, all
ZSTD. Object size on storage matches the size recorded by the writer, so
nothing was truncated in transit.
- Exactly one column chunk is bad: row group 1 of a deeply nested optional
`BINARY` leaf under several levels of 3-level `LIST` groups
(`maxRepetitionLevel = 4, maxDefinitionLevel = 20`). The column is entirely
null in every row group (`null_count == num_values`), so the page carries only
repetition and definition levels: `rlEnc=RLE dlEnc=RLE valEnc=PLAIN`, no
dictionary page, no value bytes. The same column's pages in the other `7` row
groups are fine.
## The page
Header: `type=DATA_PAGE num_values=68593 compressed_page_size=21630
uncompressed_page_size=42334`, CRC set.
- CRC32 of the compressed bytes on disk == header crc, so the bytes are
exactly what the writer produced.
- The zstd payload is a single, complete, valid frame. `zstd -d`
decompresses it with no error to 40,756 bytes, i.e. 1,578 bytes short of
uncompressed_page_size.
- Inside the decompressed payload:
- Repetition-level section: length prefix says `19,368` bytes.
- Definition-level section: its `4-byte` length prefix is found at offset
`17,794`, length `22,958`, and `17794 + 4 + 22958 == 40756`, so the definition
levels are complete and end exactly at the payload end. Decoding them with bit
width `5` yields exactly `68,593` values, matching num_values.
- So the repetition-level section actually contains only `17,790` bytes
instead of the declared `19,368`. Note that `4 + 19368 + 4 + 22958 == 42334`,
which is exactly the header's uncompressed_page_size. The writer's size
bookkeeping was consistent with `19,368` rep-level bytes; the bytes that
reached the compressor were not.
- The rep-level bytes that are present are not just truncated. Walking the
RLE/bit-packed hybrid runs (bit width `3`): the run headers stay structurally
consistent all the way to byte `17,790` (`2,635` runs, last run ends exactly at
the section end), but starting at byte `~14,661` the decoded values include
`5`, `6`, `7` and, at the point the decoder gives up, `27`, which are
impossible for `maxRepetitionLevel = 4`. The first `~14.6 KB` decode as
plausible levels (only `0..3` present, like the healthy pages). The tail looks
like foreign or stale buffer content, not a shortened copy of the real data.
- A healthy page of the same column in the neighbouring row group has the
expected layout: rep section `19,364` bytes decoding to `~69.7k` values (values
`0..3` only), def section `23,319` bytes decoding to exactly `num_values`.
## What we tried
- Fed the healthy page's rep/def payload through `CodecFactory` ZSTD
compression (parquet-hadoop `1.15.1`) as `BytesInput.concat(fromInt(len), rl,
fromInt(len), dl, empty())`, with sizes varied around the real ones,
single-threaded and with `8` concurrent compressors sharing the `CodecFactory`,
on Java 8 and Java 17, with zstd-jni `1.4.8-1` and `1.5.7-6`. Thousands of
iterations, zero size or content mismatches. So, like @arnabnandy7, we could
not reproduce through the normal path.
## Takeaways that may help narrow it down
1. It is the repetition-level buffer only, in both reports. Definition
levels written immediately afterwards by the sibling encoder are intact.
2. The zstd stage is innocent: the frame is coherent and matches the CRC.
The `RunLengthBitPackingHybridEncoder`'s `CapacityByteArrayOutputStream`
already held fewer/different bytes than `size()` claimed when `writeAllTo` ran.
3. Dictionary encoding and value bytes are not required: our column is
all-null `PLAIN` with zero value bytes.
4. Loss is not always tiny. Ours is `~8%` of the rep-level section with a
garbled tail, consistent with a slab being overwritten or swapped rather than a
single missed write.
5. Different zstd-jni versions (`1.5.2-1` there, `1.4.8-1` here) and
different writer frameworks (Paimon/Spark there, Iceberg/Spark here) point at
parquet-column/parquet-common, not the integration layer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
