arnabnandy7 commented on issue #3712: URL: https://github.com/apache/parquet-java/issues/3712#issuecomment-5331917564
@weijietong I confirmed that two bytes are missing from the final repetition-level bit-packed run. The ZSTD frame is valid and its stored CRC matches, so ZSTD compressed the page in its already-incomplete state. I also stress-tested the parquet-java 1.15.2 RLE encoder across 128,000 randomized inputs, including direct-buffer slab boundaries and the affected value count of 2898. In every case, the reported encoded size matched the serialized byte count, so I could not reproduce this through the normal encoder path. This may be a rare buffer lifecycle issue or a difference in Spark’s vendored writer path. It would be helpful to know which `ByteBufferAllocator` Spark used and whether the affected writer task reported retries, executor errors, or memory-pressure warnings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
