arnabnandy7 commented on issue #3712: URL: https://github.com/apache/parquet-java/issues/3712#issuecomment-5280746492
@weijietong the ZSTD frame contains 3125 uncompressed bytes, while the Parquet page header says 3127. The embedded RLE metadata also expects two more bytes than are present. This suggests the bytes were lost during compression or while the ZSTD frame was finalized, rather than being calculated incorrectly by the Parquet page writer. The file was written with zstd-jni 1.5.2-1, even though parquet-java 1.15.2 declares 1.5.6-6. Reading it with a newer zstd-jni version will not help because the two bytes were already missing when the file was created. Could you try writing the same data again with zstd-jni 1.5.6-6 or, preferably, 1.5.7-12? If the problem still occurs, a small input dataset or the affected page would help us reproduce it and add a regression test. Unfortunately, changing only `uncompressed_page_size` does not repair the file because the page contents are also truncated. The affected files will need to be regenerated from their source data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
