PinkCrow007 opened a new issue, #3934: URL: https://github.com/apache/parquet-java/issues/3934
There is an interoperability issue between Parquet implementations when reading DataPageV2 pages encrypted with Parquet Modular Encryption (PME) that have non-empty repetition or definition levels. Spark fails to read Arrow-written PME files (throwing Negative ciphertext length), and Arrow fails to read Spark-written files (throwing Tag mismatch). Arrow and parquet-java handle encryption of the DataPageV2 body differently: - **parquet-java**: Leaves **rep/def levels in plaintext** and encrypts only the values. Code pointer: [ColumnChunkPageWriteStore.java](https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ColumnChunkPageWriteStore.java) writePageV2 - **Arrow C++ and arrow-rs**: **Encrypt the entire page body**, including both rep/def levels and values. Code pointer: [column_writer.cc](https://github.com/apache/arrow/blob/main/cpp/src/parquet/column_writer.cc) BuildDataPageV2 To ensure interoperability and prevent metadata leakage, shall we align with Arrow by encrypting the entire DataPageV2 buffer? Here is a python script [parquet_modular_encryption_interop_test.py](https://github.com/user-attachments/files/34394744/parquet_modular_encryption_interop_test.py) to reproduce this with PyArrow 17.0.0 and Spark 4.1 (parquet-java 1.16.0). The results are: ``` == V1 data pages Arrow writes, Arrow reads: OK Spark writes, Spark reads: OK Arrow writes, Spark reads: OK Spark writes, Arrow reads: OK == V2 data pages Arrow writes, Arrow reads: OK Spark writes, Spark reads: OK Arrow writes, Spark reads: FAILED: AEADBadTagException: Tag mismatch Spark writes, Arrow reads: FAILED: Negative ciphertext length -541098369 ``` Also posted in [apache/parquet-format#636](https://github.com/apache/parquet-format/issues/636) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
