PinkCrow007 opened a new issue, #3934:
URL: https://github.com/apache/parquet-java/issues/3934

   There is an interoperability issue between Parquet implementations when 
reading DataPageV2 pages encrypted with Parquet Modular Encryption (PME) that 
have non-empty repetition or definition levels.
   
   Spark fails to read Arrow-written PME files (throwing Negative ciphertext 
length), and Arrow fails to read Spark-written files (throwing Tag mismatch).
   
   Arrow and parquet-java handle encryption of the DataPageV2 body differently:
   - **parquet-java**: Leaves **rep/def levels in plaintext** and encrypts only 
the values. Code pointer: 
[ColumnChunkPageWriteStore.java](https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ColumnChunkPageWriteStore.java)
 writePageV2
   - **Arrow C++ and arrow-rs**: **Encrypt the entire page body**, including 
both rep/def levels and values. Code pointer: 
[column_writer.cc](https://github.com/apache/arrow/blob/main/cpp/src/parquet/column_writer.cc)
 BuildDataPageV2
   
   To ensure interoperability and prevent metadata leakage, shall we align with 
Arrow by encrypting the entire DataPageV2 buffer?
   
   Here is a python script 
   
[parquet_modular_encryption_interop_test.py](https://github.com/user-attachments/files/34394744/parquet_modular_encryption_interop_test.py)
 to reproduce this with PyArrow 17.0.0 and Spark 4.1 (parquet-java 1.16.0). The 
results are:
   ```
   == V1 data pages
   Arrow writes, Arrow reads: OK
   Spark writes, Spark reads: OK
   Arrow writes, Spark reads: OK
   Spark writes, Arrow reads: OK
   
   == V2 data pages
   Arrow writes, Arrow reads: OK
   Spark writes, Spark reads: OK
   Arrow writes, Spark reads: FAILED: AEADBadTagException: Tag mismatch
   Spark writes, Arrow reads: FAILED: Negative ciphertext length -541098369
   
   ```
   Also posted in 
[apache/parquet-format#636](https://github.com/apache/parquet-format/issues/636)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to