alamb opened a new pull request, #12165: URL: https://github.com/apache/arrow-rs/pull/12165
# Which issue does this PR close? - None filed yet # Rationale for this change Writing encrypted bloom filters is not implemented in the Rust parquet writer. Today, when bloom filters are enabled together with encryption, the writer silently writes the bloom filter of an encrypted column in **plaintext**. A bloom filter reveals information about the values of the column, so this leaks data the user asked to have encrypted. The C++ writer raises an explicit "not yet implemented" error in this case rather than writing the filter: https://github.com/apache/arrow/blob/87eb47ece82003145ffff6b6b9b64321af996649/cpp/src/parquet/file_writer.cc#L523-L525 # What changes are included in this PR? - `SerializedFileWriter::new` (and therefore `ArrowWriter` / `AsyncArrowWriter`) now returns `ParquetError::NYI` if a bloom filter is enabled for any encrypted column. The error names the offending column. - Document the restriction on `WriterPropertiesBuilder::with_file_encryption_properties`. Two deliberate differences from the C++ writer: 1. The error is raised when the writer is created, rather than when the bloom filters are flushed at the end of the file, so no data is written before the failure. 2. The error is only raised for *encrypted* columns. With column keys, columns without a key are stored in plaintext, and a plaintext bloom filter for such a column leaks nothing new and is valid per the spec, so that case keeps working. The C++ writer rejects any bloom filter when encryption properties are set. I am happy to match that instead if reviewers prefer the simpler rule. # Are these changes tested? Yes, new tests in `parquet/tests/encryption/encryption.rs`: - `test_write_bloom_filter_for_encrypted_column_errors`: uniform encryption (encrypted and plaintext footer) and column keys, with bloom filters enabled globally and per column. - `test_write_bloom_filter_for_unencrypted_column`: a bloom filter on an unencrypted column of an encrypted file is still written and can be read back. The existing `write_and_read_stats` helper enabled bloom filters on encrypted columns, so it no longer does so. # Are there any user-facing changes? Yes: writing a file with encryption properties and bloom filters enabled for an encrypted column now returns an error, where it previously succeeded and wrote the bloom filter unencrypted. Users must disable bloom filters for encrypted columns. Note that a per-column `set_column_bloom_filter_enabled(col, false)` does not currently override a global `set_bloom_filter_enabled(true)`, so to combine the two, bloom filters must be enabled per column for the unencrypted columns only. That is existing behavior and not changed here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
