autoCAT6 opened a new pull request, #53947:
URL: https://github.com/apache/arrow/pull/53947
### Rationale for this change
Fixes #49821. The direct-key helpers currently support only uniform
encryption, so callers managing their own keys cannot encrypt selected columns
with separate keys through these APIs.
### What changes are included in this PR?
Add an optional `column_keys` dictionary mapping full Parquet column paths
to raw keys to `create_encryption_properties()` and
`create_decryption_properties()`. Reuse the existing native column-property
builders; omitting the dictionary preserves uniform encryption. With a nonempty
encryption dictionary, unlisted columns remain unencrypted. Reject empty
dictionaries to avoid silently falling back to uniform encryption.
Document the distinction from `EncryptionConfiguration.column_keys` and the
use of full nested leaf paths.
### Are these changes tested?
- Built Arrow C++ and PyArrow from the contribution source on macOS arm64.
- Against the unchanged implementation, the direct-key selection has **19
existing tests passing and all 20 new tests failing**. With the change, **39
direct-key tests pass**.
- The complete Python Parquet and Dataset encryption selection has **82
passing tests and 1 expected skip** for the test requiring a build without
encryption support.
- The native Parquet encryption suite has **33 passing tests**.
- Repository-configured Python and Cython lint and `git diff --check` pass.
Coverage includes different column-key lengths, encrypted/plaintext footers,
stored/external AAD prefixes, partial projection, missing and incorrect keys,
nested leaf paths, unknown columns, and argument validation.
### Are there any user-facing changes?
Callers can supply per-column keys without implementing a KMS client:
```python
column_keys = {"sensitive": b"abcdef0123456789"}
encryption_properties = pe.create_encryption_properties(
footer_key=b"0123456789abcdef", column_keys=column_keys
)
decryption_properties = pe.create_decryption_properties(
footer_key=b"0123456789abcdef", column_keys=column_keys
)
```
The footer key remains required. Readers need keys only for the encrypted
columns they read, and callers remain responsible for raw-key management.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
