akoshelev commented on issue #47435: URL: https://github.com/apache/arrow/issues/47435#issuecomment-3242942517
Thanks @adamreeve, @rok, and @ggershinsky for responding to this thread. The reason we chose to use an externally managed metadata file was to support multiple file types with the same encryption scheme. While Parquet natively supports envelope encryption, other file formats like CSV do not. I think the low-level API exposed in Python would work for us because we can accomplish what we need in C++. However, I am not sure about [internal_key_material](https://github.com/apache/arrow/blob/ed77d25149569eb9a48f61f3694fd8ea4b9a411d/cpp/src/parquet/encryption/crypto_factory.h#L77-L81), as it still seems to require Pyarrow to manage encryption keys, and the format of this metadata file will not be the same as the one we use on our side. While this solution achieves the same goal in principle, it would require migrating the decryption-side code if we decide to use it. Overall, exposing a low-level API in Python would be the preferred approach for us. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
