On Wed, Nov 29, 2023 at 5:40 PM Priyanshu Sharma <[email protected]> wrote:
> With Parquet Modular Encryption > 1. With each key rotation , Is it possible to avoid encryption and > decryption of existing data? > Yes > > 2. If master key rotation does not require modification of the data file > then how would the KMS work. > - Basic key rotation simply means the master key version is updated in the KMS, so the future parquet files are encrypted with the rotated master key (namely, their data keys will be encrypted with the new master key version). - In addition, if your threat model requires to re-wrap data keys of existing parquet files with the rotated master key - this can be done without modification of the parquet files if they were encrypted in an "external key material" mode, parquet.encryption.key.material.store.internally=false (see https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/README.md#class-propertiesdrivencryptofactory ), the data keys (encrypted with master keys in KMS) are stored in separate small key_material files. The key re-wrapping will re-encrypt the data keys with the rotated master key, and replace the key_material files. > > 3. Do we have any constraints for key structure while updating a key. > This is up to the KMS service implementation. > > It would be better if you could provide a git link having the interface to > implement KMS. I am already following this git page > https://github.com/apache/parquet-format/blob/master/Encryption.md but > still have a few doubts. > The links and basic details can be found in https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#columnar-encryption Cheers, Gidon
