Re: Modular encryption to support arrays and nested arrays

2022-11-02 Thread nicolas paris
Thanks for your help. > > The goal is to make the exception print something like: > *Caused by: org.apache.parquet.crypto.ParquetCryptoRuntimeException: > Encrypted column [rider] not in file schema column list: [foo] , > [rider.list.element.foo] , [rider.list.element.bar] , [ts] , [uuid]* >

Re: Modular encryption to support arrays and nested arrays

2022-10-31 Thread Gidon Gershinsky
Parquet columnar encryption supports these types. Currently, it requires an explicit full path for each column to be encrypted. Your sample will work with *spark.sparkContext.hadoopConfiguration.set("parquet.encryption.column.keys", "k2:rider.list.element.foo,rider.list.element.bar")* Having said

Modular encryption to support arrays and nested arrays

2022-10-28 Thread nicolas paris
Hello, apparently, modular encryption does not yet support **arrays** types. ```scala spark.sparkContext.hadoopConfiguration.set("parquet.crypto.factory.class", "org.apache.parquet.crypto.keytools.PropertiesDrivenCryptoFactory")