[ https://issues.apache.org/jira/browse/PARQUET-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17711367#comment-17711367 ]
Steve Loughran commented on PARQUET-1989: ----------------------------------------- you might want to have a design which can do the scan on a spark rdd, where the rdd is simply the deep listFiles(path) scan of the directory tree. This would give the best scale for a massive dataset compared to even some parallelised scan in a single process. I do have an RDD which can do line-by-line work, with locality of work determined on each file, which lets you schedule the work on the relevant hdfs nodes with the data; unfortunately it needs to be in the o.a.spark package to build https://github.com/hortonworks-spark/cloud-integration/blob/master/spark-cloud-integration/src/main/scala/org/apache/spark/cloudera/ParallelizedWithLocalityRDD.scala ...that could maybe be added to spark itself. > Deep verification of encrypted files > ------------------------------------ > > Key: PARQUET-1989 > URL: https://issues.apache.org/jira/browse/PARQUET-1989 > Project: Parquet > Issue Type: New Feature > Components: parquet-cli > Reporter: Gidon Gershinsky > Assignee: Maya Anderson > Priority: Major > Fix For: 1.14.0 > > > A tools that verifies encryption of parquet files in a given folder. Analyzes > the footer, and then every module (page headers, pages, column indexes, bloom > filters) - making sure they are encrypted (in relevant columns). Potentially > checking the encryption keys. > We'll start with a design doc, open for discussion. -- This message was sent by Atlassian Jira (v8.20.10#820010)