----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70676/ -----------------------------------------------------------
(Updated 五月 18, 2019, 3:28 p.m.) Review request for ranger and Don Bosco Durai. Bugs: https://issues.apache.org/jira/browse/RANGER-2128 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/RANGER-2128 Repository: ranger Description ------- Ranger Spark SQL Plugin for Spark SQL/Dataframe/Dataset Diffs ----- .gitignore c2def9d agents-audit/pom.xml 4aaec09 agents-common/pom.xml 4486af3 agents-cred/pom.xml c622683 agents-installer/pom.xml 20902cb credentialbuilder/pom.xml 461dcd0 embeddedwebserver/pom.xml a8c7cba hbase-agent/pom.xml c044ef0 hive-agent/pom.xml 0a091b2 jisql/pom.xml 07fb08b kms/pom.xml 043c67f knox-agent/pom.xml 0c45c84 plugin-atlas/pom.xml 87ff40d plugin-elasticsearch/pom.xml b997310 plugin-kafka/pom.xml 132baf7 plugin-kms/pom.xml 1940f41 plugin-kylin/pom.xml 69badfc plugin-nifi-registry/pom.xml 3e99f04 plugin-nifi/pom.xml c510d87 plugin-solr/pom.xml 619cc9e plugin-spark/.gitignore PRE-CREATION plugin-spark/pom.xml PRE-CREATION plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/RangerSparkAccessRequest.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/RangerSparkAuditHandler.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/RangerSparkAuthorizer.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/RangerSparkPlugin.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/RangerSparkResource.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/RangerSparkSQLExtension.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/SparkAccessControlException.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/SparkAccessType.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/SparkObjectType.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/SparkOperationType.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/SparkPrivObjectActionType.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/SparkPrivilegeObject.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/SparkPrivilegeObjectType.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/ranger/authorization/spark/authorizer/authorizer.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/spark/sql/AuthzUtils.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RangerSparkAuthorizerExtension.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RangerSparkMaskingExtension.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RangerSparkOptimizer.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RangerSparkRowFilterExtension.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/RangerSparkMasking.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/RangerSparkRowFilter.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/spark/sql/execution/RangerShowDatabasesCommand.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/spark/sql/execution/RangerShowTablesCommand.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/spark/sql/execution/RangerSparkPlanOmitStrategy.scala PRE-CREATION plugin-spark/src/main/scala/org/apache/spark/sql/hive/PrivilegesBuilder.scala PRE-CREATION plugin-spark/src/test/resources/data/files/kv1.txt PRE-CREATION plugin-spark/src/test/resources/log4j.properties PRE-CREATION plugin-spark/src/test/resources/ranger-spark-security.xml PRE-CREATION plugin-spark/src/test/resources/sparkSql_hive_jenkins.json PRE-CREATION plugin-spark/src/test/resources/tpcds/q1.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q10.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q11.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q12.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q13.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q14a.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q14b.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q15.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q16.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q17.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q18.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q19.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q2.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q20.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q21.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q22.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q23a.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q23b.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q24a.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q24b.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q25.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q26.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q27.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q28.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q29.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q3.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q30.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q31.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q32.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q33.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q34.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q35.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q36.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q37.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q38.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q39a.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q39b.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q4.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q40.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q41.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q42.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q43.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q44.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q45.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q46.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q47.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q48.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q49.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q5.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q50.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q51.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q52.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q53.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q54.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q55.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q56.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q57.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q58.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q59.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q6.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q60.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q61.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q62.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q63.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q64.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q65.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q66.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q67.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q68.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q69.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q7.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q70.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q71.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q72.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q73.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q74.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q75.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q76.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q77.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q78.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q79.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q8.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q80.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q81.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q82.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q83.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q84.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q85.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q86.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q87.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q88.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q89.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q9.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q90.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q91.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q92.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q93.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q94.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q95.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q96.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q97.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q98.sql PRE-CREATION plugin-spark/src/test/resources/tpcds/q99.sql PRE-CREATION plugin-spark/src/test/scala/org/apache/ranger/services/spark/RangerAdminClientImpl.scala PRE-CREATION plugin-spark/src/test/scala/org/apache/ranger/services/spark/SparkRangerAuthorizerTest.scala PRE-CREATION plugin-spark/src/test/scala/org/apache/spark/sql/RangerSparkTestUtils.scala PRE-CREATION plugin-spark/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RangerSparkAuthorizerExtensionTest.scala PRE-CREATION plugin-spark/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RangerSparkMaskingExtensionTest.scala PRE-CREATION plugin-spark/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RangerSparkRowFilterExtensionTest.scala PRE-CREATION plugin-spark/src/test/scala/org/apache/spark/sql/execution/RangerSparkPlanOmitStrategyTest.scala PRE-CREATION plugin-sqoop/pom.xml af00e97 plugin-yarn/pom.xml f273569 pom.xml 2a2540a ranger-atlas-plugin-shim/pom.xml 86d6a26 ranger-elasticsearch-plugin-shim/pom.xml 59a653b ranger-examples/pom.xml fa4cb20 ranger-examples/sampleapp/pom.xml add9546 ranger-hbase-plugin-shim/pom.xml 9ed0aeb ranger-hdfs-plugin-shim/pom.xml f45621c ranger-hive-plugin-shim/pom.xml 26258d7 ranger-kafka-plugin-shim/pom.xml 5a7353f ranger-kms-plugin-shim/pom.xml 0b505f6 ranger-knox-plugin-shim/pom.xml db342b7 ranger-kylin-plugin-shim/pom.xml 4bb9f34 ranger-plugin-classloader/pom.xml 0aa71c3 ranger-solr-plugin-shim/pom.xml 46b7821 ranger-sqoop-plugin-shim/pom.xml 86ab432 src/main/assembly/plugin-spark.xml PRE-CREATION ugsync/pom.xml c30d8ca unixauthclient/pom.xml 4625925 unixauthpam/pom.xml 0f2a459 Diff: https://reviews.apache.org/r/70676/diff/1/ Testing (updated) ------- https://issues.apache.org/jira/browse/RANGER-2128 ## UTs add Unit Tests add TPCDS 99 queries as unit tests. Can be verified via: 1. cd plugin-spark 2. mvn clean package -Pspark-2.3 or mvn clean package -Pspark-2.4 ## Test manually 1. add the the shaded jar which can be found plugin-spark/target after mvn package to SPARK_HOME/jars 2. add/create ranger-spark-security.xml etc as we do for hive, then mv all the plugin configuration file to SPARK_HOME/jars 3. cd $SPARK_HOME 4. run bin/spark-sql --conf spark.sql.extensions=org.apache.ranger.authorization.spark.authorizer.RangerSparkSQLExtension **No need to configure HiveAuthorizerFactory etc in hive-site.xml** Thanks, Kent Yao
