----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62411/#review185922 -----------------------------------------------------------
Ship it! Ship It! - Alexander Kolbasov On Sept. 21, 2017, 12:59 p.m., Sergio Pena wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62411/ > ----------------------------------------------------------- > > (Updated Sept. 21, 2017, 12:59 p.m.) > > > Review request for sentry, Alexander Kolbasov, Colm O hEigeartaigh, and > kalyan kumar kalvagadda. > > > Bugs: sentry-1958 > https://issues.apache.org/jira/browse/sentry-1958 > > > Repository: sentry > > > Description > ------- > > This patch bumps the Hive version of hive-authz1 to Hive 2.0. Moving to > authz2 has been a little complicated, so doing an incremental patch was > preffered. > Also, we're taking advantage of the unit tests nad e2e tests that already > exist on Sentry. There are tests that are not on the authz2, so this is why I > preffered to look into bumping the Hive 2.0 version first. > > The following issues with Hive 2.0 were found and addressed on Sentry: > - Hive 2.0.1 has a bug that Sentry cannot workaround. > - Hive 2.1.1 and higher has a different bug that Sentry cannot workaround. > - Hive CBO has a bug where ReadIdentity partitions do not have parents > causing Sentry to request extra privileges that the user might not have > CBO is disabled on the Sentry tests and it must be disabled on production > as well. > - HIVE-11145: Remove OFFLINE and NO_DROP from tables and partitions > Removed tests that use the protection mode operations as Hive do not > support them any more. > - HIVE-10453: HS2 leaking open file descriptors when using UDFs > Hive 2.0 clears all functions after a session is closed causing other users > who want to execute such > function to fail because they lost the function scope and they do not have > permissions to create functions > - HIVE-12320: hive.metastore.disallow.incompatible.col.type.changes should be > true by default > Sentry had some issues on the tests due to this Hive change. > - HIVE-10307: Support to use number literals in partition column > Hive 2.0 added an extra validation when using ALTER TABLE ... PARTITION > (spec) that throws an error if > spec is not a partition column. > - The HS2 webui fails to start when concurrency mode is enabled. The Sentry > tests are now configured to put > Hive in test mode so that the webui is not initialized. > - There are some column names that cannot be used as they are reservered by > Hive, i.e 'date' column name fails > in some Sentry tests. > - Hive 2.0 switched to log4j2 causing some Sentry tests to fail. > > Important changes on Sentry: > - Hive 2.0 has an authz1 bug with the use of SentryMetastoreFilterHook class. > This class is replaced automatically by Hive with a default one that uses > authz2. To make minimal changes on Sentry, a new class is created > (SentryHiveAuthorizerImpl) that only deals with metastore filtering. Also, > the class SentryHiveAuthorizerFactory is set by the > HiveAuthzBindingSessionHook automatically when a HS2 session is opened. > However, this new authorizer must be set manually on the hive-site.xml so > that other clients who use the HMS Client can use this new filter class (info > is mentioned on the SentryHiveAuthorizerFactory class. > > There will be follow-up patches to fix the following: > - Remove SentryMetastoreFilterHook and improve the SentryHiveAuthorizerImpl > to avoid the overhead of converting a list of hive privileges to a list of > strings and viceversa. > - Merge authz1 and authz2 dependencies. > - Review current authz2 and merge code into the new changes. > > > Diffs > ----- > > dev-support/test-patch.py ac91b590c93884911d27928ac60108dcaa5d81ed > pom.xml 53679f90b96bbd4db5f0ac004d0de542e439c7f7 > sentry-binding/sentry-binding-hive-follower-v2/pom.xml > fa7e9282592bf5fe15a35921b7173c81b45a1f1f > sentry-binding/sentry-binding-hive-follower/pom.xml > e69519cc63d5c89db6098fa169089b34dbaadae8 > > sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONAlterPartitionMessage.java > 1e636c94afc99678658bdafe74fdd7aff7e12d92 > > sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONDropPartitionMessage.java > d3ebf603f2d81b3fed7ec0d33031446185b80cb5 > > sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONMessageDeserializer.java > cc0bbec830436241ef69fc3dc9a727bf105ae6e7 > > sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONMessageFactory.java > efdf8b8a5b4eeca08e2a4ad39c95ab86e7855a76 > sentry-binding/sentry-binding-hive-v2/pom.xml > 5f5cbf3c15df324d741e5eca7b951e4aab5bd0d3 > sentry-binding/sentry-binding-hive/pom.xml > 92147e18b0560bd266bfc82d65b9bc70af3df627 > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/hadoop/hive/ql/exec/SentryGrantRevokeTask.java > 97dbd2a503bf9d61c18f6f4fcaa494cbd429f07d > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/HiveAuthzBindingSessionHook.java > a3aa0b0351deb7b41dec3e5bc7b0e62727e3cb21 > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/HiveAuthzPrivilegesMap.java > 9f3d42d3787fbceb62e9fd5e22395c17010867f4 > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/SentryHiveAuthorizerFactory.java > PRE-CREATION > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/SentryHiveAuthorizerImpl.java > PRE-CREATION > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/AuthorizingObjectStore.java > d20da81efcbc796456c114654df0261964c2cfc1 > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/AuthorizingObjectStoreBase.java > 9e066e12f1b3539e8787878723eec153dfd68d32 > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/MetastoreAuthzBindingBase.java > 6df4885cea3868b451ed14a5e354dd45794cbb3f > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/SentryMetastorePostEventListener.java > 3ec2eedab99e7ec66e2dc69b9c8af8eb482cac08 > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/SentryMetastorePostEventListenerBase.java > 5b9274e337d1f8a05e2b01384cb3a2e95ad07293 > > sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/SentryMetastorePostEventListenerNotificationLog.java > 58470d6ade3467ef7b03428272d61dc0dd1accea > > sentry-binding/sentry-binding-hive/src/test/java/org/apache/sentry/binding/hive/TestSentryHiveAuthorizationTaskFactory.java > aed218ec0308405c32e6a070451ef38c4c67bb27 > > sentry-binding/sentry-binding-hive/src/test/java/org/apache/sentry/binding/hive/TestURI.java > aa3de64a404b2a8e1dbe245a9cd2298c41745fc3 > sentry-binding/sentry-binding-hive/src/test/resources/log4j2.properties > PRE-CREATION > sentry-provider/sentry-provider-db/pom.xml > 6b7d3c07a0cdf43ff925c22d58c290a3a9affe8b > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/SentryPolicyStoreProcessor.java > cd854006f2b0e73a8a1bcb3c87a4df01bd30cc14 > sentry-tests/sentry-tests-hive/pom.xml > 51801dc8bdcd449f9acaa5658855a4227c47c56e > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestDbDDLAuditLog.java > e105f003b79b0ad9bec58b24d3e7fd87576e7ad7 > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationAdvanced.java > 2073d85f27349c933bd936f87dc754c6fb30a729 > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationBase.java > 718b02cbd9df9a551bd4de1768f93a914cad7b32 > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationEnd2End.java > d4bc97d2f59efee00034036ebc716ab69cf2116b > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/AbstractTestWithStaticConfiguration.java > 0a39f5954de5867b4e64e9be3529219654d31378 > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/TestMetadataObjectRetrieval.java > fb0ef19a52375126d21a4c787dd31a1deacfc232 > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/TestOperationsPart1.java > 3a4da5059463459af43fd04ed3b34b6468848e7e > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/TestPrivilegesAtFunctionScope.java > 249d3bcfe8b1f9f2a1256280cf9b80527cef1e4c > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/hiveserver/HiveServerFactory.java > f3f58f64f66ea693ab99d2dd5a9b0eb298ae7d0f > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/metastore/TestDBNotificationListenerInBuiltDeserializer.java > e9b3a43debb6d47818c54f6a1f646f13668e35f2 > > sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/metastore/TestSentryListenerSentryDeserializer.java > f43b316928a6354674e824d04b8e7c4a9e5bec00 > sentry-tests/sentry-tests-hive/src/test/resources/log4j2.properties > PRE-CREATION > sentry-tests/sentry-tests-kafka/pom.xml > 56a3ef10a9071929776cb7211bdb8ead921deace > sentry-tests/sentry-tests-solr/pom.xml > c70476808688c80e1723d5e65e3b8cf6d1b64250 > sentry-tests/sentry-tests-sqoop/pom.xml > 9e7cee95be06c68b0eaa456a80d4120576ec977b > > > Diff: https://reviews.apache.org/r/62411/diff/2/ > > > Testing > ------- > > > Thanks, > > Sergio Pena > >
