Review Request 49777: SENTRY-1321: Implement HMSFollower in Sentry service which reads the NotificationLog entries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49777/ --- Review request for sentry and Hao Hao. Repository: sentry Description --- Adding HMSFollower and changes in Sentryservice to use it. Some required and non required pom changes Test changes - start sentryservice before hive Change-Id: I1b470de33d760fd922afed97f2941cd3750c4801 Diffs - sentry-provider/sentry-provider-db/pom.xml b8143ffa3adca9e47e7cb092131d65064d57c86b sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java PRE-CREATION sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryKerberosContext.java f54f161b381088285486a5ca74972f93ee620547 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryService.java 809af064a8f54860ae796f1d9f29fd8f52568663 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/ServiceConstants.java 0ab8192a86178f4febcec7384ebc3a5be0cc69fb sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestDbPrivilegeCleanupOnDrop.java 767bcbe02e0d511f52bc869e7b6a1ee1e6584a5a sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/AbstractTestWithStaticConfiguration.java 2c4948e9d100f2cf0cb5b7772489194c62b8a857 Diff: https://reviews.apache.org/r/49777/diff/ Testing --- Testing not complete yet. Submmitting a preview of changes for review as this work also is important for SENTRY-1371 Thanks, Sravya Tirukkovalur
Re: Review Request 49777: SENTRY-1321: Implement HMSFollower in Sentry service which reads the NotificationLog entries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49777/ --- (Updated July 7, 2016, 11:55 p.m.) Review request for sentry and Hao Hao. Repository: sentry Description (updated) --- Adding HMSFollower and changes in Sentryservice to use it. Test changes - start sentryservice before hive Change-Id: I1b470de33d760fd922afed97f2941cd3750c4801 Diffs - sentry-provider/sentry-provider-db/pom.xml b8143ffa3adca9e47e7cb092131d65064d57c86b sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java PRE-CREATION sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryKerberosContext.java f54f161b381088285486a5ca74972f93ee620547 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryService.java 809af064a8f54860ae796f1d9f29fd8f52568663 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/ServiceConstants.java 0ab8192a86178f4febcec7384ebc3a5be0cc69fb sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestDbPrivilegeCleanupOnDrop.java 767bcbe02e0d511f52bc869e7b6a1ee1e6584a5a sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/AbstractTestWithStaticConfiguration.java 2c4948e9d100f2cf0cb5b7772489194c62b8a857 Diff: https://reviews.apache.org/r/49777/diff/ Testing --- Testing not complete yet. Submmitting a preview of changes for review as this work also is important for SENTRY-1371 Thanks, Sravya Tirukkovalur
Re: Review Request 49397: SENTRY-1329: Adapt SentryMetaStorePostEventListener to write HMS notification logs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49397/ --- (Updated July 7, 2016, 9:57 p.m.) Review request for sentry, Anne Yu, Colin McCabe, Hao Hao, and Nachiket Vaidya. Bugs: Sentry-1329 https://issues.apache.org/jira/browse/Sentry-1329 Repository: sentry Description --- This patch adapts SentryMetaStorePostEventListener to write HMS notification logs. Also, 1. Implements the SentryJSONMessageFactory to add custom information in the notification log entry message, which includes 1.1. Implementing Message class for each message type 1.2. Implementing a deserializer 2. Implements JSONAlterPartitionMessage and JSONAlterTableMessage to work around the issue in Hive 1.1.0. 3. Testing: 3.1. Sentry functionality: TestSentryListenerSentryDeserializer to verify functionality using Sentry's SentryMetastorePostEventListener and Sentry Notification log deserializer. 3.2. TestDbNotificationListenerSentryDeserializer uses Hive's DbNotificationListener and Sentry's JSON deserializeri. This would make sure Sentry is able to read the Notification logs written by Hive's DBNotificationListener 3.3. TestSentryListenerInBuiltDeserializer uses Sentry's SentryMetastorePostEventListener and Hive's inbuilt Notification log deserializer: This would make sure Sentry is not breaking other users of NotificationLog who might be using Hive's in built serializer Change-Id: I680beb6db4e534bb0a9e6ee042ea0d4f33f0943f update Change-Id: I6ce8846a801b3effc6a851ab3ce6f866b4a0c23e Diffs (updated) - sentry-binding/sentry-binding-hive/pom.xml 07aaae3bac34582fe8222ff166a52c1e208b223a sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/SentryMetastorePostEventListener.java d12ac151ddbf35d612fee3f869d602d6cdf54aa8 sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/JSONAlterPartitionMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/JSONAlterTableMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONAddPartitionMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONAlterPartitionMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONAlterTableMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONCreateDatabaseMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONCreateTableMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONDropDatabaseMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONDropPartitionMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONDropTableMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONMessageDeserializer.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONMessageFactory.java PRE-CREATION sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestDbPrivilegeCleanupOnDrop.java 767bcbe02e0d511f52bc869e7b6a1ee1e6584a5a sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/AbstractTestWithStaticConfiguration.java 2c4948e9d100f2cf0cb5b7772489194c62b8a857 sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/metastore/AbstractMetastoreTestWithStaticConfiguration.java b72e317621610eababf5d01886bc00dca9f5e7ec sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/metastore/TestDbNotificationListenerSentryDeserializer.java PRE-CREATION sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/metastore/TestHMSNotificationLogUsingDBNotificationListener.java 0b328d402d005d6f858e87fbe64eedd5f9f1a092 sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/metastore/TestSentryListenerInBuiltDeserializer.java PRE-CREATION sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/metastore/TestSentryListenerSentryDeserializer.java PRE-CREATION Diff: https://reviews.apache.org/r/49397/diff/ Testing --- Added new tests Thanks, Srav
Re: Review Request 49738: SENTRY-1317: Implement fencing required for active/passive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49738/ --- (Updated July 7, 2016, 9:33 p.m.) Review request for sentry, Hao Hao, Rahul Sharma, and Sravya Tirukkovalur. Bugs: SENTRY-1317 https://issues.apache.org/jira/browse/SENTRY-1317 Repository: sentry Description --- SENTRY-1317: Implement fencing required for active/passive Diffs (updated) - sentry-core/sentry-core-common/src/main/java/org/apache/sentry/core/common/exception/SentryStandbyException.java PRE-CREATION sentry-core/sentry-core-common/src/main/java/org/apache/sentry/core/common/utils/SentryConstants.java 3da4906 sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/PluginCacheSyncUtil.java 4ce16c7 sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/SentryHdfsMetricsUtil.java 5bf2f6e sentry-hdfs/sentry-hdfs-service/src/test/java/org/apache/sentry/hdfs/TestHAUpdateForwarder.java 5246e05 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/Fencer.java PRE-CREATION sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/HAContext.java cacc29f sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/SentryStore.java 7dad496 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/ServiceRegister.java 79dfe48 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/SqlAccessor.java PRE-CREATION sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/SentryPolicyStoreProcessor.java 3de1f65 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/Activator.java PRE-CREATION sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/Activators.java PRE-CREATION sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/LeaderStatus.java e846766 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/LeaderStatusAdaptor.java 80a6571 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryService.java 809af06 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/ServiceConstants.java 0ab8192 sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/generic/service/persistent/SentryStoreIntegrationBase.java f14b586 sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/generic/service/persistent/TestPrivilegeOperatePersistence.java 799d5ef sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestSentryStore.java 3ef1cf7 sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestSentryStoreImportExport.java 3ff97df sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestSentryVersion.java a8e8a03 sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/service/thrift/SentryServiceIntegrationBase.java cb2d9c9 sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/service/thrift/TestLeaderStatus.java 434ac41 Diff: https://reviews.apache.org/r/49738/diff/ Testing --- Thanks, Colin McCabe
Re: Review Request 49397: SENTRY-1329: Adapt SentryMetaStorePostEventListener to write HMS notification logs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49397/ --- (Updated July 7, 2016, 8:53 p.m.) Review request for sentry, Anne Yu, Colin McCabe, Hao Hao, and Nachiket Vaidya. Changes --- Made some test fixes Bugs: Sentry-1329 https://issues.apache.org/jira/browse/Sentry-1329 Repository: sentry Description --- This patch adapts SentryMetaStorePostEventListener to write HMS notification logs. Also, 1. Implements the SentryJSONMessageFactory to add custom information in the notification log entry message, which includes 1.1. Implementing Message class for each message type 1.2. Implementing a deserializer 2. Implements JSONAlterPartitionMessage and JSONAlterTableMessage to work around the issue in Hive 1.1.0. 3. Testing: 3.1. Sentry functionality: TestSentryListenerSentryDeserializer to verify functionality using Sentry's SentryMetastorePostEventListener and Sentry Notification log deserializer. 3.2. TestDbNotificationListenerSentryDeserializer uses Hive's DbNotificationListener and Sentry's JSON deserializeri. This would make sure Sentry is able to read the Notification logs written by Hive's DBNotificationListener 3.3. TestSentryListenerInBuiltDeserializer uses Sentry's SentryMetastorePostEventListener and Hive's inbuilt Notification log deserializer: This would make sure Sentry is not breaking other users of NotificationLog who might be using Hive's in built serializer Change-Id: I680beb6db4e534bb0a9e6ee042ea0d4f33f0943f update Change-Id: I6ce8846a801b3effc6a851ab3ce6f866b4a0c23e Diffs (updated) - sentry-binding/sentry-binding-hive/pom.xml 07aaae3bac34582fe8222ff166a52c1e208b223a sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/SentryMetastorePostEventListener.java d12ac151ddbf35d612fee3f869d602d6cdf54aa8 sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/JSONAlterPartitionMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/JSONAlterTableMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONAddPartitionMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONAlterPartitionMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONAlterTableMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONCreateDatabaseMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONCreateTableMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONDropDatabaseMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONDropPartitionMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONDropTableMessage.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONMessageDeserializer.java PRE-CREATION sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONMessageFactory.java PRE-CREATION sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestDbPrivilegeCleanupOnDrop.java 767bcbe02e0d511f52bc869e7b6a1ee1e6584a5a sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/AbstractTestWithStaticConfiguration.java 2c4948e9d100f2cf0cb5b7772489194c62b8a857 sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/metastore/AbstractMetastoreTestWithStaticConfiguration.java b72e317621610eababf5d01886bc00dca9f5e7ec sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/metastore/TestDbNotificationListenerSentryDeserializer.java PRE-CREATION sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/metastore/TestHMSNotificationLogUsingDBNotificationListener.java 0b328d402d005d6f858e87fbe64eedd5f9f1a092 sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/metastore/TestSentryListenerInBuiltDeserializer.java PRE-CREATION sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/metastore/TestSentryListenerSentryDeserializer.java PRE-CREATION Diff: https://reviews.apache.org/r/49397/diff/ Testing
Re: Review Request 49738: SENTRY-1317: Implement fencing required for active/passive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49738/ --- (Updated July 7, 2016, 6:31 p.m.) Review request for sentry, Hao Hao, Rahul Sharma, and Sravya Tirukkovalur. Changes --- Fix PMD warning Bugs: SENTRY-1317 https://issues.apache.org/jira/browse/SENTRY-1317 Repository: sentry Description --- SENTRY-1317: Implement fencing required for active/passive Diffs (updated) - sentry-core/sentry-core-common/src/main/java/org/apache/sentry/core/common/exception/SentryStandbyException.java PRE-CREATION sentry-core/sentry-core-common/src/main/java/org/apache/sentry/core/common/utils/SentryConstants.java 3da4906 sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/PluginCacheSyncUtil.java 4ce16c7 sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/SentryHDFSServiceProcessorFactory.java db55b5a sentry-hdfs/sentry-hdfs-service/src/main/java/org/apache/sentry/hdfs/SentryHdfsMetricsUtil.java 5bf2f6e sentry-hdfs/sentry-hdfs-service/src/test/java/org/apache/sentry/hdfs/TestHAUpdateForwarder.java 5246e05 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/generic/service/persistent/DelegateSentryStore.java e960dcd sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/generic/service/thrift/SentryGenericPolicyProcessorFactory.java 1cce1fc sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/Fencer.java PRE-CREATION sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/HAContext.java cacc29f sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/SentryStore.java 7dad496 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/ServiceRegister.java 79dfe48 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/SqlAccessor.java PRE-CREATION sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/SentryPolicyStoreProcessor.java 3de1f65 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/SentryPolicyStoreProcessorFactory.java 691c1fb sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/Activator.java PRE-CREATION sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/LeaderStatus.java e846766 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/LeaderStatusAdaptor.java 80a6571 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/ProcessorFactory.java a3bb6ab sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/SentryService.java 809af06 sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/ServiceConstants.java 0ab8192 sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/generic/service/persistent/SentryStoreIntegrationBase.java f14b586 sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/generic/service/persistent/TestPrivilegeOperatePersistence.java 799d5ef sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestSentryStore.java 3ef1cf7 sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestSentryStoreImportExport.java 3ff97df sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestSentryVersion.java a8e8a03 sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/service/thrift/SentryServiceIntegrationBase.java cb2d9c9 sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/service/thrift/TestLeaderStatus.java 434ac41 Diff: https://reviews.apache.org/r/49738/diff/ Testing --- Thanks, Colin McCabe
Review Request 49744: SENTRY-1366 Test Alter table rename when there are no partitions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49744/ --- Review request for sentry, Dapeng Sun and Sravya Tirukkovalur. Bugs: SENTRY-1366 https://issues.apache.org/jira/browse/SENTRY-1366 Repository: sentry Description --- Test Alter table rename when there are no partitions Diffs - sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/TestOperationsPart1.java a13aef5 Diff: https://reviews.apache.org/r/49744/diff/ Testing --- test done Thanks, Ke Jia
