Re: Review Request 48280: ATLAS-515: Ability to initialize Kafka topics with more than 1 replica
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48280/#review137355 --- Ship it! Ship It! - Suma Shivaprasad On June 13, 2016, 4:35 p.m., Hemanth Yamijala wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48280/ > --- > > (Updated June 13, 2016, 4:35 p.m.) > > > Review request for atlas. > > > Bugs: ATLAS-515 > https://issues.apache.org/jira/browse/ATLAS-515 > > > Repository: atlas > > > Description > --- > > Attached patch provides a method to automatically create Kafka topics > ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas. > > > Diffs > - > > common/src/main/java/org/apache/atlas/utils/AuthenticationUtil.java 3dbab17 > distro/src/bin/atlas_client_cmdline.py f109ad3 > distro/src/bin/atlas_config.py fab4046 > distro/src/bin/atlas_kafka_setup.py PRE-CREATION > distro/src/bin/atlas_kafka_setup_hook.py PRE-CREATION > distro/src/conf/atlas-application.properties 2b90086 > distro/src/main/assemblies/standalone-package.xml 1c7b2c5 > docs/src/site/twiki/Configuration.twiki 7150483 > docs/src/site/twiki/InstallationSteps.twiki 518c380 > notification/pom.xml b3738db > notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java > PRE-CREATION > notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java > PRE-CREATION > pom.xml e13345e > > Diff: https://reviews.apache.org/r/48280/diff/ > > > Testing > --- > > * Started multiple Kafka brokers on single machine, configured number of > replicas to be 2, started Atlas server and verified they got created > correctly. > * Started multiple Kafka brokers on single machine, configured number of > replicas to be 2, started Hive hook and verified it created ATLAS_HOOK > correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK > creation, but did ATLAS_ENTITIES creation. > * Started one Kafka broker, onfigured number of replicas to be 2, started > Atlas server and verified it received an exception as expected. But topic > could still get auto-created and continued. > * Started 2 instances of Atlas server concurrently and verified topic > creation is working fine. > * Existing UTs/ITs/new UTs pass. > > > Thanks, > > Hemanth Yamijala > >
Re: Review Request 48280: ATLAS-515: Ability to initialize Kafka topics with more than 1 replica
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48280/ --- (Updated June 13, 2016, 4:35 p.m.) Review request for atlas. Changes --- Updated the patch to handle Kerberized Kafka clusters. Bugs: ATLAS-515 https://issues.apache.org/jira/browse/ATLAS-515 Repository: atlas Description --- Attached patch provides a method to automatically create Kafka topics ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas. Diffs (updated) - common/src/main/java/org/apache/atlas/utils/AuthenticationUtil.java 3dbab17 distro/src/bin/atlas_client_cmdline.py f109ad3 distro/src/bin/atlas_config.py fab4046 distro/src/bin/atlas_kafka_setup.py PRE-CREATION distro/src/bin/atlas_kafka_setup_hook.py PRE-CREATION distro/src/conf/atlas-application.properties 2b90086 distro/src/main/assemblies/standalone-package.xml 1c7b2c5 docs/src/site/twiki/Configuration.twiki 7150483 docs/src/site/twiki/InstallationSteps.twiki 518c380 notification/pom.xml b3738db notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java PRE-CREATION notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java PRE-CREATION pom.xml e13345e Diff: https://reviews.apache.org/r/48280/diff/ Testing --- * Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Atlas server and verified they got created correctly. * Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Hive hook and verified it created ATLAS_HOOK correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK creation, but did ATLAS_ENTITIES creation. * Started one Kafka broker, onfigured number of replicas to be 2, started Atlas server and verified it received an exception as expected. But topic could still get auto-created and continued. * Started 2 instances of Atlas server concurrently and verified topic creation is working fine. * Existing UTs/ITs/new UTs pass. Thanks, Hemanth Yamijala
Re: Review Request 48280: ATLAS-515: Ability to initialize Kafka topics with more than 1 replica
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48280/ --- (Updated June 10, 2016, 8:44 a.m.) Review request for atlas. Changes --- The new patch adds documentation to Installation and Configuration pages for using the added scripts. Bugs: ATLAS-515 https://issues.apache.org/jira/browse/ATLAS-515 Repository: atlas Description --- Attached patch provides a method to automatically create Kafka topics ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas. Diffs (updated) - distro/src/bin/atlas_client_cmdline.py f109ad3 distro/src/bin/atlas_config.py fab4046 distro/src/bin/atlas_kafka_setup.py PRE-CREATION distro/src/bin/atlas_kafka_setup_hook.py PRE-CREATION distro/src/conf/atlas-application.properties 873c1e0 distro/src/main/assemblies/standalone-package.xml 1c7b2c5 docs/src/site/twiki/Configuration.twiki 7150483 docs/src/site/twiki/InstallationSteps.twiki 9699a94 notification/pom.xml b3738db notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java PRE-CREATION notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java PRE-CREATION pom.xml eac95c8 Diff: https://reviews.apache.org/r/48280/diff/ Testing --- * Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Atlas server and verified they got created correctly. * Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Hive hook and verified it created ATLAS_HOOK correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK creation, but did ATLAS_ENTITIES creation. * Started one Kafka broker, onfigured number of replicas to be 2, started Atlas server and verified it received an exception as expected. But topic could still get auto-created and continued. * Started 2 instances of Atlas server concurrently and verified topic creation is working fine. * Existing UTs/ITs/new UTs pass. Thanks, Hemanth Yamijala
Re: Review Request 48280: ATLAS-515: Ability to initialize Kafka topics with more than 1 replica
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48280/#review136932 --- Ship it! Pls add documentation on how to use the scripts. The changes looks good to me. - Suma Shivaprasad On June 9, 2016, 10:30 a.m., Hemanth Yamijala wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48280/ > --- > > (Updated June 9, 2016, 10:30 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-515 > https://issues.apache.org/jira/browse/ATLAS-515 > > > Repository: atlas > > > Description > --- > > Attached patch provides a method to automatically create Kafka topics > ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas. > > > Diffs > - > > distro/src/bin/atlas_client_cmdline.py f109ad3 > distro/src/bin/atlas_config.py fab4046 > distro/src/bin/atlas_kafka_setup.py PRE-CREATION > distro/src/bin/atlas_kafka_setup_hook.py PRE-CREATION > distro/src/conf/atlas-application.properties 873c1e0 > distro/src/main/assemblies/standalone-package.xml 1c7b2c5 > notification/pom.xml b3738db > notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java > PRE-CREATION > notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java > PRE-CREATION > pom.xml eac95c8 > > Diff: https://reviews.apache.org/r/48280/diff/ > > > Testing > --- > > * Started multiple Kafka brokers on single machine, configured number of > replicas to be 2, started Atlas server and verified they got created > correctly. > * Started multiple Kafka brokers on single machine, configured number of > replicas to be 2, started Hive hook and verified it created ATLAS_HOOK > correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK > creation, but did ATLAS_ENTITIES creation. > * Started one Kafka broker, onfigured number of replicas to be 2, started > Atlas server and verified it received an exception as expected. But topic > could still get auto-created and continued. > * Started 2 instances of Atlas server concurrently and verified topic > creation is working fine. > * Existing UTs/ITs/new UTs pass. > > > Thanks, > > Hemanth Yamijala > >
Re: Review Request 48280: ATLAS-515: Ability to initialize Kafka topics with more than 1 replica
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48280/ --- (Updated June 9, 2016, 10:30 a.m.) Review request for atlas. Changes --- As described in this [comment](https://issues.apache.org/jira/browse/ATLAS-515?focusedCommentId=15320301=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15320301) on ATLAS-515, I am changing the approach to initializing Kafka topics from being automated to being script driven, where the script can be automated by installation tools like Ambari. Changes in this patch: * Retains KafkaTopicCreator class with the logic for setting up topics * Two python scripts - atlas_kafka_setup.py and atlas_kafka_setup_hook.py. * Changes to pom.xml to include the required dependencies Bugs: ATLAS-515 https://issues.apache.org/jira/browse/ATLAS-515 Repository: atlas Description --- Attached patch provides a method to automatically create Kafka topics ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas. Diffs (updated) - distro/src/bin/atlas_client_cmdline.py f109ad3 distro/src/bin/atlas_config.py fab4046 distro/src/bin/atlas_kafka_setup.py PRE-CREATION distro/src/bin/atlas_kafka_setup_hook.py PRE-CREATION distro/src/conf/atlas-application.properties 873c1e0 distro/src/main/assemblies/standalone-package.xml 1c7b2c5 notification/pom.xml b3738db notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java PRE-CREATION notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java PRE-CREATION pom.xml eac95c8 Diff: https://reviews.apache.org/r/48280/diff/ Testing --- * Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Atlas server and verified they got created correctly. * Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Hive hook and verified it created ATLAS_HOOK correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK creation, but did ATLAS_ENTITIES creation. * Started one Kafka broker, onfigured number of replicas to be 2, started Atlas server and verified it received an exception as expected. But topic could still get auto-created and continued. * Started 2 instances of Atlas server concurrently and verified topic creation is working fine. * Existing UTs/ITs/new UTs pass. Thanks, Hemanth Yamijala
Review Request 48280: ATLAS-515: Ability to initialize Kafka topics with more than 1 replica
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48280/ --- Review request for atlas. Bugs: ATLAS-515 https://issues.apache.org/jira/browse/ATLAS-515 Repository: atlas Description --- Attached patch provides a method to automatically create Kafka topics ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas. Diffs - addons/falcon-bridge/pom.xml 14c6090 addons/hive-bridge/pom.xml 47e72e8 addons/storm-bridge/pom.xml b1a7a9b distro/src/conf/atlas-application.properties 1cdd424 notification/pom.xml b3738db notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 71029b0 notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java PRE-CREATION notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 1ee62d2 notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java cb44fc6 notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java PRE-CREATION pom.xml eac95c8 Diff: https://reviews.apache.org/r/48280/diff/ Testing --- * Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Atlas server and verified they got created correctly. * Started multiple Kafka brokers on single machine, configured number of replicas to be 2, started Hive hook and verified it created ATLAS_HOOK correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK creation, but did ATLAS_ENTITIES creation. * Started one Kafka broker, onfigured number of replicas to be 2, started Atlas server and verified it received an exception as expected. But topic could still get auto-created and continued. * Started 2 instances of Atlas server concurrently and verified topic creation is working fine. * Existing UTs/ITs/new UTs pass. Thanks, Hemanth Yamijala
