----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71006/#review216439 -----------------------------------------------------------
- Sridhar K On July 8, 2019, 1:26 p.m., Saqeeb Shaikh wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71006/ > ----------------------------------------------------------- > > (Updated July 8, 2019, 1:26 p.m.) > > > Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon > Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3313 > https://issues.apache.org/jira/browse/ATLAS-3313 > > > Repository: atlas > > > Description > ------- > > Currently Atlas server processes notifications from Kafka topic named > ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). > The name of the Kafka topic to which an Atlas Hook publishes notifications > can be configured via configuration atlas.notification.hook.topic.name. > However, since Atlas server processes notifications from a single topic, all > hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic. > > To enable different hooks to publish to different Kafka topics, Atlas server > should support processing notifications from multiple Kafka topics. > > > Add following properties in atlas: > atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK > (comma separated without space) > atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES > > > Diffs > ----- > > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 > intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java > 5c840c3 > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java > b04aba9 > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java > 449eb6f > notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java > 847caa3 > > notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java > 263903b > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java > c963830 > repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java > d5a4412 > repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java > f658caa > repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java > b56019e > > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java > 1f8e810 > > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java > fb3ff26 > > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java > 3e35511 > > > Diff: https://reviews.apache.org/r/71006/diff/2/ > > > Testing > ------- > > Tested by publishing create hive table JSON using kafka-console-producer on > custom topics configured above. I was able to search the entities in atlas. > Also tested basic use case by creating database and table from hive console. > > mvn clean package > Some test cases are failing on my local (will work on fixing the same). > > > Thanks, > > Saqeeb Shaikh > >
