Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

2019-07-11 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216530
---


Ship it!




Ship It!

- Madhan Neethiraj


On July 11, 2019, 8:41 a.m., Saqeeb Shaikh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> ---
> 
> (Updated July 11, 2019, 8:41 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
> Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas server processes notifications from Kafka topic named 
> ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). 
> The name of the Kafka topic to which an Atlas Hook publishes notifications 
> can be configured via configuration atlas.notification.hook.topic.name. 
> However, since Atlas server processes notifications from a single topic, all 
> hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
> 
> To enable different hooks to publish to different Kafka topics, Atlas server 
> should support processing notifications from multiple Kafka topics.
> 
> 
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
>  (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
>   intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> 5c840c3 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> b04aba9 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 449eb6f 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 847caa3 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  263903b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  c963830 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
> d5a4412 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java 
> f658caa 
>   repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
> b56019e 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  1f8e810 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
>  fb3ff26 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
>  3e35511 
> 
> 
> Diff: https://reviews.apache.org/r/71006/diff/4/
> 
> 
> Testing
> ---
> 
> Tested by publishing create hive table JSON using kafka-console-producer on 
> custom topics configured above. I was able to search the entities in atlas. 
> Also tested basic use case by creating database and table from hive console. 
> 
> mvn clean package 
> Some test cases are failing on my local (will work on fixing the same).
> 
> 
> Thanks,
> 
> Saqeeb Shaikh
> 
>

Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

2019-07-11 Thread Nixon Rodrigues 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216520
---


Ship it!




Ship It!

- Nixon Rodrigues


On July 11, 2019, 8:41 a.m., Saqeeb Shaikh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> ---
> 
> (Updated July 11, 2019, 8:41 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
> Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas server processes notifications from Kafka topic named 
> ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). 
> The name of the Kafka topic to which an Atlas Hook publishes notifications 
> can be configured via configuration atlas.notification.hook.topic.name. 
> However, since Atlas server processes notifications from a single topic, all 
> hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
> 
> To enable different hooks to publish to different Kafka topics, Atlas server 
> should support processing notifications from multiple Kafka topics.
> 
> 
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
>  (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
>   intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> 5c840c3 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> b04aba9 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 449eb6f 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 847caa3 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  263903b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  c963830 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
> d5a4412 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java 
> f658caa 
>   repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
> b56019e 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  1f8e810 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
>  fb3ff26 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
>  3e35511 
> 
> 
> Diff: https://reviews.apache.org/r/71006/diff/4/
> 
> 
> Testing
> ---
> 
> Tested by publishing create hive table JSON using kafka-console-producer on 
> custom topics configured above. I was able to search the entities in atlas. 
> Also tested basic use case by creating database and table from hive console. 
> 
> mvn clean package 
> Some test cases are failing on my local (will work on fixing the same).
> 
> 
> Thanks,
> 
> Saqeeb Shaikh
> 
>

Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

2019-07-11 Thread Saqeeb Shaikh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/
---

(Updated July 11, 2019, 8:41 a.m.)


Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
Rodrigues, and Sarath Subramanian.


Changes
---

Updated implementation of KafkaNotification.


Bugs: ATLAS-3313
https://issues.apache.org/jira/browse/ATLAS-3313


Repository: atlas


Description
---

Currently Atlas server processes notifications from Kafka topic named 
ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The 
name of the Kafka topic to which an Atlas Hook publishes notifications can be 
configured via configuration atlas.notification.hook.topic.name. However, since 
Atlas server processes notifications from a single topic, all hooks 
(Hive/HBase/Spark/Storm/..) are required to publish to the same topic.

To enable different hooks to publish to different Kafka topics, Atlas server 
should support processing notifications from multiple Kafka topics.


Add following properties in atlas:
atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
 (comma separated without space)
atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
  intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
5c840c3 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
b04aba9 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
449eb6f 
  notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
847caa3 
  
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
 263903b 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
 c963830 
  repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
d5a4412 
  repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa 
  repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
b56019e 
  
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 1f8e810 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
 fb3ff26 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
 3e35511 


Diff: https://reviews.apache.org/r/71006/diff/4/

Changes: https://reviews.apache.org/r/71006/diff/3-4/


Testing
---

Tested by publishing create hive table JSON using kafka-console-producer on 
custom topics configured above. I was able to search the entities in atlas. 
Also tested basic use case by creating database and table from hive console. 

mvn clean package 
Some test cases are failing on my local (will work on fixing the same).


Thanks,

Saqeeb Shaikh

Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

2019-07-10 Thread Saqeeb Shaikh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/
---

(Updated July 10, 2019, 3:22 p.m.)


Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
Rodrigues, and Sarath Subramanian.


Changes
---

Addressed new review comments.


Bugs: ATLAS-3313
https://issues.apache.org/jira/browse/ATLAS-3313


Repository: atlas


Description
---

Currently Atlas server processes notifications from Kafka topic named 
ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The 
name of the Kafka topic to which an Atlas Hook publishes notifications can be 
configured via configuration atlas.notification.hook.topic.name. However, since 
Atlas server processes notifications from a single topic, all hooks 
(Hive/HBase/Spark/Storm/..) are required to publish to the same topic.

To enable different hooks to publish to different Kafka topics, Atlas server 
should support processing notifications from multiple Kafka topics.


Add following properties in atlas:
atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
 (comma separated without space)
atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
  intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
5c840c3 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
b04aba9 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
449eb6f 
  notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
847caa3 
  
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
 263903b 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
 c963830 
  repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
d5a4412 
  repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa 
  repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
b56019e 
  
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 1f8e810 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
 fb3ff26 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
 3e35511 


Diff: https://reviews.apache.org/r/71006/diff/3/

Changes: https://reviews.apache.org/r/71006/diff/2-3/


Testing
---

Tested by publishing create hive table JSON using kafka-console-producer on 
custom topics configured above. I was able to search the entities in atlas. 
Also tested basic use case by creating database and table from hive console. 

mvn clean package 
Some test cases are failing on my local (will work on fixing the same).


Thanks,

Saqeeb Shaikh

Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

2019-07-09 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216452
---




intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 91 (patched)


I agree. Let's rename to getStringArray() - to be in sync with method names 
in  org.apache.commons.configuration.Configuration.



intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 100 (patched)


I suggest we leave trimming and handling of empty value to the callers. It 
is possible that an empty string is a valid value for some configurations.



intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java
Line 58 (original), 58 (patched)


metrics values are not persisted in Atlas. And given we need to track 
offsets for multiple topics, the metrics name need to change.


- Madhan Neethiraj


On July 8, 2019, 1:26 p.m., Saqeeb Shaikh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> ---
> 
> (Updated July 8, 2019, 1:26 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
> Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas server processes notifications from Kafka topic named 
> ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). 
> The name of the Kafka topic to which an Atlas Hook publishes notifications 
> can be configured via configuration atlas.notification.hook.topic.name. 
> However, since Atlas server processes notifications from a single topic, all 
> hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
> 
> To enable different hooks to publish to different Kafka topics, Atlas server 
> should support processing notifications from multiple Kafka topics.
> 
> 
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
>  (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
>   intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> 5c840c3 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> b04aba9 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 449eb6f 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 847caa3 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  263903b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  c963830 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
> d5a4412 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java 
> f658caa 
>   repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
> b56019e 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  1f8e810 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
>  fb3ff26 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
>  3e35511 
> 
> 
> Diff: https://reviews.apache.org/r/71006/diff/2/
> 
> 
> Testing
> ---
> 
> Tested by publishing create hive table JSON using kafka-console-producer on 
> custom topics configured above. I was able to search the entities in atlas. 
> Also tested basic use case by creating database and table from hive console. 
> 
> mvn clean package 
> Some test cases are failing on my local (will work on fixing the same).
> 
> 
> Thanks,
> 
> Saqeeb Shaikh
> 
>

Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

2019-07-08 Thread Sridhar K 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216439
---



- Sridhar K


On July 8, 2019, 1:26 p.m., Saqeeb Shaikh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> ---
> 
> (Updated July 8, 2019, 1:26 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
> Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas server processes notifications from Kafka topic named 
> ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). 
> The name of the Kafka topic to which an Atlas Hook publishes notifications 
> can be configured via configuration atlas.notification.hook.topic.name. 
> However, since Atlas server processes notifications from a single topic, all 
> hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
> 
> To enable different hooks to publish to different Kafka topics, Atlas server 
> should support processing notifications from multiple Kafka topics.
> 
> 
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
>  (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
>   intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> 5c840c3 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> b04aba9 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 449eb6f 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 847caa3 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  263903b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  c963830 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
> d5a4412 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java 
> f658caa 
>   repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
> b56019e 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  1f8e810 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
>  fb3ff26 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
>  3e35511 
> 
> 
> Diff: https://reviews.apache.org/r/71006/diff/2/
> 
> 
> Testing
> ---
> 
> Tested by publishing create hive table JSON using kafka-console-producer on 
> custom topics configured above. I was able to search the entities in atlas. 
> Also tested basic use case by creating database and table from hive console. 
> 
> mvn clean package 
> Some test cases are failing on my local (will work on fixing the same).
> 
> 
> Thanks,
> 
> Saqeeb Shaikh
> 
>

Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

2019-07-08 Thread Sridhar K 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216438
---




intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 91 (patched)


Can we please rename this method?...too generic name.



intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 100 (patched)


is it possible to go over all strings and drop strings that have zero 
length after trimming...It will cover scenarios like the following..

"value1",,"value2",
"value1", ,"value2",



intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 103 (patched)


Can we please rename this method?...too generic name.

is it possible to go over all strings and drop strings that have zero 
length after trimming...It will cover scenarios like the following..

"value1",,"value2",
"value1", ,"value2",



intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java
Line 58 (original), 58 (patched)


You are changing the metrics names. Will there be any impact on existing 
metric data...Question is valid, if these values are persisted some where.



notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Lines 176 (patched)


Can you also put a message saying that the number of consumers will be set 
to number of topics.

I will also put a message in startConsumers() method that is the initial 
call in hierarchy.


- Sridhar K


On July 8, 2019, 1:26 p.m., Saqeeb Shaikh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> ---
> 
> (Updated July 8, 2019, 1:26 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
> Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas server processes notifications from Kafka topic named 
> ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). 
> The name of the Kafka topic to which an Atlas Hook publishes notifications 
> can be configured via configuration atlas.notification.hook.topic.name. 
> However, since Atlas server processes notifications from a single topic, all 
> hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
> 
> To enable different hooks to publish to different Kafka topics, Atlas server 
> should support processing notifications from multiple Kafka topics.
> 
> 
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
>  (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
>   intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> 5c840c3 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> b04aba9 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 449eb6f 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 847caa3 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  263903b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  c963830 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
> d5a4412 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java 
> f658caa 
>   repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
> b56019e 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  1f8e810 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
>  fb3ff26 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
>  3e35511 
> 
> 
> Diff: https://reviews.apache.org/r/71006/diff/2/
> 
> 
> Testing
> ---
> 
> Tested by publishing create hive table JSON using kafka-console-producer on 
> custom topics configured above. I was able to search the entities in atlas. 
> Also tested basic use case by creating database and table from hive console. 
> 
> mvn clean package 
> Some test cases are failing on my local (will work on fixi

Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

2019-07-08 Thread Saqeeb Shaikh 


> On July 3, 2019, 4:19 p.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java
> > Lines 324 (patched)
> > 
> >
> > Please add test for this.

Working on test cases for AtlasMetricUtil.java


- Saqeeb


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216346
---


On July 8, 2019, 1:26 p.m., Saqeeb Shaikh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> ---
> 
> (Updated July 8, 2019, 1:26 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
> Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas server processes notifications from Kafka topic named 
> ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). 
> The name of the Kafka topic to which an Atlas Hook publishes notifications 
> can be configured via configuration atlas.notification.hook.topic.name. 
> However, since Atlas server processes notifications from a single topic, all 
> hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
> 
> To enable different hooks to publish to different Kafka topics, Atlas server 
> should support processing notifications from multiple Kafka topics.
> 
> 
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
>  (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
>   intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> 5c840c3 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> b04aba9 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 449eb6f 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 847caa3 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  263903b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  c963830 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
> d5a4412 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java 
> f658caa 
>   repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
> b56019e 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  1f8e810 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
>  fb3ff26 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
>  3e35511 
> 
> 
> Diff: https://reviews.apache.org/r/71006/diff/2/
> 
> 
> Testing
> ---
> 
> Tested by publishing create hive table JSON using kafka-console-producer on 
> custom topics configured above. I was able to search the entities in atlas. 
> Also tested basic use case by creating database and table from hive console. 
> 
> mvn clean package 
> Some test cases are failing on my local (will work on fixing the same).
> 
> 
> Thanks,
> 
> Saqeeb Shaikh
> 
>

Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

2019-07-08 Thread Saqeeb Shaikh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/
---

(Updated July 8, 2019, 1:26 p.m.)


Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
Rodrigues, and Sarath Subramanian.


Changes
---

Addressed review comments


Bugs: ATLAS-3313
https://issues.apache.org/jira/browse/ATLAS-3313


Repository: atlas


Description
---

Currently Atlas server processes notifications from Kafka topic named 
ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The 
name of the Kafka topic to which an Atlas Hook publishes notifications can be 
configured via configuration atlas.notification.hook.topic.name. However, since 
Atlas server processes notifications from a single topic, all hooks 
(Hive/HBase/Spark/Storm/..) are required to publish to the same topic.

To enable different hooks to publish to different Kafka topics, Atlas server 
should support processing notifications from multiple Kafka topics.


Add following properties in atlas:
atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
 (comma separated without space)
atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
  intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
5c840c3 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
b04aba9 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
449eb6f 
  notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
847caa3 
  
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
 263903b 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
 c963830 
  repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
d5a4412 
  repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa 
  repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
b56019e 
  
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 1f8e810 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
 fb3ff26 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
 3e35511 


Diff: https://reviews.apache.org/r/71006/diff/2/

Changes: https://reviews.apache.org/r/71006/diff/1-2/


Testing
---

Tested by publishing create hive table JSON using kafka-console-producer on 
custom topics configured above. I was able to search the entities in atlas. 
Also tested basic use case by creating database and table from hive console. 

mvn clean package 
Some test cases are failing on my local (will work on fixing the same).


Thanks,

Saqeeb Shaikh

Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

2019-07-03 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216360
---




notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 240 (original), 275 (patched)


kafka consumer needs to be checked if its not open already - 
"!isKafkaConsumerOpen(consumer)". This check is needed because of ATLAS-2751.


- Sarath Subramanian


On July 3, 2019, 7:52 a.m., Saqeeb Shaikh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> ---
> 
> (Updated July 3, 2019, 7:52 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
> Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas server processes notifications from Kafka topic named 
> ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). 
> The name of the Kafka topic to which an Atlas Hook publishes notifications 
> can be configured via configuration atlas.notification.hook.topic.name. 
> However, since Atlas server processes notifications from a single topic, all 
> hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
> 
> To enable different hooks to publish to different Kafka topics, Atlas server 
> should support processing notifications from multiple Kafka topics.
> 
> 
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
>  (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
>   intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> 5c840c3 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> b04aba9 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 449eb6f 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  263903b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  c963830 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
> d5a4412 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java 
> f658caa 
>   repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
> b56019e 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  1f8e810 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
>  fb3ff26 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
>  3e35511 
> 
> 
> Diff: https://reviews.apache.org/r/71006/diff/1/
> 
> 
> Testing
> ---
> 
> Tested by publishing create hive table JSON using kafka-console-producer on 
> custom topics configured above. I was able to search the entities in atlas. 
> Also tested basic use case by creating database and table from hive console. 
> 
> mvn clean package 
> Some test cases are failing on my local (will work on fixing the same).
> 
> 
> Thanks,
> 
> Saqeeb Shaikh
> 
>

Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

2019-07-03 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216349
---




notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 187 (original), 221 (patched)


Consider renaming createProducer() as getOrCreateProducer() and replace 
#218 - #223 with:

  KafkaProducer producer = getOrCreateProducer(type);



notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 271 (original), 316 (patched)


move 'synchronized' inside, to cover lines #322 - #324



notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 272 (original), 317 (patched)


LOG.info => LOG.debug - in #317 and #327.


- Madhan Neethiraj


On July 3, 2019, 2:52 p.m., Saqeeb Shaikh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> ---
> 
> (Updated July 3, 2019, 2:52 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
> Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas server processes notifications from Kafka topic named 
> ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). 
> The name of the Kafka topic to which an Atlas Hook publishes notifications 
> can be configured via configuration atlas.notification.hook.topic.name. 
> However, since Atlas server processes notifications from a single topic, all 
> hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
> 
> To enable different hooks to publish to different Kafka topics, Atlas server 
> should support processing notifications from multiple Kafka topics.
> 
> 
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
>  (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
>   intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> 5c840c3 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> b04aba9 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 449eb6f 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  263903b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  c963830 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
> d5a4412 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java 
> f658caa 
>   repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
> b56019e 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  1f8e810 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
>  fb3ff26 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
>  3e35511 
> 
> 
> Diff: https://reviews.apache.org/r/71006/diff/1/
> 
> 
> Testing
> ---
> 
> Tested by publishing create hive table JSON using kafka-console-producer on 
> custom topics configured above. I was able to search the entities in atlas. 
> Also tested basic use case by creating database and table from hive console. 
> 
> mvn clean package 
> Some test cases are failing on my local (will work on fixing the same).
> 
> 
> Thanks,
> 
> Saqeeb Shaikh
> 
>

Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

2019-07-03 Thread Ashutosh Mestry via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216346
---




notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
Line 185 (original), 217 (patched)


Rename type to notificationType for clarity.



repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java
Lines 324 (patched)


Please add test for this.


- Ashutosh Mestry


On July 3, 2019, 2:52 p.m., Saqeeb Shaikh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> ---
> 
> (Updated July 3, 2019, 2:52 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
> Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas server processes notifications from Kafka topic named 
> ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). 
> The name of the Kafka topic to which an Atlas Hook publishes notifications 
> can be configured via configuration atlas.notification.hook.topic.name. 
> However, since Atlas server processes notifications from a single topic, all 
> hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
> 
> To enable different hooks to publish to different Kafka topics, Atlas server 
> should support processing notifications from multiple Kafka topics.
> 
> 
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
>  (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
>   intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> 5c840c3 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> b04aba9 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 449eb6f 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  263903b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  c963830 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
> d5a4412 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java 
> f658caa 
>   repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
> b56019e 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  1f8e810 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
>  fb3ff26 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
>  3e35511 
> 
> 
> Diff: https://reviews.apache.org/r/71006/diff/1/
> 
> 
> Testing
> ---
> 
> Tested by publishing create hive table JSON using kafka-console-producer on 
> custom topics configured above. I was able to search the entities in atlas. 
> Also tested basic use case by creating database and table from hive console. 
> 
> mvn clean package 
> Some test cases are failing on my local (will work on fixing the same).
> 
> 
> Thanks,
> 
> Saqeeb Shaikh
> 
>

Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

2019-07-03 Thread Ashutosh Mestry via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216345
---



Can you please add a unit test to KafkaConsumerTest to include this scenario.

Overall impelemnetation looks simple enough. We are still little away for 
scaling Atlas.

I was expecting that the NotificationHookConsumer will consume a particular 
topic. But I suppose that will need another set of changes.

- Ashutosh Mestry


On July 3, 2019, 2:52 p.m., Saqeeb Shaikh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> ---
> 
> (Updated July 3, 2019, 2:52 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
> Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas server processes notifications from Kafka topic named 
> ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). 
> The name of the Kafka topic to which an Atlas Hook publishes notifications 
> can be configured via configuration atlas.notification.hook.topic.name. 
> However, since Atlas server processes notifications from a single topic, all 
> hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
> 
> To enable different hooks to publish to different Kafka topics, Atlas server 
> should support processing notifications from multiple Kafka topics.
> 
> 
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
>  (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
>   intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> 5c840c3 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> b04aba9 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 449eb6f 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  263903b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  c963830 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
> d5a4412 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java 
> f658caa 
>   repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
> b56019e 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  1f8e810 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
>  fb3ff26 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
>  3e35511 
> 
> 
> Diff: https://reviews.apache.org/r/71006/diff/1/
> 
> 
> Testing
> ---
> 
> Tested by publishing create hive table JSON using kafka-console-producer on 
> custom topics configured above. I was able to search the entities in atlas. 
> Also tested basic use case by creating database and table from hive console. 
> 
> mvn clean package 
> Some test cases are failing on my local (will work on fixing the same).
> 
> 
> Thanks,
> 
> Saqeeb Shaikh
> 
>

Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

2019-07-03 Thread Saqeeb Shaikh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/
---

Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-3313
https://issues.apache.org/jira/browse/ATLAS-3313


Repository: atlas


Description
---

Currently Atlas server processes notifications from Kafka topic named 
ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The 
name of the Kafka topic to which an Atlas Hook publishes notifications can be 
configured via configuration atlas.notification.hook.topic.name. However, since 
Atlas server processes notifications from a single topic, all hooks 
(Hive/HBase/Spark/Storm/..) are required to publish to the same topic.

To enable different hooks to publish to different Kafka topics, Atlas server 
should support processing notifications from multiple Kafka topics.


Add following properties in atlas:
atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
 (comma separated without space)
atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES


Diffs
-

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
  intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
5c840c3 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
b04aba9 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
449eb6f 
  
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
 263903b 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
 c963830 
  repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
d5a4412 
  repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa 
  repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
b56019e 
  
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 1f8e810 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
 fb3ff26 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
 3e35511 


Diff: https://reviews.apache.org/r/71006/diff/1/


Testing
---

Tested by publishing create hive table JSON using kafka-console-producer on 
custom topics configured above. I was able to search the entities in atlas. 
Also tested basic use case by creating database and table from hive console. 

mvn clean package 
Some test cases are failing on my local (will work on fixing the same).


Thanks,

Saqeeb Shaikh