Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-10-01 Thread Abhay Kulkarni 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/#review186808
---


Ship it!




- Abhay Kulkarni


On Sept. 30, 2017, 9:58 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62681/
> ---
> 
> (Updated Sept. 30, 2017, 9:58 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, 
> Sarath Subramanian, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-2075
> https://issues.apache.org/jira/browse/ATLAS-2075
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Update notification module to compress and split large messages (> 1mb) that 
> can't be sent via Kafka. When such a message is sent, the message is first 
> compressed via GZip. If compressed message is within the size limit, the 
> message is sent via Kafka. If the compressed size is larger, then the message 
> is split into multiple small messages and sent via Kafka.
> 
> Notification consumer is updated to merge and uncompress messages before the 
> message is processed further.
> 
> 
> Diffs
> -
> 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  1f9404de 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
>  b1ac2fa0 
> 
> 
> Diff: https://reviews.apache.org/r/62681/diff/6/
> 
> 
> Testing
> ---
> 
> - verified that large messages are compressed and split by Atlas hook, before 
> sending to Kafka
> - verified that Atlas server processes compressed, multi-part messages 
> correctly
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-30 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/
---

(Updated Sept. 30, 2017, 9:58 p.m.)


Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, Sarath 
Subramanian, and Suma Shivaprasad.


Changes
---

Addressed for recent review comment and updates to log messages.


Bugs: ATLAS-2075
https://issues.apache.org/jira/browse/ATLAS-2075


Repository: atlas


Description
---

Update notification module to compress and split large messages (> 1mb) that 
can't be sent via Kafka. When such a message is sent, the message is first 
compressed via GZip. If compressed message is within the size limit, the 
message is sent via Kafka. If the compressed size is larger, then the message 
is split into multiple small messages and sent via Kafka.

Notification consumer is updated to merge and uncompress messages before the 
message is processed further.


Diffs (updated)
-

  
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
 1f9404de 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
 b1ac2fa0 


Diff: https://reviews.apache.org/r/62681/diff/6/

Changes: https://reviews.apache.org/r/62681/diff/5-6/


Testing
---

- verified that large messages are compressed and split by Atlas hook, before 
sending to Kafka
- verified that Atlas server processes compressed, multi-part messages correctly


Thanks,

Madhan Neethiraj

Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-30 Thread Abhay Kulkarni 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/#review186787
---


Fix it, then Ship it!





notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
Line 198 (original), 193 (patched)


Consider using this instead.

byte[] encodedBytes = MESSAGE_COMPRESSION_ENABLED ? msgBytes : 
AtlasNotificationBaseMessage.encodeBase64(msgBytes);


- Abhay Kulkarni


On Sept. 30, 2017, 11:51 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62681/
> ---
> 
> (Updated Sept. 30, 2017, 11:51 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, 
> Sarath Subramanian, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-2075
> https://issues.apache.org/jira/browse/ATLAS-2075
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Update notification module to compress and split large messages (> 1mb) that 
> can't be sent via Kafka. When such a message is sent, the message is first 
> compressed via GZip. If compressed message is within the size limit, the 
> message is sent via Kafka. If the compressed size is larger, then the message 
> is split into multiple small messages and sent via Kafka.
> 
> Notification consumer is updated to merge and uncompress messages before the 
> message is processed further.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> d3b4e49e 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 38889ef8 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
>  ec99372d 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  cb44fc68 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
> 6ef407ac 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  956c85e0 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java
>  1929eb46 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
>  cc2099e7 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 9b712f44 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  b7474a0e 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  3b2a093b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  61107a9f 
>   
> notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java
>  587b7ebd 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
>  be324277 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
>  3724fd5f 
> 
> 
> Diff: https://reviews.apache.org/r/62681/diff/5/
> 
> 
> Testing
> ---
> 
> - verified that large messages are compressed and split by Atlas hook, before 
> sending to Kafka
> - verified that Atlas server processes compressed, multi-part messages 
> correctly
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-30 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/
---

(Updated Sept. 30, 2017, 11:51 a.m.)


Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, Sarath 
Subramanian, and Suma Shivaprasad.


Bugs: ATLAS-2075
https://issues.apache.org/jira/browse/ATLAS-2075


Repository: atlas


Description
---

Update notification module to compress and split large messages (> 1mb) that 
can't be sent via Kafka. When such a message is sent, the message is first 
compressed via GZip. If compressed message is within the size limit, the 
message is sent via Kafka. If the compressed size is larger, then the message 
is split into multiple small messages and sent via Kafka.

Notification consumer is updated to merge and uncompress messages before the 
message is processed further.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
d3b4e49e 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
38889ef8 
  
notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
 ec99372d 
  
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
 cb44fc68 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
 PRE-CREATION 
  notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
6ef407ac 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 956c85e0 
  
notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java 
1929eb46 
  
notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
 cc2099e7 
  notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
9b712f44 
  
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
 b7474a0e 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
 3b2a093b 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
 61107a9f 
  
notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java
 587b7ebd 
  
notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
 be324277 
  
notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
 3724fd5f 


Diff: https://reviews.apache.org/r/62681/diff/5/

Changes: https://reviews.apache.org/r/62681/diff/4-5/


Testing
---

- verified that large messages are compressed and split by Atlas hook, before 
sending to Kafka
- verified that Atlas server processes compressed, multi-part messages correctly


Thanks,

Madhan Neethiraj

Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-29 Thread Abhay Kulkarni 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/#review186757
---




notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
Lines 192 (patched)


Please base64 encode msgBytes if maxLengthExceedsLimit is true.


- Abhay Kulkarni


On Sept. 29, 2017, 10:32 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62681/
> ---
> 
> (Updated Sept. 29, 2017, 10:32 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, 
> Sarath Subramanian, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-2075
> https://issues.apache.org/jira/browse/ATLAS-2075
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Update notification module to compress and split large messages (> 1mb) that 
> can't be sent via Kafka. When such a message is sent, the message is first 
> compressed via GZip. If compressed message is within the size limit, the 
> message is sent via Kafka. If the compressed size is larger, then the message 
> is split into multiple small messages and sent via Kafka.
> 
> Notification consumer is updated to merge and uncompress messages before the 
> message is processed further.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> d3b4e49e 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 38889ef8 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
>  ec99372d 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  cb44fc68 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
> 6ef407ac 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  956c85e0 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java
>  1929eb46 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
>  cc2099e7 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 9b712f44 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  b7474a0e 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  3b2a093b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  61107a9f 
>   
> notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java
>  587b7ebd 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
>  be324277 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
>  3724fd5f 
> 
> 
> Diff: https://reviews.apache.org/r/62681/diff/4/
> 
> 
> Testing
> ---
> 
> - verified that large messages are compressed and split by Atlas hook, before 
> sending to Kafka
> - verified that Atlas server processes compressed, multi-part messages 
> correctly
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-29 Thread Ashutosh Mestry 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/#review186755
---


Fix it, then Ship it!




Ship It!


notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
Lines 204 (patched)


Consider using Math.Ceil.


- Ashutosh Mestry


On Sept. 29, 2017, 10:32 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62681/
> ---
> 
> (Updated Sept. 29, 2017, 10:32 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, 
> Sarath Subramanian, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-2075
> https://issues.apache.org/jira/browse/ATLAS-2075
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Update notification module to compress and split large messages (> 1mb) that 
> can't be sent via Kafka. When such a message is sent, the message is first 
> compressed via GZip. If compressed message is within the size limit, the 
> message is sent via Kafka. If the compressed size is larger, then the message 
> is split into multiple small messages and sent via Kafka.
> 
> Notification consumer is updated to merge and uncompress messages before the 
> message is processed further.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> d3b4e49e 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 38889ef8 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
>  ec99372d 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  cb44fc68 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
> 6ef407ac 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  956c85e0 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java
>  1929eb46 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
>  cc2099e7 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 9b712f44 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  b7474a0e 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  3b2a093b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  61107a9f 
>   
> notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java
>  587b7ebd 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
>  be324277 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
>  3724fd5f 
> 
> 
> Diff: https://reviews.apache.org/r/62681/diff/4/
> 
> 
> Testing
> ---
> 
> - verified that large messages are compressed and split by Atlas hook, before 
> sending to Kafka
> - verified that Atlas server processes compressed, multi-part messages 
> correctly
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-29 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/
---

(Updated Sept. 29, 2017, 10:32 p.m.)


Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, Sarath 
Subramanian, and Suma Shivaprasad.


Changes
---

Updated for the review comments. Udpated to use utf-8 bytes length of the 
string (instead of string length) in tests for message-max-limit


Bugs: ATLAS-2075
https://issues.apache.org/jira/browse/ATLAS-2075


Repository: atlas


Description
---

Update notification module to compress and split large messages (> 1mb) that 
can't be sent via Kafka. When such a message is sent, the message is first 
compressed via GZip. If compressed message is within the size limit, the 
message is sent via Kafka. If the compressed size is larger, then the message 
is split into multiple small messages and sent via Kafka.

Notification consumer is updated to merge and uncompress messages before the 
message is processed further.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
d3b4e49e 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
38889ef8 
  
notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
 ec99372d 
  
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
 cb44fc68 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
 PRE-CREATION 
  notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
6ef407ac 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 956c85e0 
  
notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java 
1929eb46 
  
notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
 cc2099e7 
  notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
9b712f44 
  
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
 b7474a0e 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
 3b2a093b 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
 61107a9f 
  
notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java
 587b7ebd 
  
notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
 be324277 
  
notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
 3724fd5f 


Diff: https://reviews.apache.org/r/62681/diff/4/

Changes: https://reviews.apache.org/r/62681/diff/3-4/


Testing
---

- verified that large messages are compressed and split by Atlas hook, before 
sending to Kafka
- verified that Atlas server processes compressed, multi-part messages correctly


Thanks,

Madhan Neethiraj

Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-29 Thread Abhay Kulkarni 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/#review186683
---




common/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 36 (patched)


Consider if default value of this parameter can be directly referenced from 
some Kafka class.


- Abhay Kulkarni


On Sept. 29, 2017, 6:30 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62681/
> ---
> 
> (Updated Sept. 29, 2017, 6:30 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, 
> Sarath Subramanian, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-2075
> https://issues.apache.org/jira/browse/ATLAS-2075
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Update notification module to compress and split large messages (> 1mb) that 
> can't be sent via Kafka. When such a message is sent, the message is first 
> compressed via GZip. If compressed message is within the size limit, the 
> message is sent via Kafka. If the compressed size is larger, then the message 
> is split into multiple small messages and sent via Kafka.
> 
> Notification consumer is updated to merge and uncompress messages before the 
> message is processed further.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> d3b4e49e 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 38889ef8 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
>  ec99372d 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  cb44fc68 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
> 6ef407ac 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  956c85e0 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java
>  1929eb46 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
>  cc2099e7 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 9b712f44 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  b7474a0e 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  3b2a093b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  61107a9f 
>   
> notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java
>  587b7ebd 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
>  be324277 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
>  3724fd5f 
> 
> 
> Diff: https://reviews.apache.org/r/62681/diff/3/
> 
> 
> Testing
> ---
> 
> - verified that large messages are compressed and split by Atlas hook, before 
> sending to Kafka
> - verified that Atlas server processes compressed, multi-part messages 
> correctly
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-29 Thread Abhay Kulkarni 


> On Sept. 29, 2017, 3:15 a.m., Madhan Neethiraj wrote:
> > notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
> > Lines 163 (patched)
> > 
> >
> > This call is already used in Atlas (and in Ranger as well) to generate 
> > GUIDs. Do you have suggestions/reference for a more efficient way?

Please take a look at generateNextAuditEventId() in 
https://git-wip-us.apache.org/repos/asf?p=ranger.git;a=blob;f=agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java;h=0e124f1a65da05bec6ecbbb523907bfd3d2375ab;hb=refs/heads/master


- Abhay


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/#review186630
---


On Sept. 29, 2017, 6:30 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62681/
> ---
> 
> (Updated Sept. 29, 2017, 6:30 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, 
> Sarath Subramanian, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-2075
> https://issues.apache.org/jira/browse/ATLAS-2075
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Update notification module to compress and split large messages (> 1mb) that 
> can't be sent via Kafka. When such a message is sent, the message is first 
> compressed via GZip. If compressed message is within the size limit, the 
> message is sent via Kafka. If the compressed size is larger, then the message 
> is split into multiple small messages and sent via Kafka.
> 
> Notification consumer is updated to merge and uncompress messages before the 
> message is processed further.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> d3b4e49e 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 38889ef8 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
>  ec99372d 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  cb44fc68 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
> 6ef407ac 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  956c85e0 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java
>  1929eb46 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
>  cc2099e7 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 9b712f44 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  b7474a0e 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  3b2a093b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  61107a9f 
>   
> notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java
>  587b7ebd 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
>  be324277 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
>  3724fd5f 
> 
> 
> Diff: https://reviews.apache.org/r/62681/diff/3/
> 
> 
> Testing
> ---
> 
> - verified that large messages are compressed and split by Atlas hook, before 
> sending to Kafka
> - verified that Atlas server processes compressed, multi-part messages 
> correctly
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-29 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/
---

(Updated Sept. 29, 2017, 6:30 a.m.)


Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, Sarath 
Subramanian, and Suma Shivaprasad.


Changes
---

Addressed review comments. Added unit tests.


Bugs: ATLAS-2075
https://issues.apache.org/jira/browse/ATLAS-2075


Repository: atlas


Description
---

Update notification module to compress and split large messages (> 1mb) that 
can't be sent via Kafka. When such a message is sent, the message is first 
compressed via GZip. If compressed message is within the size limit, the 
message is sent via Kafka. If the compressed size is larger, then the message 
is split into multiple small messages and sent via Kafka.

Notification consumer is updated to merge and uncompress messages before the 
message is processed further.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
d3b4e49e 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
38889ef8 
  
notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
 ec99372d 
  
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
 cb44fc68 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
 PRE-CREATION 
  notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
6ef407ac 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 956c85e0 
  
notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java 
1929eb46 
  
notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
 cc2099e7 
  notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
9b712f44 
  
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
 b7474a0e 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
 3b2a093b 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
 61107a9f 
  
notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java
 587b7ebd 
  
notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
 be324277 
  
notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
 3724fd5f 


Diff: https://reviews.apache.org/r/62681/diff/3/

Changes: https://reviews.apache.org/r/62681/diff/2-3/


Testing
---

- verified that large messages are compressed and split by Atlas hook, before 
sending to Kafka
- verified that Atlas server processes compressed, multi-part messages correctly


Thanks,

Madhan Neethiraj

Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-28 Thread Apoorv Naik 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/#review186641
---




notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java
Line 122 (original), 119 (patched)


Might be a good time to get rid of Gson as well and use Jackson 
ObjectMapper across Atlas codebase.


- Apoorv Naik


On Sept. 29, 2017, 12:26 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62681/
> ---
> 
> (Updated Sept. 29, 2017, 12:26 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, 
> Sarath Subramanian, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-2075
> https://issues.apache.org/jira/browse/ATLAS-2075
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Update notification module to compress and split large messages (> 1mb) that 
> can't be sent via Kafka. When such a message is sent, the message is first 
> compressed via GZip. If compressed message is within the size limit, the 
> message is sent via Kafka. If the compressed size is larger, then the message 
> is split into multiple small messages and sent via Kafka.
> 
> Notification consumer is updated to merge and uncompress messages before the 
> message is processed further.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> d3b4e49e 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 38889ef8 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
>  ec99372d 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  cb44fc68 
>   
> notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
> 6ef407ac 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  956c85e0 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java
>  1929eb46 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
>  cc2099e7 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 9b712f44 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  b7474a0e 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  3b2a093b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  61107a9f 
>   
> notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java
>  587b7ebd 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
>  be324277 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
>  3724fd5f 
> 
> 
> Diff: https://reviews.apache.org/r/62681/diff/2/
> 
> 
> Testing
> ---
> 
> - verified that large messages are compressed and split by Atlas hook, before 
> sending to Kafka
> - verified that Atlas server processes compressed, multi-part messages 
> correctly
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-28 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/#review186630
---




notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
Lines 163 (patched)


This call is already used in Atlas (and in Ranger as well) to generate 
GUIDs. Do you have suggestions/reference for a more efficient way?



notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
Lines 168 (patched)


It is theoretically possible - especially for tiny messages. For such 
smaller messages, the execution wouldn't take this path due to 
"MESSAGE_MAX_LENGHT" check in line #161. This is unlikly for JSON strings, as 
they will have a lot of duplicate tokens that would result in good amount of 
size reduction.



notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
Lines 36 (patched)


Good catch! Will fix the name.



notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
Lines 74 (patched)


Will update to use local variable.

This API is called only from Kafka consumer runs - so it is called only 
from a single thread. However, do you see any issues with multi-threaded access?



notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
Lines 80 (patched)


Added a test case in HookMessageDeserializerTest.



notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
Lines 90 (patched)


Done.



notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
Lines 138 (patched)


Such validatin may not be helpful - as the sender might use a different 
value for MESSAGE_MAX_LENGTH.


- Madhan Neethiraj


On Sept. 29, 2017, 12:26 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62681/
> ---
> 
> (Updated Sept. 29, 2017, 12:26 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, 
> Sarath Subramanian, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-2075
> https://issues.apache.org/jira/browse/ATLAS-2075
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Update notification module to compress and split large messages (> 1mb) that 
> can't be sent via Kafka. When such a message is sent, the message is first 
> compressed via GZip. If compressed message is within the size limit, the 
> message is sent via Kafka. If the compressed size is larger, then the message 
> is split into multiple small messages and sent via Kafka.
> 
> Notification consumer is updated to merge and uncompress messages before the 
> message is processed further.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> d3b4e49e 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 38889ef8 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
>  ec99372d 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  cb44fc68 
>   
> notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
> 6ef407ac 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  956c85e0 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java
>  1929eb46 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
>  cc2099e7 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 9b712f44 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  b7474a0e 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  3b2a093b 
>   
>

Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-28 Thread Abhay Kulkarni 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/#review186624
---




notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
Lines 163 (patched)


Consider using a more efficient way to generating msgId strings.



notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
Lines 168 (patched)


Is it possible that compressedMsg will be longer than msgJson? If so, will 
it be better to send uncompressed msgJson?



notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
Lines 177 (patched)


Consider more efficient way of generating msgId string.



notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
Lines 36 (patched)


Please rename this class appropriately to 
AtlasNotificationMessageDeserializer.



notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
Lines 74 (patched)


Consider using a local variable to store store msgJson. Using formal 
argument to store changing value of msgJson is confusing to read.

Is this API called in a thread-safe way?



notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
Lines 80 (patched)


Is there a specific test case for this? If not, please consider writing one.



notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
Lines 90 (patched)


Please consider logging this as ERROR. This should never happen!



notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
Lines 138 (patched)


Consider adding another validation to ensure that the msgStr's size is 
equal to or less than MESSAGE_MAX_LENGTH.


- Abhay Kulkarni


On Sept. 29, 2017, 12:26 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62681/
> ---
> 
> (Updated Sept. 29, 2017, 12:26 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, 
> Sarath Subramanian, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-2075
> https://issues.apache.org/jira/browse/ATLAS-2075
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Update notification module to compress and split large messages (> 1mb) that 
> can't be sent via Kafka. When such a message is sent, the message is first 
> compressed via GZip. If compressed message is within the size limit, the 
> message is sent via Kafka. If the compressed size is larger, then the message 
> is split into multiple small messages and sent via Kafka.
> 
> Notification consumer is updated to merge and uncompress messages before the 
> message is processed further.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> d3b4e49e 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 38889ef8 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
>  ec99372d 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  cb44fc68 
>   
> notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
> 6ef407ac 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  956c85e0 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java
>  1929eb46 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
>  cc2099e7 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 9b712f44 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  b7474a0e 
>   
>

Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-28 Thread Ashutosh Mestry 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/#review186617
---



I would have liked more unit tests, since this is core functionality.


notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
Line 133 (original), 152 (patched)


I would suggest splitting this logic into methods, preferably classes. 
There is lot of logic that will help if there are tests.



notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
Lines 78 (patched)


Recommend splitting this in methods or classes. This logic can thus be 
testtable.



notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
Lines 36 (patched)


This class seems to be doing 3 different things. Will help to model it that 
way.



notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
Lines 130 (patched)


Consider encapsulating these methods into separate utility class.


- Ashutosh Mestry


On Sept. 28, 2017, 11:08 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62681/
> ---
> 
> (Updated Sept. 28, 2017, 11:08 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, 
> Sarath Subramanian, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-2075
> https://issues.apache.org/jira/browse/ATLAS-2075
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Update notification module to compress and split large messages (> 1mb) that 
> can't be sent via Kafka. When such a message is sent, the message is first 
> compressed via GZip. If compressed message is within the size limit, the 
> message is sent via Kafka. If the compressed size is larger, then the message 
> is split into multiple small messages and sent via Kafka.
> 
> Notification consumer is updated to merge and uncompress messages before the 
> message is processed further.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> d3b4e49e 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 38889ef8 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
>  ec99372d 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  cb44fc68 
>   
> notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
> 6ef407ac 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  956c85e0 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java
>  1929eb46 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
>  cc2099e7 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 9b712f44 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  b7474a0e 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  3b2a093b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  61107a9f 
>   
> notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java
>  587b7ebd 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
>  be324277 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
>  3724fd5f 
> 
> 
> Diff: https://reviews.apache.org/r/62681/diff/1/
> 
> 
> Testing
> ---
> 
> - verified that large messages are compressed and split by Atlas hook, before 
> sending to Kafka
> - verified that Atlas server processes compressed, multi-part messages 
> correctly
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-28 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/
---

Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, Sarath 
Subramanian, and Suma Shivaprasad.


Bugs: ATLAS-2075
https://issues.apache.org/jira/browse/ATLAS-2075


Repository: atlas


Description
---

Update notification module to compress and split large messages (> 1mb) that 
can't be sent via Kafka. When such a message is sent, the message is first 
compressed via GZip. If compressed message is within the size limit, the 
message is sent via Kafka. If the compressed size is larger, then the message 
is split into multiple small messages and sent via Kafka.

Notification consumer is updated to merge and uncompress messages before the 
message is processed further.


Diffs
-

  common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
d3b4e49e 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
38889ef8 
  
notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
 ec99372d 
  
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
 cb44fc68 
  
notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
 PRE-CREATION 
  notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
6ef407ac 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 956c85e0 
  
notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java 
1929eb46 
  
notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
 cc2099e7 
  notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
9b712f44 
  
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
 b7474a0e 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
 3b2a093b 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
 61107a9f 
  
notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java
 587b7ebd 
  
notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
 be324277 
  
notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
 3724fd5f 


Diff: https://reviews.apache.org/r/62681/diff/1/


Testing
---


Thanks,

Madhan Neethiraj

Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages

2017-09-28 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/
---

(Updated Sept. 28, 2017, 11:08 p.m.)


Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, Sarath 
Subramanian, and Suma Shivaprasad.


Bugs: ATLAS-2075
https://issues.apache.org/jira/browse/ATLAS-2075


Repository: atlas


Description
---

Update notification module to compress and split large messages (> 1mb) that 
can't be sent via Kafka. When such a message is sent, the message is first 
compressed via GZip. If compressed message is within the size limit, the 
message is sent via Kafka. If the compressed size is larger, then the message 
is split into multiple small messages and sent via Kafka.

Notification consumer is updated to merge and uncompress messages before the 
message is processed further.


Diffs
-

  common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
d3b4e49e 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
38889ef8 
  
notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
 ec99372d 
  
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
 cb44fc68 
  
notification/src/main/java/org/apache/atlas/notification/AlasNotificationMessageDeserializer.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
 PRE-CREATION 
  notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
6ef407ac 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 956c85e0 
  
notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java 
1929eb46 
  
notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
 cc2099e7 
  notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
9b712f44 
  
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
 b7474a0e 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
 3b2a093b 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
 61107a9f 
  
notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java
 587b7ebd 
  
notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
 be324277 
  
notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
 3724fd5f 


Diff: https://reviews.apache.org/r/62681/diff/1/


Testing (updated)
---

- verified that large messages are compressed and split by Atlas hook, before 
sending to Kafka
- verified that Atlas server processes compressed, multi-part messages correctly


Thanks,

Madhan Neethiraj