[ https://issues.apache.org/jira/browse/ATLAS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Mestry updated ATLAS-4155: ----------------------------------- Attachment: ATLAS-4155-Kafka-commit-supplied-offset.patch > NotificationHookConsumer: Large Compressed Message Processing Problem > --------------------------------------------------------------------- > > Key: ATLAS-4155 > URL: https://issues.apache.org/jira/browse/ATLAS-4155 > Project: Atlas > Issue Type: Bug > Reporter: Ashutosh Mestry > Assignee: Ashutosh Mestry > Priority: Major > Attachments: ATLAS-4155-Kafka-commit-supplied-offset.patch > > > *Background* > Notification messages can be large in size. To get over Kafka's limitation on > message size, Atlas has compressed and split messages. If message size goes > beyond stipulated threshold, the message is compressed. If compressed message > goes beyond the size, it is split into multiple messages. > *Situation* > Consider a message that is so large that uncompressing it takes longer than > Kafka's timeout for message. This causes the problem where the large message > offset is not committed in time and that causes Kafka to present the same > message again. > Message Description: > Number of splits: 8 > Compressed message size: 7,452,640 > Uncompressed message size: 520,803,946 > Time taken to uncompress and stitch messages: > 90 seconds > > Sequence: > 2021-02-10 14:57:24,221: first message received > 2021-02-10 14:58:36,052: all splits combined – 72 seconds > 2021-02-10 15:01:06,971: message processing completed – 90 seconds > 2021-02-10 15:01:17,158: Kafka commit failed. Elapsed time since first > message: 197 seconds > 2021-02-10 15:01:19,857: attempt #2: first message received > 2021-02-10 15:03:01,993: attempt #2: all splits combined – 102 seconds > 2021-02-10 15:04:44,896: attempt #2: Kafka commit failed. Elapsed time since > first message: 205 seconds > Back to #5 > *Solution* > Maintain last offset received. If the same offset is presented, commit the > offset and move on to the next message. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)