[ https://issues.apache.org/jira/browse/ATLAS-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391990#comment-17391990 ]
ASF subversion and git services commented on ATLAS-3427: -------------------------------------------------------- Commit 6afeef4cc102931a51b12b9d365b23787b302f19 in atlas's branch refs/heads/branch-2.0 from Ashutosh Mestry [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=6afeef4 ] ATLAS-3427: Atlas Hook Enhancements for improved resiliancy. (cherry picked from commit de87bc5022627d82cb8e6048b6728e7028a4af25) > Atlas hooks enhancements for better fault-tolerance i.e. Kafka unavailability > ----------------------------------------------------------------------------- > > Key: ATLAS-3427 > URL: https://issues.apache.org/jira/browse/ATLAS-3427 > Project: Atlas > Issue Type: Improvement > Reporter: Nixon Rodrigues > Assignee: Ashutosh Mestry > Priority: Major > > *Background* > Existing implementation for sending messages from the hooks relies on Kafka. > Which means that if Kafka is not available for any reasons, the messages do > not make it to Atlas. They are simply added to the _failed_._log_. > *Solution* > Introduce a mechanism which preserves messages if destination for the > messages is not reachable. This new mechanism will also have logic to retry > these messages. > *Requirements* > The mechanism should be: > * Transparent: Existing hooks should not have to rework their > implementation. > * Configurable: This should be introduced optionally. If hook chooses not to > use this, it should fallback to existing implementation. The behavior should > be configurable via properties. > * High-performance: Using this mechanism should not introduce additional > overhead. Ideally, there should not be additional serialization introduced. > * Reliable: The new mechanism should relay these messages reliably. Existing > message formats should be supported viz. plain, compressed, compress & > multi-part. > -- This message was sent by Atlassian Jira (v8.3.4#803005)