[
https://issues.apache.org/jira/browse/ATLAS-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry updated ATLAS-3427:
-----------------------------------
Description:
*Background*
Existing implementation for sending messages from the hooks relies on Kafka.
Which means that if Kafka is not available for any reasons, the messages do not
make it to Atlas. They are simply added to the _failed_._log_.
*Solution*
Introduce a mechanism which preserves messages if destination for the messages
is not reachable. This new mechanism will also have logic to retry these
messages.
*Requirements*
The mechanism should be:
* Transparent: Existing hooks should not have to rework their implementation.
* Configurable: This should be introduced optionally. If hook chooses not to
use this, it should fallback to existing implementation. The behavior should be
configurable via properties.
* High-performance: Using this mechanism should not introduce additional
overhead. Ideally, there should not be additional serialization introduced.
* Reliable: The new mechanism should relay these messages reliably. Existing
message formats should be supported viz. plain, compressed, compress &
multi-part.
was:
*Background*
Existing implementation for sending messages from the hooks relies on Kafka.
Which means that if Kafka is not available for any reasons, the messages do not
make it to Atlas. They are simply added to the _failed_._log_.
*Solution*
Introduce a mechanism which preserves messages if destination for the messages
is not reachable. This new mechanism will also have logic to retry these
messages.
> Atlas hooks enhancements for better fault-tolerance i.e. Kafka unavailability
> -----------------------------------------------------------------------------
>
> Key: ATLAS-3427
> URL: https://issues.apache.org/jira/browse/ATLAS-3427
> Project: Atlas
> Issue Type: Improvement
> Reporter: Nixon Rodrigues
> Assignee: Ashutosh Mestry
> Priority: Major
>
> *Background*
> Existing implementation for sending messages from the hooks relies on Kafka.
> Which means that if Kafka is not available for any reasons, the messages do
> not make it to Atlas. They are simply added to the _failed_._log_.
> *Solution*
> Introduce a mechanism which preserves messages if destination for the
> messages is not reachable. This new mechanism will also have logic to retry
> these messages.
> *Requirements*
> The mechanism should be:
> * Transparent: Existing hooks should not have to rework their
> implementation.
> * Configurable: This should be introduced optionally. If hook chooses not to
> use this, it should fallback to existing implementation. The behavior should
> be configurable via properties.
> * High-performance: Using this mechanism should not introduce additional
> overhead. Ideally, there should not be additional serialization introduced.
> * Reliable: The new mechanism should relay these messages reliably. Existing
> message formats should be supported viz. plain, compressed, compress &
> multi-part.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
