[
https://issues.apache.org/jira/browse/ATLAS-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Radhika Kundam updated ATLAS-4335:
----------------------------------
Attachment: Ordering REST based Hook Notification.pdf
> Hook Notifications through Rest Interface
> -----------------------------------------
>
> Key: ATLAS-4335
> URL: https://issues.apache.org/jira/browse/ATLAS-4335
> Project: Atlas
> Issue Type: New Feature
> Components: atlas-core
> Reporter: Deep Singh
> Assignee: Radhika Kundam
> Priority: Critical
> Attachments: Ordering REST based Hook Notification.pdf
>
>
> In Data-lake <-> Workload-cluster scenario, versions of the services
> available on clusters are not always the same. Especially when Kafka versions
> are different on the clusters, many a time the Kafka client (on
> workload-cluster) becomes incompatible with the Kafka broker (on data-lake).
> This leads to failure in the delivery of Atlas Hook notification as the
> mechanism is dependent on the Kafka client's capability to deliver messages.
> To resolve such an issue there is no other way but to make Kafka client
> compatible with the broker by upgrading or degrading the client or the
> broker. Often this is not straight-fwd and the easiest thing to do.
> We need another mechanism independent of the Kafka client to deliver hook
> notifications to the Atlas consumer.
> There are various REST API available today to perform CURD on Atlas entities.
> Those APIs process notifications in real-time as they arrive, since event
> processing is slow and time-consuming these APIs cannot give the throughput
> which hooks require.
> We need another REST interface that is lightweight and does not process
> messages as they arrive instead, it could dump the messages to the local
> kafka. Since Kafka will be available locally in the cluster, the interface
> will never face the problem of version mismatch of client-broker, as faced by
> service hooks running on a different cluster.
> As publishing messages through REST notification is not instantaneous
> compared to Kafka notification, we need to explicitly handle the ordering of
> messages published through REST notification. Attached the document explains
> the new architecture of REST Notification and how it handles out of ordered
> messages.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
