[
https://issues.apache.org/jira/browse/HIVE-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019049#comment-16019049
]
Sushanth Sowmyan commented on HIVE-16164:
-----------------------------------------
Hi, just making a note here, post patch:
a) I like that we did not reuse EnvironmentContext for this, and instead went
with the extra maps that was passed only to the "post"-event-listeners.
b) I like the refactor usage of MetastoreEventNotifier - I think it does make
code usage from HMSHandler and from tests easier to be generic.
c) If you need another pair of eyes on DbNotificationListener or Listener
changes in general, please feel free to ping me or [~thejas], we tend to be
looking at these classes for our replication work. I will admit to hoping to
move DbNotificationListener from the hcat package to hive packages, and was
hoping to deprecate-move it if there were no other users, but I'd not started
discussion on that yet. If you're amenable to that discussion, we could still
do so.
d) Also, as an additional note, [~akolb] mentioned a generic usecase of a
persistence id or a kafka-based one - that is definitely something we're
considering strongly, in that over time, we might not want to have the events
in the metastore db itself since it can be a choke-point over time, and things
like Kafka-based systems are under consideration(probably only after we get the
replication overhaul done), and to this end, to consumers of events, we're
looking at org.apache.hadoop.hive.metastore.messaging.* as our generic way of
consuming events - currently we have a metastore-based notification fetcher as
our only implementation, but the goal is to try and expand that. We still
intend that event ids should be generated on the metastore side, just that it
may not always happen on the metastore db itself.
> Provide mechanism for passing HMS notification ID between transactional and
> non-transactional listeners.
> --------------------------------------------------------------------------------------------------------
>
> Key: HIVE-16164
> URL: https://issues.apache.org/jira/browse/HIVE-16164
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Sergio Peña
> Assignee: Sergio Peña
> Fix For: 2.3.0, 3.0.0, 2.4.0
>
> Attachments: HIVE-16164.1.patch, HIVE-16164.2.patch,
> HIVE-16164.3.patch, HIVE-16164.6.patch, HIVE-16164.7.patch, HIVE-16164.8.patch
>
>
> The HMS DB notification listener currently stores an event ID on the HMS
> backend DB so that external applications (such as backup apps) can request
> incremental notifications based on the last event ID requested.
> The HMS DB notification and backup applications are asynchronous. However,
> there are sometimes that applications may be required to be in sync with the
> latest HMS event in order to process an action. These applications will
> provide a listener implementation that is called by the HMS after an HMS
> transaction happened.
> The problem is that the listener running after the transaction (or during the
> non-transactional context) may need the DB event ID in order to sync all
> events happened previous to that event ID, but this ID is never passed to the
> non-transactional listeners.
> We can pass this event information through the EnvironmentContext found on
> each ListenerEvent implementations (such as CreateTableEvent), and send the
> EnvironmentContext to the non-transactional listeners to get the event ID.
> The DbNotificactionListener already knows the event ID after calling the
> ObjectStore.addNotificationEvent(). We just need to set this event ID to the
> EnvironmentContext from each of the event notifications and make sure that
> this EnvironmentContext is sent to the non-transactional listeners.
> Here's the code example when creating a table on {{create_table_core}}:
> {noformat}
> ms.createTable(tbl);
> if (transactionalListeners.size() > 0) {
> CreateTableEvent createTableEvent = new CreateTableEvent(tbl, true, this);
> createTableEvent.setEnvironmentContext(envContext);
> for (MetaStoreEventListener transactionalListener :
> transactionalListeners) {
> transactionalListener.onCreateTable(createTableEvent); // <-
> Here the notification ID is generated
> }
> }
> success = ms.commitTransaction();
> } finally {
> if (!success) {
> ms.rollbackTransaction();
> if (madeDir) {
> wh.deleteDir(tblPath, true);
> }
> }
> for (MetaStoreEventListener listener : listeners) {
> CreateTableEvent createTableEvent =
> new CreateTableEvent(tbl, success, this);
> createTableEvent.setEnvironmentContext(envContext);
> listener.onCreateTable(createTableEvent); // <-
> Here we would like to consume notification ID
> }
> {noformat}
> We could use a specific key name that will be used on the EnvironmentContext,
> such as DB_NOTIFICATION_EVENT_ID.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
