[
https://issues.apache.org/jira/browse/HIVE-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399904#comment-16399904
]
anishek commented on HIVE-18940:
--------------------------------
having a separate global commit id might not solve the problem since
association between the events ( if we use auto increment ) and the global
commit has to be done in the same txn. Also since we cant have ordering of the
commit ids different to the point of commits, lock on this table will have to
be taken to make sure one txn commits before the other one can commit. if
there was some way to do fine grained control such that txn's with commit ids
fire the db (COMMIT sql statement) in order of their id's. We can trim down the
time the lock is taken by doing some directsql and putting in commit-id based
ordering, which again across multiple metastores would add latency to the whole
process.
> Hive notifications serialize all write DDL operations
> -----------------------------------------------------
>
> Key: HIVE-18940
> URL: https://issues.apache.org/jira/browse/HIVE-18940
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 3.0.0
> Reporter: Alexander Kolbasov
> Priority: Major
>
> The implementation of DbNotificationListener uses a single row to store
> current notification ID and uses {{SELECT FOR UPDATE}} to lock the row. This
> serializes all write DDL operations which isn't good.
> We should consider using database auto-increment for notification ID instead.
> Especially on mMySQL/innoDb it is supported natively with relatively
> light-weight locking.
> This creates potential issue for consumers though because such IDs may have
> holes. There are two types of holes - transient hole for a transaction which
> have not committed yet and will be committed shortly and permanent holes for
> transactions that fail. Consumers need to deal with it. It may be useful to
> add DB-generated timestamp as well to assist in recovery from holes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
