[ https://issues.apache.org/jira/browse/HIVE-21115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740045#comment-16740045 ]
Vihang Karajgaonkar commented on HIVE-21115: -------------------------------------------- [~thejas] [~alangates] Do you have any thoughts or concerns with the above approach? If there are easier ways to do this I would be happy to use them instead of datanucleus based approach. > Add support for object versions in metastore > -------------------------------------------- > > Key: HIVE-21115 > URL: https://issues.apache.org/jira/browse/HIVE-21115 > Project: Hive > Issue Type: Improvement > Reporter: Vihang Karajgaonkar > Priority: Major > > Currently, metastore objects are identified uniquely by their names (eg. > catName, dbName and tblName for a table is unique). Once a table or partition > is created it could be altered in many ways. There is no good way currently > to identify the version of the object once it is altered. For example, > suppose there are two clients (Hive and Impala) using the same metastore. > Once some alter operations are performed by a client, another client which > wants to do a alter operation has no good way to know if the object which it > has is the same as the one stored in metastore. Metastore updates the > {{transient_lastDdlTime}} every time there is a DDL operation on the object. > However, this value cannot be relied for all the clients since after > HIVE-1768 metastore updates the value only when it is not set in the > parameters. It is possible that a client which alters the object state, does > not remove the {{transient_lastDdlTime}} and metastore will not update it. > Secondly, if there is a clock skew between multiple HMS instances when HMS-HA > is configured, time values cannot be relied on to find out the sequence of > alter operations on a given object. > This JIRA propose to use JDO versioning support by Datanucleus > http://www.datanucleus.org/products/accessplatform_4_2/jdo/versioning.html to > generate a incrementing sequence number every time a object is altered. The > value of this object can be set as one of the values in the parameters. The > advantage of using Datanucleus the versioning can be done across HMS > instances as part of the database transaction and it should work for all the > supported databases. > In theory such a version can be used to detect if the client is presenting a > object which is "stale" when issuing a alter request. Metastore can choose to > reject such a alter request since the client may be caching a old version of > the object and any alter operation on such stale object can potentially > overwrite previous operations. However, this is can be done in a separate > JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005)