[
https://issues.apache.org/jira/browse/HIVE-20198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16548724#comment-16548724
]
Vihang Karajgaonkar edited comment on HIVE-20198 at 7/19/18 3:00 AM:
---------------------------------------------------------------------
{{TBLS.TBL_ID}} is internally managed by datanucleus and not exposed to the
Thrift Table object. Its value is dependent on the backing database according
to datanucleus documentation (although I have almost always seen it as a
monotonically increasing number). Its value is guaranteed to be unique so I
think we can potentially use it. But in my opinion there is still value to
expose such a id which is controlled by metastore at the thrift level. For
instance such ids can be used to identify versions of the objects to provide
optimistic concurrency control model instead of the lock based concurrency
model we have currently. When a table object which is altered metastore can
change uuid but different currently the TBL_ID which managed by datanucleus
does not change as far as I understand.
was (Author: vihangk1):
{{TBLS.TBL_ID}} is internally managed by datanucleus and not exposed to the
Thrift Table object. Its value is dependent on the backing database according
to datanucleus documentation (although I have almost always seen it as a
monotonically increasing number). Its value is guaranteed to be unique so I
think we can potentially use it. But in my opinion there is still value to
expose such a id which is controlled by metastore at the thrift level. For
instance such ids can be used to identify versions of the objects to provide
optimistic concurrency control model instead of the lock based concurrency
model we have currently. A table object which is altered can have a different
uuid but not different currently the TBL_ID does not change.
> Constant time table drops/renames
> ---------------------------------
>
> Key: HIVE-20198
> URL: https://issues.apache.org/jira/browse/HIVE-20198
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Affects Versions: 4.0.0
> Reporter: Alexander Kolbasov
> Priority: Major
>
> Currently table drops and table renames have O(P) performance (where P is the
> number of partitions). When a managed table is deleted, the implementation
> deletes table metadata and then deletes all partitions in HDFS. HDFS
> operations are optimized and only do a sequential deletes for partitions
> outside of table prefix. This operation is O(P)where Pis the number of
> partitions.
> Table rename goes through the list of partitions and modifies table name (and
> potentially db name) in each partition. It also modifies each partition
> location to match the new db/table name and renames directories (which is a
> non-atomic and slow operation on S3). This is O(P) operation where P is the
> number of partitions.
> Basic idea is to do the following:
> # Assign unique ID to each table
> # Create directory name based on unique ID rather then the name
> # Table rename then becomes metadata-only operation - there is no need to
> change any location information.
> # Table drop can become an asynchronous operation where the table is marked
> as "deleted". Subsequent public metadata APIs should skip such tables. A
> background cleaner thread may then go and clean up directories.
> Since the table location is unique for each table, new tables will not reuse
> existing locations. This change isn't compatible with the current behavior
> where there is an assumption that table location is based on table name. We
> can get around this by providing "opt-in" mechanism - special table property
> that tells that the table can have such new behavior, so the improvement will
> initially work for new tables created with this feature enabled. We may later
> provide some tool to convert existing tables to the new scheme.
> One complication is there in case where impersonation is enabled - the FS
> operations should be performed using client UGI rather then server's, so the
> cleaner thread should be able to use client UGIs.
> Initially we can punt on this and do standard table drops when impersonation
> is enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
