Eugene Koifman created HIVE-16564:
-------------------------------------

             Summary: StreamingAPI is locking too much?
                 Key: HIVE-16564
                 URL: https://issues.apache.org/jira/browse/HIVE-16564
             Project: Hive
          Issue Type: Bug
          Components: HCatalog, Transactions
    Affects Versions: 1.0.0
            Reporter: Eugene Koifman
            Assignee: Eugene Koifman


Currently _TransactionBatchImpl.beginNextTransactionImpl()_ acquires Shared 
locks for each Transaction in the batch.  
Especially under high load this creates pressure on the LockManager (i.e. 
Metastore) and degrades performance of Ingest itself.
Because all transactions in a batch write to the same physical file and the 
fact that for Acid tables (which are required for Streaming Ingest) shared 
locks only protect against Exclusive locks (like drop table), 
acquiring/releasing locks doesn't for each txn doesn't achieve much.

One possibility to acquire all locks (i.e. for all txns) at the time the batch 
is created (same as is done for openTxn() for all txns in the batch).  Locks 
for each txn in the batch will be released automatically when commit is called 
for the respective txn.

Alternatively, don't acquire any locks - this means someone may drop a table 
while it's written to but using locks here doesn't buy much.  Say a Drop 
request is issued when a write is in progress.  It will block until the write 
releases it's lock and execute immediately after that.  Thus none of the data 
of that write is visible for any meaningful length of time anyway.

Allow a "meta lock" - a lock not associated with any specific txn, that is held 
for the duration of the TransactionBatch.  This sort of breaks the model 
(especially since HIVE-12636).  Perhaps each batch can open one "extra" txn for 
internal purposes, just to acquire this "meta lock".  No data will ever be 
tagged with this "extra" txn.







--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Reply via email to