[
https://issues.apache.org/jira/browse/HIVE-10165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621078#comment-14621078
]
Eugene Koifman commented on HIVE-10165:
---------------------------------------
Hive had several implementations of lock managers prior to ACID work that the
user could choose among.
https://cwiki.apache.org/confluence/display/Hive/Locking applies to behavior of
those managers.
When you choose to use hive.txn.manager=DbTxnManager, you get a new lock
manager implementation that is installed automatically. This lock manger is
designed to support Snapshot Isolation.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowLocks
mentions the type of locks it has. The terminology is somewhat inconsistent
but basically there are 3 types of locks
1. S/read/shared/shared_read - acquired to read or insert
2. SS/write/semi_shared/shared_write - acquired to update/delete
3. X/exclusive - acquired for "drop" type DDL operations.
X - conflicts with everything.
SS - compatible with S but conflicts with other SS (thus there is at most 1
update/delete happening concurrently)
There are 3 types of resources that can be locked: database, table and
partition.
The current (acid) lock manager is built to support auto-commit mode where all
locks for a given transaction are known at the start of the txn. As you
mention in the docs it doesn't yet have/need deadlock detection.
So given the current state of things, I would vote to have the Mutable
Streaming API acquire SS locks on tables that it writes. This will of course
limit concurrent writes but I think correctness (i.e. properly working Snapshot
Isolation) is more important.
I intend to build deadlock detection as part of HIVE-9675, which will hopefully
happen in the next couple of month. Then it will be possible to acquire
partition level locks as you go.
There is one crucial concern here: where do you know that you need to lock a
partition? If it's in the MutatorClient or Mutator? (If I understood things
correctly, the former runs on the client and the later on the grid). Since the
current lock mangers uses the metastore RDBMS to manage it's state, you cannot
"talk to metastore" from the grid as it will likely overwhelm it. So if
Mutator is the entity that needs to acquire locks, this would have to wait for
the HBase based metastore that [~alangates] is working on (and I believe will
only be available in branch-2 aka master).
> Improve hive-hcatalog-streaming extensibility and support updates and deletes.
> ------------------------------------------------------------------------------
>
> Key: HIVE-10165
> URL: https://issues.apache.org/jira/browse/HIVE-10165
> Project: Hive
> Issue Type: Improvement
> Components: HCatalog
> Affects Versions: 1.2.0
> Reporter: Elliot West
> Assignee: Elliot West
> Labels: TODOC2.0, streaming_api
> Fix For: 2.0.0
>
> Attachments: HIVE-10165.0.patch, HIVE-10165.10.patch,
> HIVE-10165.4.patch, HIVE-10165.5.patch, HIVE-10165.6.patch,
> HIVE-10165.7.patch, HIVE-10165.9.patch, mutate-system-overview.png
>
>
> h3. Overview
> I'd like to extend the
> [hive-hcatalog-streaming|https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest]
> API so that it also supports the writing of record updates and deletes in
> addition to the already supported inserts.
> h3. Motivation
> We have many Hadoop processes outside of Hive that merge changed facts into
> existing datasets. Traditionally we achieve this by: reading in a
> ground-truth dataset and a modified dataset, grouping by a key, sorting by a
> sequence and then applying a function to determine inserted, updated, and
> deleted rows. However, in our current scheme we must rewrite all partitions
> that may potentially contain changes. In practice the number of mutated
> records is very small when compared with the records contained in a
> partition. This approach results in a number of operational issues:
> * Excessive amount of write activity required for small data changes.
> * Downstream applications cannot robustly read these datasets while they are
> being updated.
> * Due to scale of the updates (hundreds or partitions) the scope for
> contention is high.
> I believe we can address this problem by instead writing only the changed
> records to a Hive transactional table. This should drastically reduce the
> amount of data that we need to write and also provide a means for managing
> concurrent access to the data. Our existing merge processes can read and
> retain each record's {{ROW_ID}}/{{RecordIdentifier}} and pass this through to
> an updated form of the hive-hcatalog-streaming API which will then have the
> required data to perform an update or insert in a transactional manner.
> h3. Benefits
> * Enables the creation of large-scale dataset merge processes
> * Opens up Hive transactional functionality in an accessible manner to
> processes that operate outside of Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
