----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72470/ -----------------------------------------------------------
(Updated May 7, 2020, 2:49 p.m.) Review request for hive, Marton Bod, Peter Varga, and Peter Vary. Changes ------- removed not related change Bugs: HIVE-23349 https://issues.apache.org/jira/browse/HIVE-23349 Repository: hive-git Description ------- 2 concurrent MERGE INSERT operations generate duplicates due to lack of locking. MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or EXCL_WRITE if hive.txn.write.xlock=false; create table target (a int, b int) stored as orc TBLPROPERTIES ('transactional'='true')"); insert into target values (1,2), (3,4) create table source (a int, b int) execute in parallel: insert into source values (5,6), (7,8) PS: Current patch doesn cover following scenario: 1) T1 (merge-insert) opens txns & records snapshot; 2) T2 (insert/merge-insert) opens txns, records snapshot & locks it; 3) T2 commits it's changes and unlocks T1; 4) T1 locks snapshot and validates txn list, however only txns with txnId lower than T1's is beeing considered, T2 changes are ignored -> duplicates are generated. merge-insert/merge-insert scenario could be fixed by leveraging write-write conflict check mechanism. We just need to set operation type for merge-insert to update. However it won't solve issue with merge-insert/insert. We should consider moving locking before snapshot generation, current snapshot re-check mechanism doesn't handle described scenarios. Diffs (updated) ----- ql/src/java/org/apache/hadoop/hive/ql/Context.java 9f59d4cea3 ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java c1f94d165b ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 998c05e37d ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java deaab89c1f ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java 3ffdcec528 ql/src/test/org/apache/hadoop/hive/ql/lockmgr/DbTxnManagerEndToEndTestBase.java b435e79c3c ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 1687425bcb ql/src/test/results/clientpositive/llap/explain_locks.q.out d62f6ccafd Diff: https://reviews.apache.org/r/72470/diff/5/ Changes: https://reviews.apache.org/r/72470/diff/4-5/ Testing ------- Manual, added number of merge related test scenarios into TestDbTxnManager2, modified explain_locks.q Thanks, Denys Kuzmenko