Sankar Hariappan created HIVE-20264: ---------------------------------------
Summary: Bootstrap repl dump with concurrent write and drop of ACID table makes target inconsistent. Key: HIVE-20264 URL: https://issues.apache.org/jira/browse/HIVE-20264 Project: Hive Issue Type: Sub-task Components: HiveServer2, repl Affects Versions: 4.0.0, 3.2.0 Reporter: Sankar Hariappan Assignee: Sankar Hariappan During bootstrap dump of ACID tables, let's consider the below sequence. - Get lastReplId = last event ID logged. - Current session (Thread-1), REPL DUMP -> Open txn (Txn1) - Event-10 - Another session (Thread-2), Open txn (Txn2) - Event-11 - Thread-2 -> Insert data (T1.D1) to ACID table. - Event-12 - Thread-2 -> Commit Txn (Txn2) - Event-13 - Thread-2 -> Drop table (T1) - Event-14 - Thread-1 -> Dump ACID tables based on validTxnList based on Txn1. --> This step skips all the data written by txns > Txn1. So, T1 will be missing. - Thread-1 -> Commit Txn (Txn1) - REPL LOAD from bootstrap dump will skip T1. - Incremental REPL DUMP will start from Event-10 and hence allocate write id for table T1 and drop table(T1) is idempotent. So, at target, exist entries in TXN_TO_WRITE_ID and NEXT_WRITE_ID metastore tables. - Now, when we create another table at source with same name T1 and replicate, then it may lead to incorrect data for readers at target on T1. Couple of proposals: 1. Make allocate write ID idempotent which is not possible as table doesn't exist and MM table import may lead to allocate write id before creating table. So, cannot differentiate these 2 cases. 2. Make Drop table event to drop entries from TXN_TO_WRITE_ID and NEXT_WRITE_ID tables irrespective of table exist or not at target. -- This message was sent by Atlassian JIRA (v7.6.3#76005)