Sankar Hariappan created HIVE-20264:
---------------------------------------
Summary: Bootstrap repl dump with concurrent write and drop of
ACID table makes target inconsistent.
Key: HIVE-20264
URL: https://issues.apache.org/jira/browse/HIVE-20264
Project: Hive
Issue Type: Sub-task
Components: HiveServer2, repl
Affects Versions: 4.0.0, 3.2.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
During bootstrap dump of ACID tables, let's consider the below sequence.
- Get lastReplId = last event ID logged.
- Current session (Thread-1), REPL DUMP -> Open txn (Txn1) - Event-10
- Another session (Thread-2), Open txn (Txn2) - Event-11
- Thread-2 -> Insert data (T1.D1) to ACID table. - Event-12
- Thread-2 -> Commit Txn (Txn2) - Event-13
- Thread-2 -> Drop table (T1) - Event-14
- Thread-1 -> Dump ACID tables based on validTxnList based on Txn1. --> This
step skips all the data written by txns > Txn1. So, T1 will be missing.
- Thread-1 -> Commit Txn (Txn1)
- REPL LOAD from bootstrap dump will skip T1.
- Incremental REPL DUMP will start from Event-10 and hence allocate write id
for table T1 and drop table(T1) is idempotent. So, at target, exist entries in
TXN_TO_WRITE_ID and NEXT_WRITE_ID metastore tables.
- Now, when we create another table at source with same name T1 and replicate,
then it may lead to incorrect data for readers at target on T1.
Couple of proposals:
1. Make allocate write ID idempotent which is not possible as table doesn't
exist and MM table import may lead to allocate write id before creating table.
So, cannot differentiate these 2 cases.
2. Make Drop table event to drop entries from TXN_TO_WRITE_ID and NEXT_WRITE_ID
tables irrespective of table exist or not at target.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
