sankarh commented on a change in pull request #541: HIVE-21197 : Hive
Replication can add duplicate data during migration to a target with
hive.strict.managed.tables enabled
URL: https://github.com/apache/hive/pull/541#discussion_r259589143
##########
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java
##########
@@ -289,12 +296,21 @@ private boolean shouldReplayEvent(FileStatus dir,
DumpType dumpType, String dbNa
return updateReplIdTask;
}
- private Task<? extends Serializable> dbUpdateReplStateTask(String dbName,
String replState,
+ private Task<? extends Serializable> dbUpdateReplStateTask(String dbName,
String replState, String incLoadPendFlag,
Task<? extends
Serializable> preCursor) {
HashMap<String, String> mapProp = new HashMap<>();
- mapProp.put(ReplicationSpec.KEY.CURR_STATE_ID.toString(), replState);
- AlterDatabaseDesc alterDbDesc = new AlterDatabaseDesc(dbName, mapProp, new
ReplicationSpec(replState, replState));
+ // if the update is for incLoadPendFlag, then send replicationSpec as null
to avoid replacement check.
+ ReplicationSpec replicationSpec = null;
+ if (incLoadPendFlag == null) {
+ mapProp.put(ReplicationSpec.KEY.CURR_STATE_ID.toString(), replState);
+ replicationSpec = new ReplicationSpec(replState, replState);
+ } else {
+ assert replState == null;
+ mapProp.put(ReplUtils.REPL_FIRST_INC_PENDING_FLAG, incLoadPendFlag);
Review comment:
There is one corner case for A->B->C where EVENT_ALTER_DATABASE should skip
this additional parameter while dumping.
Let's say, bootstrap and first incremental done in B. So, this flag is false.
Now, we trigger bootstrap dump from B->C and concurrently, there is a
alterDb operation which logs an event with this flag as false.
Now, when we bootstrap load in C, we set it there as true. During first
incremental load in C, when we process AlterDb event, we set it to false, so
that we open up compaction after this event and later events may not guarantee
duplicate check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
