[
https://issues.apache.org/jira/browse/HIVE-22068?focusedWorklogId=296244&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296244
]
ASF GitHub Bot logged work on HIVE-22068:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 16/Aug/19 11:58
Start Date: 16/Aug/19 11:58
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #742:
HIVE-22068 : Add more logging to notification cleaner and replication to track
events
URL: https://github.com/apache/hive/pull/742#discussion_r314690479
##########
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##########
@@ -750,6 +766,38 @@ public Table apply(@Nullable Table table) {
.verifyResults(Arrays.asList("1", "2"));
}
+ @Test
+ public void testIncrementalDumpEmptyDumpDirectory() throws Throwable {
+ WarehouseInstance.Tuple tuple = primary.run("use " + primaryDbName)
+ .run("create external table t1 (id int)")
+ .run("insert into table t1 values (1)")
+ .run("insert into table t1 values (2)")
+ .dump(primaryDbName, null);
+
+ replica.load(replicatedDbName, tuple.dumpLocation)
+ .status(replicatedDbName)
+ .verifyResult(tuple.lastReplicationId);
+
+ WarehouseInstance.Tuple incTuple = primary.dump(primaryDbName,
tuple.lastReplicationId);
+
+ replica.load(replicatedDbName, incTuple.dumpLocation)
+ .status(replicatedDbName)
+ .verifyResult(incTuple.lastReplicationId);
+
+ // create events for some other database and then dump the primaryDbName
to dump an empty directory.
+ primary.run("create database " + extraPrimaryDb + " WITH DBPROPERTIES ( '"
+
+ SOURCE_OF_REPLICATION + "' = '1,2,3')");
+ WarehouseInstance.Tuple inc2Tuple = primary.run("use " + extraPrimaryDb)
+ .run("create table tbl (fld int)")
+ .run("use " + primaryDbName)
+ .dump(primaryDbName, incTuple.lastReplicationId);
+
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 296244)
Time Spent: 40m (was: 0.5h)
> Return the last event id dumped as repl status to avoid notification event
> missing error.
> -----------------------------------------------------------------------------------------
>
> Key: HIVE-22068
> URL: https://issues.apache.org/jira/browse/HIVE-22068
> Project: Hive
> Issue Type: Improvement
> Reporter: Ashutosh Bapat
> Assignee: Ashutosh Bapat
> Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22068.01.patch, HIVE-22068.02.patch,
> HIVE-22068.03.patch, HIVE-22068.04.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In repl load, update the status of target database to the last event dumped
> so that repl status returns that and next incremental can specify it as the
> event from which to start the dump. WIthout that repl status might return and
> old event which might cause, older events to be dumped again and/or a
> notification event missing error if the older events are cleaned by the
> cleaner.
> While at it
> * Add more logging to DB notification listener cleaner thread
> ** The time when it considered cleaning, the interval and time before which
> events were cleared, the min and max id at that time
> ** how many events were cleared
> ** min and max id after the cleaning.
> * In REPL::START document the starting event, end event if specified and the
> maximum number of events, if specified.
> *
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
