[ https://issues.apache.org/jira/browse/HIVE-21764?focusedWorklogId=264649&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264649 ]
ASF GitHub Bot logged work on HIVE-21764: ----------------------------------------- Author: ASF GitHub Bot Created on: 21/Jun/19 13:59 Start Date: 21/Jun/19 13:59 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #679: HIVE-21764 : REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included. URL: https://github.com/apache/hive/pull/679#discussion_r296247443 ########## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java ########## @@ -82,14 +92,71 @@ private Scenario scenarioType(org.apache.hadoop.hive.metastore.api.Table before, } } + // return true, if event needs to be dumped, else return false. + private boolean handleForTableLevelReplication(Context withinContext) { + String oldName = before.getTableName(); + String newName = after.getTableName(); + + if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) { + // If the table is renamed after being added to the list of tables to be bootstrapped, then remove it from the + // list of tables to be bootstrapped. + boolean oldTableIsPresent = withinContext.removeFromListOfTablesForBootstrap(before.getTableName()); + + // If old table satisfies the filter, but the new table does not, then the old table should be dropped. + // This should be done, only if the old table is not in the list of tables to be bootstrapped which is a multi + // rename case. In case of multi rename, only the first rename should do the drop. + if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, newName)) { + if (oldTableIsPresent) { + // If the old table was present in the list of tables to be bootstrapped, then just ignore the event. + return false; + } else { + scenario = Scenario.DROP; + LOG.info("Table " + oldName + " will be dropped as the table is renamed to " + newName); + return true; + } + } + + // If the old table was in the list of tables to be bootstrapped which is a multi rename case, the old table + // is removed from the list of tables to be bootstrapped and new one is added. + if (oldTableIsPresent) { + withinContext.addToListOfTablesForBootstrap(newName); + return false; + } + + // If both old and new table satisfies the filter and old table is present at target, then dump the rename event. + LOG.info("both old and new table satisfies the filter"); + return true; + } else { + // if the old table does not satisfies the filter, but the new one satisfies, then the new table should be + // added to the list of tables to be bootstrapped and don't dump the event. + if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, newName)) { + LOG.info("Table " + newName + " is added for bootstrap " + " during rename from " + oldName); + withinContext.addToListOfTablesForBootstrap(newName); + return false; + } + + // if both old and new table does not satisfies the filter, then don't dump the event. + LOG.info("both old and new table not satisfies the filter"); + return false; + } + } + @Override public void handle(Context withinContext) throws Exception { LOG.info("Processing#{} ALTER_TABLE message : {}", fromEventId(), eventMessageAsJSON); Table qlMdTableBefore = new Table(before); + Set<String> bootstrapTableList; + if (Scenario.RENAME == scenario) { + // Handling for table level replication is done in handleForTableLevelReplication method. + bootstrapTableList = null; Review comment: replScope is for replace policy (old policy) ..its n not handled in this patch ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 264649) Time Spent: 4h 40m (was: 4.5h) > REPL DUMP should detect and bootstrap any rename table events where old table > was excluded but renamed table is included. > ------------------------------------------------------------------------------------------------------------------------- > > Key: HIVE-21764 > URL: https://issues.apache.org/jira/browse/HIVE-21764 > Project: Hive > Issue Type: Sub-task > Components: repl > Reporter: Sankar Hariappan > Assignee: mahesh kumar behera > Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21764.01.patch, HIVE-21764.02.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > > REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular > expression + inclusion/exclusion list. So, in case of rename table event, the > event will be ignored if old table doesn't match the pattern but the new > table should be bootstrapped. REPL DUMP should have a mechanism to detect > such tables and automatically bootstrap with incremental replication.Also, if > renamed table is excluded from replication policy, then need to drop the old > table at target as well. > There are 4 scenarios that needs to be handled. > # Both new name and old name satisfies the table name pattern filter. > ## No need to do anything. The incremental event for rename should take care > of the replication. > # Both the names does not satisfy the table name pattern filter. > ## Both the names are not in the scope of the policy and thus nothing needs > to be done. > # New name satisfies the pattern but the old name does not. > ## The table will not be present at the target. > ## Rename event handler for dump should detect this case and add the new > table name to the list of table for bootstrap. > ## All the events related to the table (new name) should be ignored. > ## If there is a drop event for the table (with new name), then remove the > table from the list of tables to be bootstrapped. > ## In case of rename (double rename) > ### If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ### If the new name does not satisfies then just removed the table name from > the list of tables to be bootstrapped. > # New name does not satisfies the pattern but the old name satisfies. > ## Change the rename event to a drop event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)