[ https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872984#comment-16872984 ]
Hive QA commented on HIVE-21886: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12972889/HIVE-21886.01.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 16284 tests executed *Failed tests:* {noformat} TestReplAcrossInstancesWithJsonMessageFormat - did not produce a TEST-*.xml file (likely timed out) (batchId=255) TestReplicationScenariosExternalTables - did not produce a TEST-*.xml file (likely timed out) (batchId=255) TestTableLevelReplicationScenarios - did not produce a TEST-*.xml file (likely timed out) (batchId=255) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17743/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17743/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17743/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12972889 - PreCommit-HIVE-Build > REPL - With table list - Handle rename events during replace policy > ------------------------------------------------------------------- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl > Reporter: mahesh kumar behera > Assignee: mahesh kumar behera > Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > #### New name not matching new policy > ***** Ignore the event, no need to do anything. > #### New name matching new policy > ***** The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > ***** All the future events will be ignored as part of check added by > replace policy handling. > ***** All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > #### New name not matching new policy > ***** As the new name is not matching the new policy, the table need not be > replicated. > ***** As the old name is not matching the new policy, the rename events will > be ignored. > ***** So nothing to be done for this scenario. > #### New name matching new policy > ***** As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > ***** Add the table to the list of tables to be bootstrapped. > ***** Ignore all the events with new name. > ***** If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > ***** In case of rename event (double rename) > ****** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ****** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on old table, the table should be bootstrapped and event should > be ignored. But rename handler should decide based on new name.The old table > name will not be returned by get-all-table, so replace handler will not d > anything for the old table. > ### New name not matching old policy > #### New name not matching new policy > ***** As the old table is not there at target and new name is not matching > new policy. Ignore the event. > ***** No need to add the table to the list of tables to be bootstrapped. > ***** All the subsequent events will be ignored as the new name is not > matching the new policy. > #### New name matching new policy > ***** As the new name is not matching old policy but matching new policy, > the table will be bootstrapped by replace policy handler. So rename event > need not add this table to list of table to be bootstrapped. > ***** All the future events will be ignored by replace policy handler. > ***** For rename event (double rename) > ****** If there is a rename, the table (with intermittent new name) will not > be present and thus replace handler will not bootstrap the table. > ****** So if the new name (the latest one) is matching the new policy, then > add it to the list of table to be bootstrapped. > ****** And If the new name (the latest one) is not matching the new policy, > then just ignore the event as the intermittent new name would not have added > to the list of table to be bootstrapped. > ### New name matching old policy > #### New name not matching new policy > ***** Dump the event. The table will be dropped by repl load at the target. > #### New name matching new policy > ***** Replace handler will not bootstrap this table as the new name is > matching both policies. > ***** As old name is not matching the old policy, the table will not be > there at target. The rename event should add the new name to the list of > table to be bootstrapped. > ***** Subsequent events with new table name should be ignored. > ***** Drop events should not be ignored as if the table is present during > bootstrapped, then its a new table and thus should be dropped. > ***** In case of rename event (double rename) > ****** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ****** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > # Old name is matching old policy – The old table will be there at the > target cluster. The table will not be returned by get-all-table. Repl load > should delete the old table as it is not matching the new policy. > ## Old name is not matching new policy > ### New name not matching old policy > #### New name not matching new policy > ***** Nothing to be done. Ignore the event. > #### New name matching new policy > ***** As the new name is not matching old policy but matching new policy, > the table will be bootstrapped by replace policy handler. So rename event > need not add this table to list of table to be bootstrapped. > ***** All the future events will be ignored by replace policy handler. > ***** For rename event (double rename) > ****** If there is a rename, the table (with intermittent new name) will not > be present and thus replace handler will not bootstrap the table. > ****** So if the new name (the latest one) is matching the new policy, then > add it to the list of table to be bootstrapped. > ****** And If the new name (the latest one) is not matching the new policy, > then just ignore the event as the intermittent new name would not have added > to the list of table to be bootstrapped. > ### New name matching old policy > #### New name not matching new policy > ***** Table with new name will be dropped by repl load > ***** Along with other event, ignore the rename event also. > #### New name matching new policy > ***** As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > ***** Add the table to the list of tables to be bootstrapped. > ***** Ignore all the events with new name. > ***** If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > ***** In case of rename event (double rename) > ****** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ****** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy > ### New name not matching old policy > #### New name not matching new policy > ***** The old table needs to be dropped at target. Ignore this event, as the > old table is not matching the new policy, it will be dropped by repl load. > #### New name matching new policy > ***** Allow the event to dump and replayed at target. > ***** Allow further events to be handled as usual. > ***** In case of rename event (double rename) > ****** If the latest new name is matching the new policy, then keep it as is > it. Let rename event replayed at target. > ****** If the latest new name is not matching the new policy, then change > the rename event to drop event. > ### New name matching old policy > #### New name not matching new policy > ##### Nothing to be done. > #### New name matching new policy > ##### Add the table name to the list of tables to be bootstrapped. -- This message was sent by Atlassian JIRA (v7.6.3#76005)