[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268234&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268234
]
ASF GitHub Bot logged work on HIVE-21886:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:12
Start Date: 27/Jun/19 05:12
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298008351
##########
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##########
@@ -92,21 +93,46 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
- String oldName = before.getTableName();
- String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+ return (withinContext.oldReplScope != null)
+ &&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+ && (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
+
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+ // For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+ // policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+ // policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+ // then anyways the table should be ignored. The event should be ignored
if the table is already in the list of
+ // tables to be bootstrapped.
+ if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+ } else if
((withinContext.getTablesForBootstrap().contains(tblName.toLowerCase()))
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 268234)
Time Spent: 10h 10m (was: 10h)
> REPL - With table list - Handle rename events during replace policy
> -------------------------------------------------------------------
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
> Reporter: mahesh kumar behera
> Assignee: mahesh kumar behera
> Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> #### New name not matching new policy
> ***** Ignore the event, no need to do anything.
> #### New name matching new policy
> ***** The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> ***** All the future events will be ignored as part of check added by
> replace policy handling.
> ***** All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> #### New name not matching new policy
> ***** As the new name is not matching the new policy, the table need not be
> replicated.
> ***** As the old name is not matching the new policy, the rename events will
> be ignored.
> ***** So nothing to be done for this scenario.
> #### New name matching new policy
> ***** As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> ***** Add the table to the list of tables to be bootstrapped.
> ***** Ignore all the events with new name.
> ***** If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> ***** In case of rename event (double rename)
> ****** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ****** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
> checks based on old table, the table should be bootstrapped and event should
> be ignored. But rename handler should decide based on new name.The old table
> name will not be returned by get-all-table, so replace handler will not d
> anything for the old table.
> ### New name not matching old policy
> #### New name not matching new policy
> ***** As the old table is not there at target and new name is not matching
> new policy. Ignore the event.
> ***** No need to add the table to the list of tables to be bootstrapped.
> ***** All the subsequent events will be ignored as the new name is not
> matching the new policy.
> #### New name matching new policy
> ***** As the new name is not matching old policy but matching new policy,
> the table will be bootstrapped by replace policy handler. So rename event
> need not add this table to list of table to be bootstrapped.
> ***** All the future events will be ignored by replace policy handler.
> ***** For rename event (double rename)
> ****** If there is a rename, the table (with intermittent new name) will not
> be present and thus replace handler will not bootstrap the table.
> ****** So if the new name (the latest one) is matching the new policy, then
> add it to the list of table to be bootstrapped.
> ****** And If the new name (the latest one) is not matching the new policy,
> then just ignore the event as the intermittent new name would not have added
> to the list of table to be bootstrapped.
> ### New name matching old policy
> #### New name not matching new policy
> ***** Dump the event. The table will be dropped by repl load at the target.
> #### New name matching new policy
> ***** Replace handler will not bootstrap this table as the new name is
> matching both policies.
> ***** As old name is not matching the old policy, the table will not be
> there at target. The rename event should add the new name to the list of
> table to be bootstrapped.
> ***** Subsequent events with new table name should be ignored.
> ***** Drop events should not be ignored as if the table is present during
> bootstrapped, then its a new table and thus should be dropped.
> ***** In case of rename event (double rename)
> ****** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ****** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> # Old name is matching old policy – The old table will be there at the
> target cluster. The table will not be returned by get-all-table. Repl load
> should delete the old table as it is not matching the new policy.
> ## Old name is not matching new policy
> ### New name not matching old policy
> #### New name not matching new policy
> ***** Nothing to be done. Ignore the event.
> #### New name matching new policy
> ***** As the new name is not matching old policy but matching new policy,
> the table will be bootstrapped by replace policy handler. So rename event
> need not add this table to list of table to be bootstrapped.
> ***** All the future events will be ignored by replace policy handler.
> ***** For rename event (double rename)
> ****** If there is a rename, the table (with intermittent new name) will not
> be present and thus replace handler will not bootstrap the table.
> ****** So if the new name (the latest one) is matching the new policy, then
> add it to the list of table to be bootstrapped.
> ****** And If the new name (the latest one) is not matching the new policy,
> then just ignore the event as the intermittent new name would not have added
> to the list of table to be bootstrapped.
> ### New name matching old policy
> #### New name not matching new policy
> ***** Table with new name will be dropped by repl load
> ***** Along with other event, ignore the rename event also.
> #### New name matching new policy
> ***** As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> ***** Add the table to the list of tables to be bootstrapped.
> ***** Ignore all the events with new name.
> ***** If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> ***** In case of rename event (double rename)
> ****** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ****** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy
> ### New name not matching old policy
> #### New name not matching new policy
> ***** The old table needs to be dropped at target. Ignore this event, as the
> old table is not matching the new policy, it will be dropped by repl load.
> #### New name matching new policy
> ***** Allow the event to dump and replayed at target.
> ***** Allow further events to be handled as usual.
> ***** In case of rename event (double rename)
> ****** If the latest new name is matching the new policy, then keep it as is
> it. Let rename event replayed at target.
> ****** If the latest new name is not matching the new policy, then change
> the rename event to drop event.
> ### New name matching old policy
> #### New name not matching new policy
> ##### Nothing to be done.
> #### New name matching new policy
> ##### Add the table name to the list of tables to be bootstrapped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
