[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=446187&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446187 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 16/Jun/20 00:24 Start Date: 16/Jun/20 00:24 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #688: URL: https://github.com/apache/hive/pull/688 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446187) Time Spent: 11.5h (was: 11h 20m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, > HIVE-21886.03.patch, HIVE-21886.04.patch, HIVE-21886.04.patch, > HIVE-21886.05.patch > > Time Spent: 11.5h > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. If the old table does not match the old policy drop it, even if the > table is not present at target. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > 2. If the new policy does not match the table name, then ignore the > event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on old table, the table should be bootstrapped and event should > be ignored. But rename handler should decide based on new name.The old table > name will not be returned by get-all-table, so replace handler will not d > anything for the old table. > ### New name not matching old policy > New name not matching new policy > * As
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=442711&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-442711 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 09/Jun/20 15:50 Start Date: 09/Jun/20 15:50 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #688: URL: https://github.com/apache/hive/pull/688#issuecomment-640955799 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 442711) Time Spent: 11h 20m (was: 11h 10m) > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch, > HIVE-21886.03.patch, HIVE-21886.04.patch, HIVE-21886.04.patch, > HIVE-21886.05.patch > > Time Spent: 11h 20m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. If the old table does not match the old policy drop it, even if the > table is not present at target. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > 2. If the new policy does not match the table name, then ignore the > event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on old table, the table should be bootstrapped and event should > be
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268272&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268272
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 06:12
Start Date: 27/Jun/19 06:12
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298020420
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -118,25 +147,25 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
// If the old table was in the list of tables to be bootstrapped which
is a multi rename case, the old table
// is removed from the list of tables to be bootstrapped and new one is
added.
- if (oldTableIsPresent) {
+ if (oldTableInBootstrapList) {
withinContext.addToListOfTablesForBootstrap(newName);
return false;
}
- // If both old and new table satisfies the filter and old table is
present at target, then dump the rename event.
- LOG.info("both old and new table satisfies the filter");
+ // If both old and new table satisfies the policy and old table is
present at target, then dump the rename event.
+ LOG.info("both old and new table satisfies the policy");
return true;
} else {
- // if the old table does not satisfies the filter, but the new one
satisfies, then the new table should be
+ // if the old table does not satisfies the policy, but the new one
satisfies, then the new table should be
// added to the list of tables to be bootstrapped and don't dump the
event.
if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
LOG.info("Table " + newName + " is added for bootstrap " + " during
rename from " + oldName);
withinContext.addToListOfTablesForBootstrap(newName);
return false;
}
- // if both old and new table does not satisfies the filter, then don't
dump the event.
- LOG.info("both old and new table not satisfies the filter");
+ // if both old and new table does not satisfies the policy, then don't
dump the event.
+ LOG.info("both old and new table not satisfies the policy");
Review comment:
Shall log table names also in the log message.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268272)
Time Spent: 11h 10m (was: 11h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. If the old table does not match the old policy drop it, even if the
> table is not present at target.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
> 2. If the new policy does not match the table name, then ignore the
> event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> ##
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268262&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268262
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:52
Start Date: 27/Jun/19 05:52
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298015828
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,21 +93,49 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleRenameForReplacePolicy(Context withinContext, String
oldName, String newName) {
+// If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
Review comment:
with the addition if withinContext.oldReplScope != null ..before calling
..the assert is not required
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268262)
Time Spent: 11h (was: 10h 50m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch
>
> Time Spent: 11h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. If the old table does not match the old policy drop it, even if the
> table is not present at target.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
> 2. If the new policy does not match the table name, then ignore the
> event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a dr
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268259&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268259
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:49
Start Date: 27/Jun/19 05:49
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298015157
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -168,10 +205,19 @@ public void handle(Context withinContext) throws
Exception {
}
}
-// If the tables are filtered based on name, then needs to handle the
rename scenarios.
-if (!withinContext.replScope.includeAllTables()) {
- if (!handleForTableLevelReplication(withinContext)) {
-LOG.info("Alter event for table " + before.getTableName() + " is
skipped from dumping");
+// Handle renames for table level replication.
+if ((Scenario.RENAME == scenario)
+&& (!withinContext.replScope.includeAllTables() ||
withinContext.oldReplScope != null)) {
+ String oldName = before.getTableName();
+ String newName = after.getTableName();
+ boolean needDump;
+ if (withinContext.oldReplScope == null) {
+needDump = handleRenameForTableLevelReplication(withinContext,
oldName, newName);
+ } else {
+needDump = handleRenameForReplacePolicy(withinContext, oldName,
newName);
+ }
+ if (!needDump) {
+LOG.info("Rename event for table " + oldName + " is skipped from
dumping");
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268259)
Time Spent: 10h 50m (was: 10h 40m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch
>
> Time Spent: 10h 50m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. If the old table does not match the old policy drop it, even if the
> table is not present at target.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
> 2. If the new policy does not match the table name, then ignore the
> event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename eve
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268258&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268258
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:49
Start Date: 27/Jun/19 05:49
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298015144
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -168,10 +205,19 @@ public void handle(Context withinContext) throws
Exception {
}
}
-// If the tables are filtered based on name, then needs to handle the
rename scenarios.
-if (!withinContext.replScope.includeAllTables()) {
- if (!handleForTableLevelReplication(withinContext)) {
-LOG.info("Alter event for table " + before.getTableName() + " is
skipped from dumping");
+// Handle renames for table level replication.
+if ((Scenario.RENAME == scenario)
+&& (!withinContext.replScope.includeAllTables() ||
withinContext.oldReplScope != null)) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268258)
Time Spent: 10h 40m (was: 10.5h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch
>
> Time Spent: 10h 40m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. If the old table does not match the old policy drop it, even if the
> table is not present at target.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
> 2. If the new policy does not match the table name, then ignore the
> event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * I
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268251&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268251
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:39
Start Date: 27/Jun/19 05:39
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298010710
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -168,10 +205,19 @@ public void handle(Context withinContext) throws
Exception {
}
}
-// If the tables are filtered based on name, then needs to handle the
rename scenarios.
-if (!withinContext.replScope.includeAllTables()) {
- if (!handleForTableLevelReplication(withinContext)) {
-LOG.info("Alter event for table " + before.getTableName() + " is
skipped from dumping");
+// Handle renames for table level replication.
+if ((Scenario.RENAME == scenario)
+&& (!withinContext.replScope.includeAllTables() ||
withinContext.oldReplScope != null)) {
+ String oldName = before.getTableName();
+ String newName = after.getTableName();
+ boolean needDump;
+ if (withinContext.oldReplScope == null) {
+needDump = handleRenameForTableLevelReplication(withinContext,
oldName, newName);
+ } else {
+needDump = handleRenameForReplacePolicy(withinContext, oldName,
newName);
+ }
+ if (!needDump) {
+LOG.info("Rename event for table " + oldName + " is skipped from
dumping");
Review comment:
Shall change the log to include "newName" as well. ... oldName to newName ...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268251)
Time Spent: 10.5h (was: 10h 20m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch
>
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. If the old table does not match the old policy drop it, even if the
> table is not present at target.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
> 2. If the new policy does not match the table name, then ignore the
> event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> **
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268252&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268252
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:39
Start Date: 27/Jun/19 05:39
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298010206
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -168,10 +205,19 @@ public void handle(Context withinContext) throws
Exception {
}
}
-// If the tables are filtered based on name, then needs to handle the
rename scenarios.
-if (!withinContext.replScope.includeAllTables()) {
- if (!handleForTableLevelReplication(withinContext)) {
-LOG.info("Alter event for table " + before.getTableName() + " is
skipped from dumping");
+// Handle renames for table level replication.
+if ((Scenario.RENAME == scenario)
+&& (!withinContext.replScope.includeAllTables() ||
withinContext.oldReplScope != null)) {
Review comment:
These checks can be moved inside as follows. Will be less complex.
if (Scenario.RENAME == scenario) {
...
boolean needDump = true;
if (withinContext.oldReplScope != null) {
needDump = handleRenameForReplacePolicy();
} else if (!withinContext.replScope.includeAllTables()) {
needDump = handleRenameForTableLevelReplication();
}
if (!needDump) {
return;
}
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268252)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch
>
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. If the old table does not match the old policy drop it, even if the
> table is not present at target.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
> 2. If the new policy does not match the table name, then ignore the
> event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> N
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268250&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268250
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:39
Start Date: 27/Jun/19 05:39
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298012981
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,21 +93,49 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleRenameForReplacePolicy(Context withinContext, String
oldName, String newName) {
+// If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
Review comment:
Just add an assert that withinContext.oldReplScope != null.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268250)
Time Spent: 10h 20m (was: 10h 10m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch,
> HIVE-21886.03.patch
>
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. If the old table does not match the old policy drop it, even if the
> table is not present at target.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
> 2. If the new policy does not match the table name, then ignore the
> event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name)
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268234&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268234
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:12
Start Date: 27/Jun/19 05:12
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298008351
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,21 +93,46 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
+
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored. The event should be ignored
if the table is already in the list of
+// tables to be bootstrapped.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if
((withinContext.getTablesForBootstrap().contains(tblName.toLowerCase()))
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268234)
Time Spent: 10h 10m (was: 10h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check ad
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268233&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268233
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:12
Start Date: 27/Jun/19 05:12
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298008286
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -147,16 +197,19 @@ public void handle(Context withinContext) throws
Exception {
Table qlMdTableBefore = new Table(before);
Set bootstrapTableList;
+ReplScope oldReplScope;
if (Scenario.RENAME == scenario) {
// Handling for table level replication is done in
handleForTableLevelReplication method.
Review comment:
code is removed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268233)
Time Spent: 10h (was: 9h 50m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 10h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
> checks based on old table, the table should be bootstrapped and
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268232&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268232
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:11
Start Date: 27/Jun/19 05:11
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298008125
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -118,13 +144,30 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
// If the old table was in the list of tables to be bootstrapped which
is a multi rename case, the old table
// is removed from the list of tables to be bootstrapped and new one is
added.
- if (oldTableIsPresent) {
+ if (oldTableInBootstrapList) {
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+ }
+
+ // All the subsequent events on this table newName are going to be
skipped as the table is going to be
+ // bootstrapped by replace handler, so the rename is also skipped.
+ if (isSetForBootstrapByReplaceHandler(withinContext, newName)) {
+// This addition is not actually required. But is added just to be in
safer side.
withinContext.addToListOfTablesForBootstrap(newName);
+
+// If the old table satisfies the new policy and is not in the list of
tables to be bootstrapped
+// (as per previous check based on oldTableIsPresent), then drop it.
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
oldName)) {
+ scenario = Scenario.DROP;
+ LOG.info("Table " + oldName + " will be dropped as the table " +
newName + " will be bootstrapped.");
+ return true;
+}
+LOG.info("Table " + newName + " is set to be bootstrapped by replace
policy handler.");
return false;
}
// If both old and new table satisfies the filter and old table is
present at target, then dump the rename event.
- LOG.info("both old and new table satisfies the filter");
+ LOG.info("Both old " + oldName + " and new table " + newName + "
satisfies the filter");
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268232)
Time Spent: 9h 50m (was: 9h 40m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the futu
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268231&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268231
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:11
Start Date: 27/Jun/19 05:11
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298008102
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -135,8 +178,15 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
}
+ // In case of replace policy, even if the old table matches the new
policy, none of the events including create
+ // table will be replayed as the old table is set of bootstrap by
replace handler. So rename event can be skipped.
Review comment:
code is removed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268231)
Time Spent: 9h 40m (was: 9.5h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
> checks based on ol
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268229&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268229
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:10
Start Date: 27/Jun/19 05:10
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298008044
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
+&&
(!ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))) {
+ LOG.debug("Table " + tblName + " is set for bootstrap");
+ return false;
+} else {
+ // Table satisfies both old (if its there) and current policy, dump the
alter event.
+ return true;
+}
+ }
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplicationForRename(Context
withinContext, String oldName, String newName) {
Review comment:
new Jira is created
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268229)
Time Spent: 9h 20m (was: 9h 10m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268230&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268230
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:10
Start Date: 27/Jun/19 05:10
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298008064
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -196,27 +196,27 @@ private void verifyBootstrapDirInIncrementalDump(String
dumpLocation, String[] b
@Test
public void testBasicBootstrapWithIncludeList() throws Throwable {
-String[] originalNonAcidTables = new String[] {"t1", "t2" };
-String[] originalFullAcidTables = new String[] {"t3", "t4" };
-String[] originalMMAcidTables = new String[] {"t5" };
+String[] originalNonAcidTables = new String[] {"t1", "t2"};
+String[] originalFullAcidTables = new String[] {"t3", "t4"};
+String[] originalMMAcidTables = new String[] {"t5"};
createTables(originalNonAcidTables, CreateTableType.NON_ACID);
createTables(originalFullAcidTables, CreateTableType.FULL_ACID);
createTables(originalMMAcidTables, CreateTableType.MM_ACID);
// Replicate and verify if only 2 tables are replicated to target.
String replPolicy = primaryDbName + ".['t1', 't4', 't5']";
-String[] replicatedTables = new String[] {"t1", "t4", "t5" };
+String[] replicatedTables = new String[] {"t1", "t4", "t5"};
replicateAndVerify(replPolicy, null, null, null, replicatedTables);
}
@Test
public void testBasicBootstrapWithIncludeAndExcludeList() throws Throwable {
-String[] originalTables = new String[] {"t1", "t11", "t2", "t3", "t100" };
+String[] originalTables = new String[] {"t1", "t11", "t2", "t3", "t100"};
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268230)
Time Spent: 9.5h (was: 9h 20m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
>
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268226&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268226
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:08
Start Date: 27/Jun/19 05:08
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298007695
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -123,8 +148,21 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
}
+ // All the subsequent events on this table newName are going to be
skipped, so the rename is also skipped.
Review comment:
code removed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268226)
Time Spent: 9h 10m (was: 9h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
> checks based on old table, the table should be bootstrapped and event should
> be ignored. But rename handler should decide based on new name.The old
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268225&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268225
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:08
Start Date: 27/Jun/19 05:08
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298007642
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
Review comment:
code removed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268225)
Time Spent: 9h (was: 8h 50m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 9h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268224&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268224
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:07
Start Date: 27/Jun/19 05:07
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298007570
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
Review comment:
code removed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268224)
Time Spent: 8h 50m (was: 8h 40m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 8h 50m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching n
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=268222&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-268222
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 27/Jun/19 05:06
Start Date: 27/Jun/19 05:06
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r298007457
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -120,18 +121,18 @@ private boolean handleForTableLevelReplication(Context
withinContext, String tbl
}
// Return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplicationForRename(Context
withinContext, String oldName, String newName) {
+ private boolean renameHandlerForTableLevelReplication(Context withinContext,
String oldName, String newName) {
// If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
// list of tables to be bootstrapped.
-boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(oldName);
+boolean oldTableInBootstrapList =
withinContext.removeFromListOfTablesForBootstrap(oldName);
ReplScope oldPolicy = withinContext.oldReplScope == null ?
withinContext.replScope : withinContext.oldReplScope;
if (ReplUtils.tableIncludedInReplScope(oldPolicy, oldName)) {
// If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
// This should be done, only if the old table is not in the list of
tables to be bootstrapped which is a multi
// rename case. In case of multi rename, only the first rename should do
the drop.
if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
-if (oldTableIsPresent) {
+if (oldTableInBootstrapList) {
// If the old table was present in the list of tables to be
bootstrapped, then just ignore the event.
return false;
} else {
Review comment:
there is no issue if C doesn't match new policy. As drop is idempotent. But
the real issue is when C match new policy. The rename event will get dumped and
that will cause data loss as B is not present at target. I have modified the
code to avoid all these complexity. Now if old table does not satisfy the old
policy then dump drop event(although that is also not required). If new policy
matches new name then add it to the bootstrap. So now for rename with replace
policy its always bootstrap. Thats will make the code simple and as replace and
rename are not very frequently executed operations, it should be fine.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 268222)
Time Spent: 8h 40m (was: 8.5h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cl
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267667&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267667
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 14:09
Start Date: 26/Jun/19 14:09
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297686895
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,21 +93,46 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
+
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored. The event should be ignored
if the table is already in the list of
+// tables to be bootstrapped.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if
((withinContext.getTablesForBootstrap().contains(tblName.toLowerCase()))
Review comment:
I think, the else if flow is dead code as we skip this event from caller
itself. Utils.shouldReplicate method returns false for this case. Pls check.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267667)
Time Spent: 8.5h (was: 8h 20m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootst
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267660&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267660
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 14:03
Start Date: 26/Jun/19 14:03
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297683504
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -147,16 +197,19 @@ public void handle(Context withinContext) throws
Exception {
Table qlMdTableBefore = new Table(before);
Set bootstrapTableList;
+ReplScope oldReplScope;
if (Scenario.RENAME == scenario) {
// Handling for table level replication is done in
handleForTableLevelReplication method.
Review comment:
handleForTableLevelReplication doesn't handle rename flow. Incorrect comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267660)
Time Spent: 8h 20m (was: 8h 10m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267656&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267656
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 14:01
Start Date: 26/Jun/19 14:01
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297682711
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -118,13 +144,30 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
// If the old table was in the list of tables to be bootstrapped which
is a multi rename case, the old table
// is removed from the list of tables to be bootstrapped and new one is
added.
- if (oldTableIsPresent) {
+ if (oldTableInBootstrapList) {
+withinContext.addToListOfTablesForBootstrap(newName);
+return false;
+ }
+
+ // All the subsequent events on this table newName are going to be
skipped as the table is going to be
+ // bootstrapped by replace handler, so the rename is also skipped.
+ if (isSetForBootstrapByReplaceHandler(withinContext, newName)) {
+// This addition is not actually required. But is added just to be in
safer side.
withinContext.addToListOfTablesForBootstrap(newName);
+
+// If the old table satisfies the new policy and is not in the list of
tables to be bootstrapped
+// (as per previous check based on oldTableIsPresent), then drop it.
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
oldName)) {
+ scenario = Scenario.DROP;
+ LOG.info("Table " + oldName + " will be dropped as the table " +
newName + " will be bootstrapped.");
+ return true;
+}
+LOG.info("Table " + newName + " is set to be bootstrapped by replace
policy handler.");
return false;
}
// If both old and new table satisfies the filter and old table is
present at target, then dump the rename event.
- LOG.info("both old and new table satisfies the filter");
+ LOG.info("Both old " + oldName + " and new table " + newName + "
satisfies the filter");
Review comment:
filter is not the right term. We can say, replication scope/policy. Pls
check other places as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267656)
Time Spent: 8h 10m (was: 8h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap th
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267654&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267654
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:59
Start Date: 26/Jun/19 13:59
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297681357
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -135,8 +178,15 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
}
+ // In case of replace policy, even if the old table matches the new
policy, none of the events including create
+ // table will be replayed as the old table is set of bootstrap by
replace handler. So rename event can be skipped.
Review comment:
There is nothing called replace handler. So, shall change the comment that
"Old table is set for bootstrap due to replace policy."
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267654)
Time Spent: 8h (was: 7h 50m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 8h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be boots
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267648&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267648
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:54
Start Date: 26/Jun/19 13:54
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297678333
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
Review comment:
As discussed, let's keep the checks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267648)
Time Spent: 7.5h (was: 7h 20m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching n
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267650&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267650
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:54
Start Date: 26/Jun/19 13:54
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297678731
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
Review comment:
As discussed, let's keep the checks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267650)
Time Spent: 7h 50m (was: 7h 40m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 7h 50m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
> checks based on old table, the table should be bootstrapped and event should
> be ignored. But rename handler should decide based on new name.The old table
> name will not be returned by get-all-table, so replace handler will not d
> anything for th
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267649&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267649
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:54
Start Date: 26/Jun/19 13:54
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297678557
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
}
+ // In case of replace policy, even if the old table matches the new
policy, none of the events including create
+ // table will be replayed as the old table is set of bootstrap by
replace handler. So rename event can be skipped.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
oldName)) {
Review comment:
Ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267649)
Time Spent: 7h 40m (was: 7.5h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267647&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267647
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:52
Start Date: 26/Jun/19 13:52
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297677615
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
Review comment:
I disagree. If renameHandler method is called within this method, then this
name is fine. As per your argument, it handles only alter and truncate but
gives the feel that it handles rename as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267647)
Time Spent: 7h 20m (was: 7h 10m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
>
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267645&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267645
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:48
Start Date: 26/Jun/19 13:48
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297675249
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
}
+ // In case of replace policy, even if the old table matches the new
policy, none of the events including create
+ // table will be replayed as the old table is set of bootstrap by
replace handler. So rename event can be skipped.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
oldName)) {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267645)
Time Spent: 7h (was: 6h 50m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 7h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new polic
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267646&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267646
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:48
Start Date: 26/Jun/19 13:48
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297668771
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -120,18 +121,18 @@ private boolean handleForTableLevelReplication(Context
withinContext, String tbl
}
// Return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplicationForRename(Context
withinContext, String oldName, String newName) {
+ private boolean renameHandlerForTableLevelReplication(Context withinContext,
String oldName, String newName) {
// If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
// list of tables to be bootstrapped.
-boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(oldName);
+boolean oldTableInBootstrapList =
withinContext.removeFromListOfTablesForBootstrap(oldName);
ReplScope oldPolicy = withinContext.oldReplScope == null ?
withinContext.replScope : withinContext.oldReplScope;
if (ReplUtils.tableIncludedInReplScope(oldPolicy, oldName)) {
// If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
// This should be done, only if the old table is not in the list of
tables to be bootstrapped which is a multi
// rename case. In case of multi rename, only the first rename should do
the drop.
if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
-if (oldTableIsPresent) {
+if (oldTableInBootstrapList) {
// If the old table was present in the list of tables to be
bootstrapped, then just ignore the event.
return false;
} else {
Review comment:
A -> B where A doesn't matches old policy but B matches old but not new
policy. So, no-op and don't dump event as well. Now, B -> C where B matches old
and new policy and C doesn't match new policy. In this case, we add drop event
for B which is wrong as it is missing in bootstrap list. Pls check if this is
possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267646)
Time Spent: 7h 10m (was: 7h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by ge
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267633&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267633
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:36
Start Date: 26/Jun/19 13:36
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297668771
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -120,18 +121,18 @@ private boolean handleForTableLevelReplication(Context
withinContext, String tbl
}
// Return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplicationForRename(Context
withinContext, String oldName, String newName) {
+ private boolean renameHandlerForTableLevelReplication(Context withinContext,
String oldName, String newName) {
// If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
// list of tables to be bootstrapped.
-boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(oldName);
+boolean oldTableInBootstrapList =
withinContext.removeFromListOfTablesForBootstrap(oldName);
ReplScope oldPolicy = withinContext.oldReplScope == null ?
withinContext.replScope : withinContext.oldReplScope;
if (ReplUtils.tableIncludedInReplScope(oldPolicy, oldName)) {
// If old table satisfies the filter, but the new table does not, then
the old table should be dropped.
// This should be done, only if the old table is not in the list of
tables to be bootstrapped which is a multi
// rename case. In case of multi rename, only the first rename should do
the drop.
if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope,
newName)) {
-if (oldTableIsPresent) {
+if (oldTableInBootstrapList) {
// If the old table was present in the list of tables to be
bootstrapped, then just ignore the event.
return false;
} else {
Review comment:
A -> B where A doesn't matches old policy but B matches old but not new
policy. So, no-op and don't dump event as well. Now, B -> C where B matches old
policy and C doesn't match new policy. In this case, we add drop event for B
which is wrong as it is missing in bootstrap list. Pls check if this is
possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267633)
Time Spent: 6h 50m (was: 6h 40m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-al
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267632&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267632
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:34
Start Date: 26/Jun/19 13:34
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297668081
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
Review comment:
i think the name looks fine ..adding default does not make difference.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267632)
Time Spent: 6h 40m (was: 6.5h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267631&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267631
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:32
Start Date: 26/Jun/19 13:32
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297666847
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
}
+ // In case of replace policy, even if the old table matches the new
policy, none of the events including create
+ // table will be replayed as the old table is set of bootstrap by
replace handler. So rename event can be skipped.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
oldName)) {
Review comment:
but it wont says ..if the event is replayed and not ..and why its done so
..so i think its better to keep the checks ..it also keeps track of the
conditions covered ..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267631)
Time Spent: 6.5h (was: 6h 20m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be b
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267625&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267625
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:25
Start Date: 26/Jun/19 13:25
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297663093
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -196,27 +196,27 @@ private void verifyBootstrapDirInIncrementalDump(String
dumpLocation, String[] b
@Test
public void testBasicBootstrapWithIncludeList() throws Throwable {
-String[] originalNonAcidTables = new String[] {"t1", "t2" };
-String[] originalFullAcidTables = new String[] {"t3", "t4" };
-String[] originalMMAcidTables = new String[] {"t5" };
+String[] originalNonAcidTables = new String[] {"t1", "t2"};
+String[] originalFullAcidTables = new String[] {"t3", "t4"};
+String[] originalMMAcidTables = new String[] {"t5"};
createTables(originalNonAcidTables, CreateTableType.NON_ACID);
createTables(originalFullAcidTables, CreateTableType.FULL_ACID);
createTables(originalMMAcidTables, CreateTableType.MM_ACID);
// Replicate and verify if only 2 tables are replicated to target.
String replPolicy = primaryDbName + ".['t1', 't4', 't5']";
-String[] replicatedTables = new String[] {"t1", "t4", "t5" };
+String[] replicatedTables = new String[] {"t1", "t4", "t5"};
replicateAndVerify(replPolicy, null, null, null, replicatedTables);
}
@Test
public void testBasicBootstrapWithIncludeAndExcludeList() throws Throwable {
-String[] originalTables = new String[] {"t1", "t11", "t2", "t3", "t100" };
+String[] originalTables = new String[] {"t1", "t11", "t2", "t3", "t100"};
Review comment:
Checkstyle throws warning if no space before "}" as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267625)
Time Spent: 6h 20m (was: 6h 10m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267623&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267623
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:22
Start Date: 26/Jun/19 13:22
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297661738
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
+&&
(!ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))) {
+ LOG.debug("Table " + tblName + " is set for bootstrap");
+ return false;
+} else {
+ // Table satisfies both old (if its there) and current policy, dump the
alter event.
+ return true;
+}
+ }
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplicationForRename(Context
withinContext, String oldName, String newName) {
Review comment:
I think, TO/LIMIT is helpful mostly to control the replication cycle time.
Otherwise, single REPL LOAD would run continuously blocking other readers in
target. But, with table level replication, indirectly, they controls the amount
of events coming in per REPL DUMP, so we can disallow it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267623)
Time Spent: 6h 10m (was: 6h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new tabl
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267618&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267618
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:20
Start Date: 26/Jun/19 13:20
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297660432
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
Review comment:
Same as above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267618)
Time Spent: 6h (was: 5h 50m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 6h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
> checks based on old table, the table should be bootstrapped and event should
> be ignored. But rename handler should decide based on new name.The old table
> name will not be returned by get-all-table, so replace handler will not d
> anything for the old table.
> ### New name n
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267617&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267617
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:20
Start Date: 26/Jun/19 13:20
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297660387
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
}
+ // In case of replace policy, even if the old table matches the new
policy, none of the events including create
+ // table will be replayed as the old table is set of bootstrap by
replace handler. So rename event can be skipped.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
oldName)) {
Review comment:
I think, too many checks make the code not readable. Also, from debugging
perspective, we have necessary info from other logs such as event message and
ReplScope logs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267617)
Time Spent: 5h 50m (was: 5h 40m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267613&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267613
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:19
Start Date: 26/Jun/19 13:19
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297659783
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -123,8 +148,21 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
}
+ // All the subsequent events on this table newName are going to be
skipped, so the rename is also skipped.
Review comment:
But the comment doesn't explicitly say that. It gives the meaning like, all
the events on newName will be always skipped unconditionally.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267613)
Time Spent: 5h 40m (was: 5.5h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
> checks based on old table
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267611&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267611
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:17
Start Date: 26/Jun/19 13:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297658930
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
Review comment:
I think, it is redundant as the general rule is if oldName is not matching
the old policy, skip the event. I don't think, it is hard to debug without
these logs, as we dump the event messages and also both old and new ReplScope.
Having too many conditions is also makes the code not readable. Pls check if we
can keep it simple.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267611)
Time Spent: 5.5h (was: 5h 20m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old tabl
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267605&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267605
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:14
Start Date: 26/Jun/19 13:14
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297657448
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
Review comment:
But this name seems to be superset for all options including rename. May be
better to change it to defaultHandlerForTableLevelReplication which gives the
meaning that if nobody overrides the behaviour for any type, then it takes this
flow.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267605)
Time Spent: 5h 20m (was: 5h 10m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new pol
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267602&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267602
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:08
Start Date: 26/Jun/19 13:08
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297654641
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -786,4 +786,127 @@ public void testRenameTableScenariosExternalTable()
throws Throwable {
replicateAndVerify(replPolicy, null, lastReplId, dumpWithClause,
loadWithClause, bootstrapTables, replicatedTables);
}
+
+ @Test
+ public void testRenameTableScenariosWithReplaceExternalTable() throws
Throwable {
+List loadWithClause =
ReplicationTestUtils.externalTableBasePathWithClause(REPLICA_EXTERNAL_BASE,
replica);
+List dumpWithClause = Arrays.asList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='false'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname +
"'='false'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES.varname +
"'='false'",
+"'" + ReplUtils.REPL_DUMP_INCLUDE_ACID_TABLES + "'='false'"
+);
+String replPolicy = primaryDbName + ".['in[0-9]+'].['in100', 'in200',
'in1400']";
+String lastReplId = replicateAndVerify(replPolicy, null, null,
dumpWithClause,
+loadWithClause, new String[]{}, new String[]{});
+
+String[] originalFullAcidTables = new String[]{"in1", "in2", "out3",
"out4", "out5", "in100", "in200", "in300"};
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267602)
Time Spent: 5h (was: 4h 50m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
>
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267601&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267601
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:08
Start Date: 26/Jun/19 13:08
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297654471
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -786,4 +786,127 @@ public void testRenameTableScenariosExternalTable()
throws Throwable {
replicateAndVerify(replPolicy, null, lastReplId, dumpWithClause,
loadWithClause, bootstrapTables, replicatedTables);
}
+
+ @Test
+ public void testRenameTableScenariosWithReplaceExternalTable() throws
Throwable {
+List loadWithClause =
ReplicationTestUtils.externalTableBasePathWithClause(REPLICA_EXTERNAL_BASE,
replica);
+List dumpWithClause = Arrays.asList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='false'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname +
"'='false'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES.varname +
"'='false'",
+"'" + ReplUtils.REPL_DUMP_INCLUDE_ACID_TABLES + "'='false'"
+);
+String replPolicy = primaryDbName + ".['in[0-9]+'].['in100', 'in200',
'in1400']";
+String lastReplId = replicateAndVerify(replPolicy, null, null,
dumpWithClause,
+loadWithClause, new String[]{}, new String[]{});
+
+String[] originalFullAcidTables = new String[]{"in1", "in2", "out3",
"out4", "out5", "in100", "in200", "in300"};
+String[] originalExternalTables = new String[]{"in400", "out500"};
+String[] originalNonAcidTables = new String[]{"in1400", "out1500",
"out1600"};
+createTables(originalNonAcidTables, CreateTableType.NON_ACID);
+createTables(originalFullAcidTables, CreateTableType.FULL_ACID);
+createTables(originalExternalTables, CreateTableType.EXTERNAL);
+
+// Replicate and verify, no table should be there.
+String[] replicatedTables = new String[]{};
+String[] bootstrapTables = new String[]{};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId,
dumpWithClause,
+loadWithClause, bootstrapTables, replicatedTables);
+
+// Rename the tables to satisfy the condition also replace the policy.
+primary.run("use " + primaryDbName)
+.run("alter table out1600 rename to in1600")
+.run("alter table in1400 rename to out1400")
+.run("alter table in1 rename to out7")
+.run("alter table out3 rename to in8")
+.run("alter table out4 rename to in9")
+.run("drop table in9")
+.run("alter table out5 rename to in10")
+.run("alter table in10 rename to out11")
+.run("alter table out500 rename to in500")
+.run("alter table in400 rename to out400");
+
+String newPolicy = primaryDbName + ".['in[0-9]+', 'out1500'].['in2']";
+dumpWithClause = Arrays.asList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='true'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname +
"'='true'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES.varname +
"'='true'",
+"'" + ReplUtils.REPL_DUMP_INCLUDE_ACID_TABLES + "'='true'"
+);
+
+// in2 should be dropped.
+replicatedTables = new String[]{"in8", "in100", "in200", "in300", "in500",
"in1600", "out1500"};
+bootstrapTables = new String[]{"in8", "in100", "in200", "in300", "in500",
"in1600", "out1500"};
+replicateAndVerify(newPolicy, replPolicy, lastReplId, dumpWithClause,
Review comment:
I think, the bootstrap config overrides every other cases, that's why wanted
to skip that. Otherwise, this test case doesn't make any sense for this patch.
Anyways, if covered under managed, then also fine.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267601)
Time Spent: 4h 50m (was: 4h 40m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
>
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267604&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267604
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:11
Start Date: 26/Jun/19 13:11
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297656264
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
Review comment:
ReplaceHandler is something odd. Rename and Replace seems to give similar
meaning in this context. Also, we don't set any table for bootstrap. The term
"set" confuses with bootstrapTables list but we don't refer to that list here.
So, better to change in my opinion.
May be, isTableCandidateForBootstrapByReplacePolicy is also relevant. Think
about it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267604)
Time Spent: 5h 10m (was: 5h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> **
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267588&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267588
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297650846
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -123,8 +148,21 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
}
+ // All the subsequent events on this table newName are going to be
skipped, so the rename is also skipped.
+ if (isSetForBootstrapByReplaceHandler(withinContext, newName)) {
+// If the old table satisfies the new policy and is not in the list of
tables to be bootstrapped
+// (as per previous check based on oldTableIsPresent), then drop it.
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
oldName)) {
+ scenario = Scenario.DROP;
+ LOG.info("Table " + oldName + " will be dropped as the table " +
newName + " will be bootstrapped.");
+ return true;
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267588)
Time Spent: 3h 20m (was: 3h 10m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfie
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267582&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267582
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297641943
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -786,4 +786,127 @@ public void testRenameTableScenariosExternalTable()
throws Throwable {
replicateAndVerify(replPolicy, null, lastReplId, dumpWithClause,
loadWithClause, bootstrapTables, replicatedTables);
}
+
+ @Test
+ public void testRenameTableScenariosWithReplaceExternalTable() throws
Throwable {
+List loadWithClause =
ReplicationTestUtils.externalTableBasePathWithClause(REPLICA_EXTERNAL_BASE,
replica);
+List dumpWithClause = Arrays.asList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='false'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname +
"'='false'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES.varname +
"'='false'",
+"'" + ReplUtils.REPL_DUMP_INCLUDE_ACID_TABLES + "'='false'"
+);
+String replPolicy = primaryDbName + ".['in[0-9]+'].['in100', 'in200',
'in1400']";
+String lastReplId = replicateAndVerify(replPolicy, null, null,
dumpWithClause,
+loadWithClause, new String[]{}, new String[]{});
+
+String[] originalFullAcidTables = new String[]{"in1", "in2", "out3",
"out4", "out5", "in100", "in200", "in300"};
Review comment:
its required for covering all scenarios.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267582)
Time Spent: 2h 20m (was: 2h 10m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267586&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267586
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297643367
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267586)
Time Spent: 3h (was: 2h 50m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267584&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267584
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297642471
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
Review comment:
yes ..but isSetForBootstrapByReplaceHandler also looks fine :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267584)
Time Spent: 2h 40m (was: 2.5h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267589&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267589
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297646657
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
+&&
(!ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))) {
+ LOG.debug("Table " + tblName + " is set for bootstrap");
+ return false;
+} else {
+ // Table satisfies both old (if its there) and current policy, dump the
alter event.
+ return true;
+}
+ }
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplicationForRename(Context
withinContext, String oldName, String newName) {
Review comment:
i think with multi iteration model in incremental load .."from" and "to"
does not make any sense. I don't see a reason why user will use it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267589)
Time Spent: 3.5h (was: 3h 20m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table i
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267590&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267590
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297651698
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -168,10 +216,18 @@ public void handle(Context withinContext) throws
Exception {
}
}
-// If the tables are filtered based on name, then needs to handle the
rename scenarios.
-if (!withinContext.replScope.includeAllTables()) {
- if (!handleForTableLevelReplication(withinContext)) {
-LOG.info("Alter event for table " + before.getTableName() + " is
skipped from dumping");
+// If the tables are filtered based on name or policy is replaced, then
needs to handle differently.
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267590)
Time Spent: 3h 40m (was: 3.5h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267594&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267594
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297651132
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
}
+ // In case of replace policy, even if the old table matches the new
policy, none of the events including create
+ // table will be replayed as the old table is set of bootstrap by
replace handler. So rename event can be skipped.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
oldName)) {
Review comment:
just for code maintainability the checks are kept.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267594)
Time Spent: 4h 10m (was: 4h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables t
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267593&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267593
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297647569
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -123,8 +148,21 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
}
+ // All the subsequent events on this table newName are going to be
skipped, so the rename is also skipped.
Review comment:
the skip is because of "isSetForBootstrapByReplaceHandler" ..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267593)
Time Spent: 4h 10m (was: 4h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
> checks based on old table, the table should be bootstrapped and event should
> be ignored. But rename
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267583&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267583
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297642233
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -786,4 +786,127 @@ public void testRenameTableScenariosExternalTable()
throws Throwable {
replicateAndVerify(replPolicy, null, lastReplId, dumpWithClause,
loadWithClause, bootstrapTables, replicatedTables);
}
+
+ @Test
+ public void testRenameTableScenariosWithReplaceExternalTable() throws
Throwable {
+List loadWithClause =
ReplicationTestUtils.externalTableBasePathWithClause(REPLICA_EXTERNAL_BASE,
replica);
+List dumpWithClause = Arrays.asList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='false'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname +
"'='false'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES.varname +
"'='false'",
+"'" + ReplUtils.REPL_DUMP_INCLUDE_ACID_TABLES + "'='false'"
+);
+String replPolicy = primaryDbName + ".['in[0-9]+'].['in100', 'in200',
'in1400']";
+String lastReplId = replicateAndVerify(replPolicy, null, null,
dumpWithClause,
+loadWithClause, new String[]{}, new String[]{});
+
+String[] originalFullAcidTables = new String[]{"in1", "in2", "out3",
"out4", "out5", "in100", "in200", "in300"};
+String[] originalExternalTables = new String[]{"in400", "out500"};
+String[] originalNonAcidTables = new String[]{"in1400", "out1500",
"out1600"};
+createTables(originalNonAcidTables, CreateTableType.NON_ACID);
+createTables(originalFullAcidTables, CreateTableType.FULL_ACID);
+createTables(originalExternalTables, CreateTableType.EXTERNAL);
+
+// Replicate and verify, no table should be there.
+String[] replicatedTables = new String[]{};
+String[] bootstrapTables = new String[]{};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId,
dumpWithClause,
+loadWithClause, bootstrapTables, replicatedTables);
+
+// Rename the tables to satisfy the condition also replace the policy.
+primary.run("use " + primaryDbName)
+.run("alter table out1600 rename to in1600")
+.run("alter table in1400 rename to out1400")
+.run("alter table in1 rename to out7")
+.run("alter table out3 rename to in8")
+.run("alter table out4 rename to in9")
+.run("drop table in9")
+.run("alter table out5 rename to in10")
+.run("alter table in10 rename to out11")
+.run("alter table out500 rename to in500")
+.run("alter table in400 rename to out400");
+
+String newPolicy = primaryDbName + ".['in[0-9]+', 'out1500'].['in2']";
+dumpWithClause = Arrays.asList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='true'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname +
"'='true'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES.varname +
"'='true'",
+"'" + ReplUtils.REPL_DUMP_INCLUDE_ACID_TABLES + "'='true'"
+);
+
+// in2 should be dropped.
+replicatedTables = new String[]{"in8", "in100", "in200", "in300", "in500",
"in1600", "out1500"};
+bootstrapTables = new String[]{"in8", "in100", "in200", "in300", "in500",
"in1600", "out1500"};
+replicateAndVerify(newPolicy, replPolicy, lastReplId, dumpWithClause,
Review comment:
i think external table behavior is same as managed ..so i don't think its
required.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267583)
Time Spent: 2.5h (was: 2h 20m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahe
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267597&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267597
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297641791
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -196,27 +196,27 @@ private void verifyBootstrapDirInIncrementalDump(String
dumpLocation, String[] b
@Test
public void testBasicBootstrapWithIncludeList() throws Throwable {
-String[] originalNonAcidTables = new String[] {"t1", "t2" };
-String[] originalFullAcidTables = new String[] {"t3", "t4" };
-String[] originalMMAcidTables = new String[] {"t5" };
+String[] originalNonAcidTables = new String[]{"t1", "t2"};
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267597)
Time Spent: 4h 40m (was: 4.5h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is ma
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267585&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267585
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297642679
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
Review comment:
its for alter and truncate so kept is simple
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267585)
Time Spent: 2h 50m (was: 2h 40m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
>
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267587&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267587
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297645445
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
+&&
(!ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))) {
+ LOG.debug("Table " + tblName + " is set for bootstrap");
+ return false;
+} else {
+ // Table satisfies both old (if its there) and current policy, dump the
alter event.
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267587)
Time Spent: 3h 10m (was: 3h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target c
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267591&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267591
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297643959
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
Review comment:
i have kept all the possible scenarios with logs and comments so that its
easier to debug later. and the code is easy to maintain. If we have missed some
condition and somebody else has to change the code, then it will be difficult
without these checks, logs and comments.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267591)
Time Spent: 3h 50m (was: 3h 40m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267595&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267595
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297651030
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
Review comment:
just for code maintainability the checks are kept.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267595)
Time Spent: 4h 20m (was: 4h 10m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
> checks based on old table, the table should be bootstrapped and event should
> be ignored. But rename handler should decide based on new name.The old table
> name will not be returned by get-all-table, so replace handler will not d
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267592&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267592
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297645678
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
+&&
(!ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))) {
+ LOG.debug("Table " + tblName + " is set for bootstrap");
+ return false;
+} else {
+ // Table satisfies both old (if its there) and current policy, dump the
alter event.
+ return true;
+}
+ }
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplicationForRename(Context
withinContext, String oldName, String newName) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267592)
Time Spent: 4h (was: 3h 50m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the tab
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267596&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267596
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 13:03
Start Date: 26/Jun/19 13:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297646878
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
+&&
(!ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))) {
+ LOG.debug("Table " + tblName + " is set for bootstrap");
+ return false;
+} else {
+ // Table satisfies both old (if its there) and current policy, dump the
alter event.
+ return true;
+}
+ }
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplicationForRename(Context
withinContext, String oldName, String newName) {
+// If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+// list of tables to be bootstrapped.
+boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(oldName);
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267596)
Time Spent: 4.5h (was: 4h 20m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy,
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267495&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267495
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297592298
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
+&&
(!ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))) {
+ LOG.debug("Table " + tblName + " is set for bootstrap");
+ return false;
+} else {
+ // Table satisfies both old (if its there) and current policy, dump the
alter event.
+ return true;
+}
+ }
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplicationForRename(Context
withinContext, String oldName, String newName) {
+// If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
+// list of tables to be bootstrapped.
+boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(oldName);
Review comment:
The variable name can be oldTableInBootstrapList.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267495)
Time Spent: 0.5h (was: 20m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267505&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267505
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297576225
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -196,27 +196,27 @@ private void verifyBootstrapDirInIncrementalDump(String
dumpLocation, String[] b
@Test
public void testBasicBootstrapWithIncludeList() throws Throwable {
-String[] originalNonAcidTables = new String[] {"t1", "t2" };
-String[] originalFullAcidTables = new String[] {"t3", "t4" };
-String[] originalMMAcidTables = new String[] {"t5" };
+String[] originalNonAcidTables = new String[]{"t1", "t2"};
Review comment:
I think, checkstyle gives warnings if no space before { and before }. That's
why I did those changes in earlier patches. Shall revert this change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267505)
Time Spent: 2h (was: 1h 50m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** I
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267504&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267504
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297604088
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -123,8 +148,21 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
}
+ // All the subsequent events on this table newName are going to be
skipped, so the rename is also skipped.
+ if (isSetForBootstrapByReplaceHandler(withinContext, newName)) {
+// If the old table satisfies the new policy and is not in the list of
tables to be bootstrapped
+// (as per previous check based on oldTableIsPresent), then drop it.
+if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
oldName)) {
+ scenario = Scenario.DROP;
+ LOG.info("Table " + oldName + " will be dropped as the table " +
newName + " will be bootstrapped.");
+ return true;
Review comment:
I think, we should also add newName to the bootstrapTablesList. I cannot
think of any specific case but guess it may cause DROP event for any
intermediate table name if we have multiple renames. Anyways, it is safe to
have it in the list so that it is fool-proof for oldTableIsPresent=true flows.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267504)
Time Spent: 1h 50m (was: 1h 40m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of t
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267498&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267498
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297587363
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -168,10 +216,18 @@ public void handle(Context withinContext) throws
Exception {
}
}
-// If the tables are filtered based on name, then needs to handle the
rename scenarios.
-if (!withinContext.replScope.includeAllTables()) {
- if (!handleForTableLevelReplication(withinContext)) {
-LOG.info("Alter event for table " + before.getTableName() + " is
skipped from dumping");
+// If the tables are filtered based on name or policy is replaced, then
needs to handle differently.
Review comment:
The comment is not clear. Shall change it to "Handle alter and renames for
table level replication."
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267498)
Time Spent: 1h (was: 50m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
>
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267499&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267499
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297588891
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
Review comment:
No need to explicitly check (withinContext.oldReplScope != null) here.
ReplUtils.tableIncludedInReplScope handles it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267499)
Time Spent: 1h 10m (was: 1h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new pol
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267506&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267506
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297583519
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
Review comment:
The method name can be isTableMarkedForBootstrapDump()
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267506)
Time Spent: 2h 10m (was: 2h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstr
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267492&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267492
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297586695
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
Review comment:
The method name can be alterHandlerForTableLevelReplication().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267492)
Time Spent: 20m (was: 10m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
>
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267501&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267501
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297595694
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -123,8 +148,21 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
}
+ // All the subsequent events on this table newName are going to be
skipped, so the rename is also skipped.
Review comment:
Shall correct the comment saying that "If newName is not matching old
policy, then all subsequent events on this table would be skipped."
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267501)
Time Spent: 1.5h (was: 1h 20m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
> checks based on old table, t
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267496&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267496
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297581731
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -786,4 +786,127 @@ public void testRenameTableScenariosExternalTable()
throws Throwable {
replicateAndVerify(replPolicy, null, lastReplId, dumpWithClause,
loadWithClause, bootstrapTables, replicatedTables);
}
+
+ @Test
+ public void testRenameTableScenariosWithReplaceExternalTable() throws
Throwable {
+List loadWithClause =
ReplicationTestUtils.externalTableBasePathWithClause(REPLICA_EXTERNAL_BASE,
replica);
+List dumpWithClause = Arrays.asList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='false'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname +
"'='false'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES.varname +
"'='false'",
+"'" + ReplUtils.REPL_DUMP_INCLUDE_ACID_TABLES + "'='false'"
+);
+String replPolicy = primaryDbName + ".['in[0-9]+'].['in100', 'in200',
'in1400']";
+String lastReplId = replicateAndVerify(replPolicy, null, null,
dumpWithClause,
+loadWithClause, new String[]{}, new String[]{});
+
+String[] originalFullAcidTables = new String[]{"in1", "in2", "out3",
"out4", "out5", "in100", "in200", "in300"};
+String[] originalExternalTables = new String[]{"in400", "out500"};
+String[] originalNonAcidTables = new String[]{"in1400", "out1500",
"out1600"};
+createTables(originalNonAcidTables, CreateTableType.NON_ACID);
+createTables(originalFullAcidTables, CreateTableType.FULL_ACID);
+createTables(originalExternalTables, CreateTableType.EXTERNAL);
+
+// Replicate and verify, no table should be there.
+String[] replicatedTables = new String[]{};
+String[] bootstrapTables = new String[]{};
+lastReplId = replicateAndVerify(replPolicy, null, lastReplId,
dumpWithClause,
+loadWithClause, bootstrapTables, replicatedTables);
+
+// Rename the tables to satisfy the condition also replace the policy.
+primary.run("use " + primaryDbName)
+.run("alter table out1600 rename to in1600")
+.run("alter table in1400 rename to out1400")
+.run("alter table in1 rename to out7")
+.run("alter table out3 rename to in8")
+.run("alter table out4 rename to in9")
+.run("drop table in9")
+.run("alter table out5 rename to in10")
+.run("alter table in10 rename to out11")
+.run("alter table out500 rename to in500")
+.run("alter table in400 rename to out400");
+
+String newPolicy = primaryDbName + ".['in[0-9]+', 'out1500'].['in2']";
+dumpWithClause = Arrays.asList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='true'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname +
"'='true'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES.varname +
"'='true'",
+"'" + ReplUtils.REPL_DUMP_INCLUDE_ACID_TABLES + "'='true'"
+);
+
+// in2 should be dropped.
+replicatedTables = new String[]{"in8", "in100", "in200", "in300", "in500",
"in1600", "out1500"};
+bootstrapTables = new String[]{"in8", "in100", "in200", "in300", "in500",
"in1600", "out1500"};
+replicateAndVerify(newPolicy, replPolicy, lastReplId, dumpWithClause,
Review comment:
For external tables, can we have one more iteration with replace policy +
rename but no BOOTSTRAP config.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267496)
Time Spent: 40m (was: 0.5h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267502&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267502
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297611643
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
+&&
(!ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))) {
+ LOG.debug("Table " + tblName + " is set for bootstrap");
+ return false;
+} else {
+ // Table satisfies both old (if its there) and current policy, dump the
alter event.
+ return true;
+}
+ }
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplicationForRename(Context
withinContext, String oldName, String newName) {
Review comment:
Out of context of this patch. Since, dynamic bootstrap have issues(such as
bootstrapping wrong tables, subsequent REPL DUMP includes events on old table
etc) if user specifies TO or LIMIT clause in REPL DUMP, should we validate it
and return error if table level replication is enabled in repl policy.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267502)
Time Spent: 1h 40m (was: 1.5h)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2.
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267494&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267494
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297586465
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
+&&
(!ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))) {
+ LOG.debug("Table " + tblName + " is set for bootstrap");
+ return false;
+} else {
+ // Table satisfies both old (if its there) and current policy, dump the
alter event.
+ return true;
+}
+ }
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplicationForRename(Context
withinContext, String oldName, String newName) {
Review comment:
The method name can be renameHandlerForTableLevelReplication().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267494)
Time Spent: 0.5h (was: 20m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
>
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267493&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267493
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297606106
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
Review comment:
In fact, all the flows in this else path returns false. So, shall remove
this return statement.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267493)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed the table name
> from the list of tables to be bootstrapped.
> ## Old name is matching new policy – As per replace policy handler, which
> checks based on old table, the table should be bootstrapped and event should
> be ignored. But rename handler should decide based on new name.The old table
> name will not be returned by get-all-table, so replace handler will not d
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267491&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267491
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297582247
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -786,4 +786,127 @@ public void testRenameTableScenariosExternalTable()
throws Throwable {
replicateAndVerify(replPolicy, null, lastReplId, dumpWithClause,
loadWithClause, bootstrapTables, replicatedTables);
}
+
+ @Test
+ public void testRenameTableScenariosWithReplaceExternalTable() throws
Throwable {
+List loadWithClause =
ReplicationTestUtils.externalTableBasePathWithClause(REPLICA_EXTERNAL_BASE,
replica);
+List dumpWithClause = Arrays.asList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='false'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname +
"'='false'",
+"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES.varname +
"'='false'",
+"'" + ReplUtils.REPL_DUMP_INCLUDE_ACID_TABLES + "'='false'"
+);
+String replPolicy = primaryDbName + ".['in[0-9]+'].['in100', 'in200',
'in1400']";
+String lastReplId = replicateAndVerify(replPolicy, null, null,
dumpWithClause,
+loadWithClause, new String[]{}, new String[]{});
+
+String[] originalFullAcidTables = new String[]{"in1", "in2", "out3",
"out4", "out5", "in100", "in200", "in300"};
Review comment:
Can we reduce the number of tables?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267491)
Time Spent: 20m (was: 10m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
>
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267503&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267503
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297605541
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -135,8 +173,15 @@ private boolean handleForTableLevelReplication(Context
withinContext) {
return false;
}
+ // In case of replace policy, even if the old table matches the new
policy, none of the events including create
+ // table will be replayed as the old table is set of bootstrap by
replace handler. So rename event can be skipped.
+ if (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
oldName)) {
Review comment:
This check is not significant as the else flow is also return false. Shall
remove it and keep the comment to explain this case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267503)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not matching old policy
> New name not matching new policy
> * Ignore the event, no need to do anything.
> New name matching new policy
> * The table will be returned by get-all-table. Replace policy handler
> will bootstrap this table as its matching new policy and not matching old
> policy.
> * All the future events will be ignored as part of check added by
> replace policy handling.
> * All the event with old table name will anyways be ignored as the old
> name is not matching the new policy.
> ### New name matching old policy
> New name not matching new policy
> * As the new name is not matching the new policy, the table need not be
> replicated.
> * As the old name is not matching the new policy, the rename events will
> be ignored.
> * So nothing to be done for this scenario.
> New name matching new policy
> * As the new name is matching both old and new policy, replace handler
> will not bootstrap the table.
> * Add the table to the list of tables to be bootstrapped.
> * Ignore all the events with new name.
> * If there is a drop event for the table (with new name), then remove
> the table from the the list of table to be bootstrapped.
> * In case of rename event (double rename)
> ** If the new name satisfies the table pattern, then add the new name to
> the list of tables to be bootstrapped and remove the old name from the list
> of tables to be bootstrapped.
> ** If the new name does not satisfies then just removed th
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267500&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267500
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297590982
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
Review comment:
Shall combine this check with if condition itself as both yields "return
false". For readability, we can reply on comments. I think, it is self
explanatory as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267500)
Time Spent: 1h 20m (was: 1h 10m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in the list of tables to be bootstrapped,
> then ignore the event.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> ## Old name is not matching new policy
> ### New name not match
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[
https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267497&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267497
]
ASF GitHub Bot logged work on HIVE-21886:
-
Author: ASF GitHub Bot
Created on: 26/Jun/19 11:17
Start Date: 26/Jun/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #688: HIVE-21886 :
REPL - With table list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297589673
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -92,16 +93,40 @@ private Scenario
scenarioType(org.apache.hadoop.hive.metastore.api.Table before,
}
}
- // return true, if event needs to be dumped, else return false.
- private boolean handleForTableLevelReplication(Context withinContext) {
-String oldName = before.getTableName();
-String newName = after.getTableName();
+ private boolean isSetForBootstrapByReplaceHandler(Context withinContext,
String tblName) {
+return (withinContext.oldReplScope != null)
+&&
!(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+&& (ReplUtils.tableIncludedInReplScope(withinContext.replScope,
tblName));
+ }
-if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
- // If the table is renamed after being added to the list of tables to be
bootstrapped, then remove it from the
- // list of tables to be bootstrapped.
- boolean oldTableIsPresent =
withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+ // Return true, if event needs to be dumped, else return false.
+ private boolean handleForTableLevelReplication(Context withinContext, String
tblName) {
+// For alter, if the table does not satisfy the new policy then ignore the
event. In case of replace
+// policy, if the table does not satisfy the old policy, then ignore the
event. As, if the table satisfy the new
+// policy, then the table will be bootstrapped by replace handler anb if
the table does not satisfy the new policy,
+// then anyways the table should be ignored.
+if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName))
{
+ // In case of replace, it will be dropped during load. In normal case
just ignore the alter event.
+ LOG.debug("Table " + tblName + " does not satisfy the policy");
+ return false;
+} else if ((withinContext.oldReplScope != null)
+&&
(!ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))) {
+ LOG.debug("Table " + tblName + " is set for bootstrap");
+ return false;
+} else {
+ // Table satisfies both old (if its there) and current policy, dump the
alter event.
Review comment:
What if this is a renamed table which is marked for bootstrap and this alter
event would be dumped? Need to check if this table is part of bootstrap tables
list as well and decide if need to dump this event.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 267497)
Time Spent: 50m (was: 40m)
> REPL - With table list - Handle rename events during replace policy
> ---
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
> 1. Create a list of tables to be bootstrapped.
> 2. During handling of alter table, if the alter type is rename
> 1. If the old table name is present in the list of table to be
> bootstrapped, remove it.
> 2. If the new table name, matches the new policy, add it to the list
> of tables to be bootstrapped.
> 3. During handling of drop table
> 1. if the table is in the list of tables to be bootstrapped, then
> remove it and ignore the event.
> 4. During other event handling
> 1. if the table is there in th
[jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
[ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=266841&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266841 ] ASF GitHub Bot logged work on HIVE-21886: - Author: ASF GitHub Bot Created on: 25/Jun/19 16:47 Start Date: 25/Jun/19 16:47 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #688: HIVE-21886 : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688 … This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266841) Time Spent: 10m Remaining Estimate: 0h > REPL - With table list - Handle rename events during replace policy > --- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace > policy is getting executed, it needs to take care of the policy inclusion in > both the policy for each table name. > 1. Create a list of tables to be bootstrapped. > 2. During handling of alter table, if the alter type is rename > 1. If the old table name is present in the list of table to be > bootstrapped, remove it. > 2. If the new table name, matches the new policy, add it to the list > of tables to be bootstrapped. > 3. During handling of drop table > 1. if the table is in the list of tables to be bootstrapped, then > remove it and ignore the event. > 4. During other event handling > 1. if the table is there in the list of tables to be bootstrapped, > then ignore the event. > > Rename handling during replace policy > # Old name not matching old policy – The old table will not be there at the > target cluster. The table will not be returned by get-all-table. > ## Old name is not matching new policy > ### New name not matching old policy > New name not matching new policy > * Ignore the event, no need to do anything. > New name matching new policy > * The table will be returned by get-all-table. Replace policy handler > will bootstrap this table as its matching new policy and not matching old > policy. > * All the future events will be ignored as part of check added by > replace policy handling. > * All the event with old table name will anyways be ignored as the old > name is not matching the new policy. > ### New name matching old policy > New name not matching new policy > * As the new name is not matching the new policy, the table need not be > replicated. > * As the old name is not matching the new policy, the rename events will > be ignored. > * So nothing to be done for this scenario. > New name matching new policy > * As the new name is matching both old and new policy, replace handler > will not bootstrap the table. > * Add the table to the list of tables to be bootstrapped. > * Ignore all the events with new name. > * If there is a drop event for the table (with new name), then remove > the table from the the list of table to be bootstrapped. > * In case of rename event (double rename) > ** If the new name satisfies the table pattern, then add the new name to > the list of tables to be bootstrapped and remove the old name from the list > of tables to be bootstrapped. > ** If the new name does not satisfies then just removed the table name > from the list of tables to be bootstrapped. > ## Old name is matching new policy – As per replace policy handler, which > checks based on old table, the table should be bootstrapped and event should > be ignored. But rename handler should decide based on new name.The old table > name will not be returned by get-all-table, so replace handler will not d > anything for the old table. > ### New name not matching old policy > New name not matching new policy > * As the old table is not there at target and new name is not matching > new policy. Ignore the event. > * No need to add the table to the list of tables to be bootstrapped. > * All the subsequent eve
