[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[ https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=446414=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446414 ] ASF GitHub Bot logged work on HIVE-21731: - Author: ASF GitHub Bot Created on: 16/Jun/20 10:54 Start Date: 16/Jun/20 10:54 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #628: URL: https://github.com/apache/hive/pull/628 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446414) Time Spent: 3h 40m (was: 3.5h) > Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 > cluster with strict managed table set to true. > - > > Key: HIVE-21731 > URL: https://issues.apache.org/jira/browse/HIVE-21731 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch, > HIVE-21731.03.patch, HIVE-21731.04.patch, HIVE-21731.05.patch > > Time Spent: 3h 40m > Remaining Estimate: 0h > > The scenario is > # Replication policy is set with hive 3.0 source cluster (strict managed > table set to false) and hive 4.0 target cluster with strict managed table set > true. > # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool. > # The upgrade converts all managed tables to acid tables. > # In the next repl dump, user sets hive .repl .dump .include .acid .tables > and hive .repl .bootstrap. acid. tables set true triggering bootstrap of > newly converted ACID tables. > # As the old tables are non-txn tables, dump is not filtering the events > even tough bootstrap acid table is set to true. This is causing the repl load > to fail as the write id is not set in the table object. > # If we ignore the event replay, the bootstrap is failing with dump > directory mismatch error. > The fix should be > # Ignore dumping the alter table event if bootstrap acid table is set true > and the alter is converting a non-acid table to acid table. > # In case of bootstrap during incremental load, ignore the dump directory > property set in table object. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[ https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=442808=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-442808 ] ASF GitHub Bot logged work on HIVE-21731: - Author: ASF GitHub Bot Created on: 09/Jun/20 16:00 Start Date: 09/Jun/20 16:00 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #628: URL: https://github.com/apache/hive/pull/628#issuecomment-641143437 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 442808) Time Spent: 3.5h (was: 3h 20m) > Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 > cluster with strict managed table set to true. > - > > Key: HIVE-21731 > URL: https://issues.apache.org/jira/browse/HIVE-21731 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch, > HIVE-21731.03.patch, HIVE-21731.04.patch, HIVE-21731.05.patch > > Time Spent: 3.5h > Remaining Estimate: 0h > > The scenario is > # Replication policy is set with hive 3.0 source cluster (strict managed > table set to false) and hive 4.0 target cluster with strict managed table set > true. > # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool. > # The upgrade converts all managed tables to acid tables. > # In the next repl dump, user sets hive .repl .dump .include .acid .tables > and hive .repl .bootstrap. acid. tables set true triggering bootstrap of > newly converted ACID tables. > # As the old tables are non-txn tables, dump is not filtering the events > even tough bootstrap acid table is set to true. This is causing the repl load > to fail as the write id is not set in the table object. > # If we ignore the event replay, the bootstrap is failing with dump > directory mismatch error. > The fix should be > # Ignore dumping the alter table event if bootstrap acid table is set true > and the alter is converting a non-acid table to acid table. > # In case of bootstrap during incremental load, ignore the dump directory > property set in table object. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=244551=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244551
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 18/May/19 14:18
Start Date: 18/May/19 14:18
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r285343933
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##
@@ -471,6 +472,7 @@ private int executeIncrementalLoad(DriverContext
driverContext) {
if (work.hasBootstrapLoadTasks()) {
LOG.debug("Current incremental dump have tables to be bootstrapped.
Switching to bootstrap "
+ "mode after applying all events.");
+ work.setBootstrapDuringIncLoad(true);
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244551)
Time Spent: 3h 20m (was: 3h 10m)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch,
> HIVE-21731.03.patch, HIVE-21731.04.patch, HIVE-21731.05.patch
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=244550=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244550
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 18/May/19 14:18
Start Date: 18/May/19 14:18
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r285343930
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##
@@ -264,6 +264,7 @@ a database ( directory )
|| work.getPathsToCopyIterator().hasNext();
if (addAnotherLoadTask) {
+// pass on the bootstrap during incremental flag for next iteration.
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244550)
Time Spent: 3h 10m (was: 3h)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch,
> HIVE-21731.03.patch, HIVE-21731.04.patch, HIVE-21731.05.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=244549=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244549
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 18/May/19 14:17
Start Date: 18/May/19 14:17
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r285343925
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/WarehouseInstance.java
##
@@ -563,7 +563,21 @@ public void testEventCounts(String dbName, long
fromEventId, Long toEventId, Int
}
public boolean isAcidEnabled() {
-return hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY);
+if (hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY) &&
+
hiveConf.getVar(HiveConf.ConfVars.HIVE_TXN_MANAGER).equals("org.apache.hadoop.hive.ql.lockmgr.DbTxnManager"))
{
+ return true;
+}
+return false;
+ }
+
+ public void disableAcid() {
+hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY, false);
+hiveConf.setVar(HiveConf.ConfVars.HIVE_TXN_MANAGER,
"org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager");
+ }
+
+ public void enableAcid() {
+hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY, true);
+hiveConf.setVar(HiveConf.ConfVars.HIVE_TXN_MANAGER,
"org.apache.hadoop.hive.ql.lockmgr.DbTxnManager");
Review comment:
removed it from warehouse instance and added to the specific test
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244549)
Time Spent: 3h (was: 2h 50m)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch,
> HIVE-21731.03.patch, HIVE-21731.04.patch, HIVE-21731.05.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=244164=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244164
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 17/May/19 17:54
Start Date: 17/May/19 17:54
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r285227479
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/WarehouseInstance.java
##
@@ -563,7 +563,21 @@ public void testEventCounts(String dbName, long
fromEventId, Long toEventId, Int
}
public boolean isAcidEnabled() {
-return hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY);
+if (hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY) &&
+
hiveConf.getVar(HiveConf.ConfVars.HIVE_TXN_MANAGER).equals("org.apache.hadoop.hive.ql.lockmgr.DbTxnManager"))
{
+ return true;
+}
+return false;
+ }
+
+ public void disableAcid() {
+hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY, false);
+hiveConf.setVar(HiveConf.ConfVars.HIVE_TXN_MANAGER,
"org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager");
+ }
+
+ public void enableAcid() {
+hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY, true);
+hiveConf.setVar(HiveConf.ConfVars.HIVE_TXN_MANAGER,
"org.apache.hadoop.hive.ql.lockmgr.DbTxnManager");
Review comment:
sure. Can we add a comment about this in this method? So that later someone
need not wonder why is it not working.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244164)
Time Spent: 2h 50m (was: 2h 40m)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch,
> HIVE-21731.03.patch, HIVE-21731.04.patch
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=244000=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244000
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 17/May/19 12:33
Start Date: 17/May/19 12:33
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r28510
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/WarehouseInstance.java
##
@@ -563,7 +563,21 @@ public void testEventCounts(String dbName, long
fromEventId, Long toEventId, Int
}
public boolean isAcidEnabled() {
-return hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY);
+if (hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY) &&
+
hiveConf.getVar(HiveConf.ConfVars.HIVE_TXN_MANAGER).equals("org.apache.hadoop.hive.ql.lockmgr.DbTxnManager"))
{
+ return true;
+}
+return false;
+ }
+
+ public void disableAcid() {
+hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY, false);
+hiveConf.setVar(HiveConf.ConfVars.HIVE_TXN_MANAGER,
"org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager");
+ }
+
+ public void enableAcid() {
+hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY, true);
+hiveConf.setVar(HiveConf.ConfVars.HIVE_TXN_MANAGER,
"org.apache.hadoop.hive.ql.lockmgr.DbTxnManager");
Review comment:
ok ..when ever its required ..it can be done ..currently i don't see any
reason to add extra init if its not required
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244000)
Time Spent: 2h 40m (was: 2.5h)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch,
> HIVE-21731.03.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243804=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243804
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 17/May/19 03:23
Start Date: 17/May/19 03:23
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284966682
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##
@@ -471,6 +472,7 @@ private int executeIncrementalLoad(DriverContext
driverContext) {
if (work.hasBootstrapLoadTasks()) {
LOG.debug("Current incremental dump have tables to be bootstrapped.
Switching to bootstrap "
+ "mode after applying all events.");
+ work.setBootstrapDuringIncLoad(true);
Review comment:
We shall set it in ReplLoadWork constructor once for all.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243804)
Time Spent: 2.5h (was: 2h 20m)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch,
> HIVE-21731.03.patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243803=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243803
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 17/May/19 03:22
Start Date: 17/May/19 03:22
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284966591
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##
@@ -264,6 +264,7 @@ a database ( directory )
|| work.getPathsToCopyIterator().hasNext();
if (addAnotherLoadTask) {
+// pass on the bootstrap during incremental flag for next iteration.
Review comment:
No need of this comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243803)
Time Spent: 2h 20m (was: 2h 10m)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch,
> HIVE-21731.03.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243802
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 17/May/19 03:21
Start Date: 17/May/19 03:21
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284966493
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/WarehouseInstance.java
##
@@ -563,7 +563,21 @@ public void testEventCounts(String dbName, long
fromEventId, Long toEventId, Int
}
public boolean isAcidEnabled() {
-return hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY);
+if (hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY) &&
+
hiveConf.getVar(HiveConf.ConfVars.HIVE_TXN_MANAGER).equals("org.apache.hadoop.hive.ql.lockmgr.DbTxnManager"))
{
+ return true;
+}
+return false;
+ }
+
+ public void disableAcid() {
+hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY, false);
+hiveConf.setVar(HiveConf.ConfVars.HIVE_TXN_MANAGER,
"org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager");
+ }
+
+ public void enableAcid() {
+hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY, true);
+hiveConf.setVar(HiveConf.ConfVars.HIVE_TXN_MANAGER,
"org.apache.hadoop.hive.ql.lockmgr.DbTxnManager");
Review comment:
It is working for this test because, replica warehouse is acid enabled and
it initialises the derby DB which is shared by both warehouse instances. But,
if both primary and replica are acid disabled, then this won't work if you
dynamically change this config.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243802)
Time Spent: 2h 10m (was: 2h)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch,
> HIVE-21731.03.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243532=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243532
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 16/May/19 18:20
Start Date: 16/May/19 18:20
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284828151
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -91,6 +92,14 @@ public void handle(Context withinContext) throws Exception {
return;
}
+if
(withinContext.hiveConf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES))
{
+ if (!AcidUtils.isTransactionalTable(before) &&
AcidUtils.isTransactionalTable(after)) {
+LOG.info("The table " + after.getTableName() + " is converted to ACID
table." +
+" It will be replicated with bootstrap load as
REPL_BOOTSTRAP_ACID_TABLES is set to true.");
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243532)
Time Spent: 1h 40m (was: 1.5h)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243533=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243533
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 16/May/19 18:20
Start Date: 16/May/19 18:20
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284828193
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadTable.java
##
@@ -159,10 +159,20 @@ public TaskTracker tasks() throws Exception {
return tracker;
}
- private ReplLoadOpType getLoadTableType(Table table) throws
InvalidOperationException, HiveException {
+ private ReplLoadOpType getLoadTableType(Table table, boolean
isBootstrapDuringInc)
+ throws InvalidOperationException, HiveException {
if (table == null) {
return ReplLoadOpType.LOAD_NEW;
}
+
+// In case user has asked for bootstrap of transactional table, we replace
the old one if present. This is to
+// make sure that the transactional info like write id etc for the table
is consistent between the
+// source and target cluster.
+if (isBootstrapDuringInc && AcidUtils.isTransactionalTable(table)) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243533)
Time Spent: 1h 40m (was: 1.5h)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243534=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243534
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 16/May/19 18:20
Start Date: 16/May/19 18:20
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284829909
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationWithTableMigration.java
##
@@ -504,4 +507,56 @@ public void dynamicallyConvertExternalToManagedTable()
throws Throwable {
.runFailure("alter table t1 set tblproperties('EXTERNAL'='false')")
.runFailure("alter table t2 set
tblproperties('EXTERNAL'='false')");
}
+
+ @Test
+ public void testMigrationWithUpgrade() throws Throwable {
+WarehouseInstance.Tuple tuple = primary.run("use " + primaryDbName)
+.run("create table tacid (id int) clustered by(id) into 3 buckets
stored as orc ")
+.run("create table texternal (id int) ")
+.run("insert into texternal values (1)")
+.dump(primaryDbName, null);
+replica.load(replicatedDbName, tuple.dumpLocation)
+.run("use " + replicatedDbName)
+.run("repl status " + replicatedDbName)
+.verifyResult(tuple.lastReplicationId)
+.run("select count(*) from tacid")
+.verifyResult("0")
+.run("select id from texternal")
+.verifyResult("1");
+
+assertTrue(isFullAcidTable(replica.getTable(replicatedDbName, "tacid")));
+
assertFalse(MetaStoreUtils.isExternalTable(replica.getTable(replicatedDbName,
"texternal")));
+
+// forcefully (setting db property) alter the table type. For acid table,
set the bootstrap acid table to true. For
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243534)
Time Spent: 1h 50m (was: 1h 40m)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243531=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243531
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 16/May/19 18:20
Start Date: 16/May/19 18:20
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284828988
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/WarehouseInstance.java
##
@@ -563,7 +563,21 @@ public void testEventCounts(String dbName, long
fromEventId, Long toEventId, Int
}
public boolean isAcidEnabled() {
-return hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY);
+if (hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY) &&
+
hiveConf.getVar(HiveConf.ConfVars.HIVE_TXN_MANAGER).equals("org.apache.hadoop.hive.ql.lockmgr.DbTxnManager"))
{
+ return true;
+}
+return false;
+ }
+
+ public void disableAcid() {
+hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY, false);
+hiveConf.setVar(HiveConf.ConfVars.HIVE_TXN_MANAGER,
"org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager");
+ }
+
+ public void enableAcid() {
+hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY, true);
+hiveConf.setVar(HiveConf.ConfVars.HIVE_TXN_MANAGER,
"org.apache.hadoop.hive.ql.lockmgr.DbTxnManager");
Review comment:
its working fine for this test ..i could see the table getting converted to
acid and bootstrap happening
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243531)
Time Spent: 1.5h (was: 1h 20m)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243535=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243535
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 16/May/19 18:20
Start Date: 16/May/19 18:20
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284830345
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationWithTableMigration.java
##
@@ -504,4 +507,56 @@ public void dynamicallyConvertExternalToManagedTable()
throws Throwable {
.runFailure("alter table t1 set tblproperties('EXTERNAL'='false')")
.runFailure("alter table t2 set
tblproperties('EXTERNAL'='false')");
}
+
+ @Test
+ public void testMigrationWithUpgrade() throws Throwable {
+WarehouseInstance.Tuple tuple = primary.run("use " + primaryDbName)
+.run("create table tacid (id int) clustered by(id) into 3 buckets
stored as orc ")
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243535)
Time Spent: 2h (was: 1h 50m)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243530=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243530
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 16/May/19 18:20
Start Date: 16/May/19 18:20
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284828762
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##
@@ -471,7 +473,7 @@ private int executeIncrementalLoad(DriverContext
driverContext) {
if (work.hasBootstrapLoadTasks()) {
LOG.debug("Current incremental dump have tables to be bootstrapped.
Switching to bootstrap "
+ "mode after applying all events.");
- return executeBootStrapLoad(driverContext);
+ return executeBootStrapLoad(driverContext, true);
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243530)
Time Spent: 1h 20m (was: 1h 10m)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243425=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243425
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 16/May/19 15:52
Start Date: 16/May/19 15:52
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284759007
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationWithTableMigration.java
##
@@ -504,4 +507,56 @@ public void dynamicallyConvertExternalToManagedTable()
throws Throwable {
.runFailure("alter table t1 set tblproperties('EXTERNAL'='false')")
.runFailure("alter table t2 set
tblproperties('EXTERNAL'='false')");
}
+
+ @Test
+ public void testMigrationWithUpgrade() throws Throwable {
+WarehouseInstance.Tuple tuple = primary.run("use " + primaryDbName)
+.run("create table tacid (id int) clustered by(id) into 3 buckets
stored as orc ")
Review comment:
Better to have some data in tacid table too. To know if writeId seeding
impacts data read after bootstrap.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243425)
Time Spent: 1h 10m (was: 1h)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243422=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243422
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 16/May/19 15:52
Start Date: 16/May/19 15:52
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284758759
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationWithTableMigration.java
##
@@ -504,4 +507,56 @@ public void dynamicallyConvertExternalToManagedTable()
throws Throwable {
.runFailure("alter table t1 set tblproperties('EXTERNAL'='false')")
.runFailure("alter table t2 set
tblproperties('EXTERNAL'='false')");
}
+
+ @Test
+ public void testMigrationWithUpgrade() throws Throwable {
+WarehouseInstance.Tuple tuple = primary.run("use " + primaryDbName)
+.run("create table tacid (id int) clustered by(id) into 3 buckets
stored as orc ")
+.run("create table texternal (id int) ")
+.run("insert into texternal values (1)")
+.dump(primaryDbName, null);
+replica.load(replicatedDbName, tuple.dumpLocation)
+.run("use " + replicatedDbName)
+.run("repl status " + replicatedDbName)
+.verifyResult(tuple.lastReplicationId)
+.run("select count(*) from tacid")
+.verifyResult("0")
+.run("select id from texternal")
+.verifyResult("1");
+
+assertTrue(isFullAcidTable(replica.getTable(replicatedDbName, "tacid")));
+
assertFalse(MetaStoreUtils.isExternalTable(replica.getTable(replicatedDbName,
"texternal")));
+
+// forcefully (setting db property) alter the table type. For acid table,
set the bootstrap acid table to true. For
Review comment:
Shall explicitly mention that this is a mock up for
HiveStrictManagedMigration tool which is used during upgrade to migrate the
tables based on migration rules.
We simulate the scenario by explicitly allowing alter commands to change
table types.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243422)
Time Spent: 40m (was: 0.5h)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243424=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243424
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 16/May/19 15:52
Start Date: 16/May/19 15:52
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284776123
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
##
@@ -91,6 +92,14 @@ public void handle(Context withinContext) throws Exception {
return;
}
+if
(withinContext.hiveConf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES))
{
+ if (!AcidUtils.isTransactionalTable(before) &&
AcidUtils.isTransactionalTable(after)) {
+LOG.info("The table " + after.getTableName() + " is converted to ACID
table." +
+" It will be replicated with bootstrap load as
REPL_BOOTSTRAP_ACID_TABLES is set to true.");
Review comment:
Use hive.repl.bootstrap.acid.tables instead of REPL_BOOTSTRAP_ACID_TABLES.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243424)
Time Spent: 1h (was: 50m)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243423=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243423
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 16/May/19 15:52
Start Date: 16/May/19 15:52
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284777561
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadTable.java
##
@@ -159,10 +159,20 @@ public TaskTracker tasks() throws Exception {
return tracker;
}
- private ReplLoadOpType getLoadTableType(Table table) throws
InvalidOperationException, HiveException {
+ private ReplLoadOpType getLoadTableType(Table table, boolean
isBootstrapDuringInc)
+ throws InvalidOperationException, HiveException {
if (table == null) {
return ReplLoadOpType.LOAD_NEW;
}
+
+// In case user has asked for bootstrap of transactional table, we replace
the old one if present. This is to
+// make sure that the transactional info like write id etc for the table
is consistent between the
+// source and target cluster.
+if (isBootstrapDuringInc && AcidUtils.isTransactionalTable(table)) {
Review comment:
I think, this check should be done even for external tables. Why don't we
always replace table in case of isBootstrapDuringInc?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243423)
Time Spent: 50m (was: 40m)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243420=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243420
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 16/May/19 15:52
Start Date: 16/May/19 15:52
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284771911
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/WarehouseInstance.java
##
@@ -563,7 +563,21 @@ public void testEventCounts(String dbName, long
fromEventId, Long toEventId, Int
}
public boolean isAcidEnabled() {
-return hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY);
+if (hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY) &&
+
hiveConf.getVar(HiveConf.ConfVars.HIVE_TXN_MANAGER).equals("org.apache.hadoop.hive.ql.lockmgr.DbTxnManager"))
{
+ return true;
+}
+return false;
+ }
+
+ public void disableAcid() {
+hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY, false);
+hiveConf.setVar(HiveConf.ConfVars.HIVE_TXN_MANAGER,
"org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager");
+ }
+
+ public void enableAcid() {
+hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY, true);
+hiveConf.setVar(HiveConf.ConfVars.HIVE_TXN_MANAGER,
"org.apache.hadoop.hive.ql.lockmgr.DbTxnManager");
Review comment:
Is it enough to do this? I think, TxnDbUtil.prepDb should be done,
otherwise, it won't work. As the modified test suit is migration case, replica
warehouse instance init has initialised it and so it worked. But won't work
otherwise.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243420)
Time Spent: 20m (was: 10m)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[
https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=243421=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243421
]
ASF GitHub Bot logged work on HIVE-21731:
-
Author: ASF GitHub Bot
Created on: 16/May/19 15:52
Start Date: 16/May/19 15:52
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #628: HIVE-21731 :
Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster
with strict managed table set to true.
URL: https://github.com/apache/hive/pull/628#discussion_r284773568
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##
@@ -471,7 +473,7 @@ private int executeIncrementalLoad(DriverContext
driverContext) {
if (work.hasBootstrapLoadTasks()) {
LOG.debug("Current incremental dump have tables to be bootstrapped.
Switching to bootstrap "
+ "mode after applying all events.");
- return executeBootStrapLoad(driverContext);
+ return executeBootStrapLoad(driverContext, true);
Review comment:
Can we set isBootstrapDuringIncLoad in ReplLoadWork object constructor
itself? so that need not pass it as extra argument and need not set it after
each iteration.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243421)
Time Spent: 0.5h (was: 20m)
> Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0
> cluster with strict managed table set to true.
> -
>
> Key: HIVE-21731
> URL: https://issues.apache.org/jira/browse/HIVE-21731
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21731.01.patch, HIVE-21731.02.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The scenario is
> # Replication policy is set with hive 3.0 source cluster (strict managed
> table set to false) and hive 4.0 target cluster with strict managed table set
> true.
> # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool.
> # The upgrade converts all managed tables to acid tables.
> # In the next repl dump, user sets hive .repl .dump .include .acid .tables
> and hive .repl .bootstrap. acid. tables set true triggering bootstrap of
> newly converted ACID tables.
> # As the old tables are non-txn tables, dump is not filtering the events
> even tough bootstrap acid table is set to true. This is causing the repl load
> to fail as the write id is not set in the table object.
> # If we ignore the event replay, the bootstrap is failing with dump
> directory mismatch error.
> The fix should be
> # Ignore dumping the alter table event if bootstrap acid table is set true
> and the alter is converting a non-acid table to acid table.
> # In case of bootstrap during incremental load, ignore the dump directory
> property set in table object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21731) Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true.
[ https://issues.apache.org/jira/browse/HIVE-21731?focusedWorklogId=242345=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242345 ] ASF GitHub Bot logged work on HIVE-21731: - Author: ASF GitHub Bot Created on: 15/May/19 08:52 Start Date: 15/May/19 08:52 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #628: HIVE-21731 : Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 cluster with strict managed table set to true. URL: https://github.com/apache/hive/pull/628 … This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 242345) Time Spent: 10m Remaining Estimate: 0h > Hive import fails, post upgrade of source 3.0 cluster, to a target 4.0 > cluster with strict managed table set to true. > - > > Key: HIVE-21731 > URL: https://issues.apache.org/jira/browse/HIVE-21731 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The scenario is > # Replication policy is set with hive 3.0 source cluster (strict managed > table set to false) and hive 4.0 target cluster with strict managed table set > true. > # User upgrades the 3.0 source cluster to 4.0 cluster using upgrade tool. > # The upgrade converts all managed tables to acid tables. > # In the next repl dump, user sets hive .repl .dump .include .acid .tables > and hive .repl .bootstrap. acid. tables set true triggering bootstrap of > newly converted ACID tables. > # As the old tables are non-txn tables, dump is not filtering the events > even tough bootstrap acid table is set to true. This is causing the repl load > to fail as the write id is not set in the table object. > # If we ignore the event replay, the bootstrap is failing with dump > directory mismatch error. > The fix should be > # Ignore dumping the alter table event if bootstrap acid table is set true > and the alter is converting a non-acid table to acid table. > # In case of bootstrap during incremental load, ignore the dump directory > property set in table object. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
