[GitHub] [hive] sankarh closed pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
sankarh closed pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target. URL: https://github.com/apache/hive/pull/578 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
sankarh commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268483462
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -673,6 +674,59 @@ public void
retryIncBootstrapExternalTablesFromDifferentDumpWithoutCleanTablesCo
ErrorMsg.REPL_BOOTSTRAP_LOAD_PATH_NOT_VALID.getErrorCode());
}
+ @Test
+ public void dynamicallyConvertManagedToExternalTable() throws Throwable {
+List dumpWithClause = Collections.singletonList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='true'"
+);
+List loadWithClause = externalTableBasePathWithClause();
+
+WarehouseInstance.Tuple tupleBootstrapManagedTable = primary.run("use " +
primaryDbName)
+.run("create table t1 (id int)")
+.run("insert into table t1 values (1)")
+.run("create table t2 (id int) partitioned by (key int)")
Review comment:
Did you mean, a non-acid table is created as acid table in target (due to
migration) and later in source they change to external? I think, this case,
need lot of changes to avoid distcp for external table and so on. I will create
another ticket for this as there might be other cases too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
sankarh commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268483462
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -673,6 +674,59 @@ public void
retryIncBootstrapExternalTablesFromDifferentDumpWithoutCleanTablesCo
ErrorMsg.REPL_BOOTSTRAP_LOAD_PATH_NOT_VALID.getErrorCode());
}
+ @Test
+ public void dynamicallyConvertManagedToExternalTable() throws Throwable {
+List dumpWithClause = Collections.singletonList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='true'"
+);
+List loadWithClause = externalTableBasePathWithClause();
+
+WarehouseInstance.Tuple tupleBootstrapManagedTable = primary.run("use " +
primaryDbName)
+.run("create table t1 (id int)")
+.run("insert into table t1 values (1)")
+.run("create table t2 (id int) partitioned by (key int)")
Review comment:
Did you mean, a non-acid table is created as acid table in target (due to
migration) and later in source they change to external?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
sankarh commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268480500
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
##
@@ -400,7 +405,26 @@ public void alterTable(RawStore msdb, Warehouse wh,
String catName, String dbnam
"Unable to change partition or table. Object " + e.getMessage() + "
does not exist."
+ " Check metastore logs for detailed stack.");
} finally {
- if (!success) {
+ if (success) {
+// Txn was committed successfully.
+// If data location is changed in replication flow, then need to
delete the old path.
+if (replDataLocationChanged) {
+ assert(olddb != null);
+ assert(oldt != null);
+ Path deleteOldDataLoc = new Path(oldt.getSd().getLocation());
+ boolean isAutoPurge =
"true".equalsIgnoreCase(oldt.getParameters().get("auto.purge"));
+ try {
+wh.deleteDir(deleteOldDataLoc, true, isAutoPurge, olddb);
+LOG.info("Deleted the old data location: {} for the table: {}",
+deleteOldDataLoc, dbname + "." + name);
+ } catch (MetaException ex) {
+// Eat the exception as it doesn't affect the state of existing
tables.
+// Expect, user to manually drop this path when exception and so
logging a warning.
+LOG.warn("Unable to delete the old data location: {} for the
table: {}",
Review comment:
I think, if old dir is deleted, then metadata update was already successful.
In this case, during replay of event in next cycle would not set the flag as
the table/partition locations were already pointing to new location under base
dir.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
sankarh commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268480500
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
##
@@ -400,7 +405,26 @@ public void alterTable(RawStore msdb, Warehouse wh,
String catName, String dbnam
"Unable to change partition or table. Object " + e.getMessage() + "
does not exist."
+ " Check metastore logs for detailed stack.");
} finally {
- if (!success) {
+ if (success) {
+// Txn was committed successfully.
+// If data location is changed in replication flow, then need to
delete the old path.
+if (replDataLocationChanged) {
+ assert(olddb != null);
+ assert(oldt != null);
+ Path deleteOldDataLoc = new Path(oldt.getSd().getLocation());
+ boolean isAutoPurge =
"true".equalsIgnoreCase(oldt.getParameters().get("auto.purge"));
+ try {
+wh.deleteDir(deleteOldDataLoc, true, isAutoPurge, olddb);
+LOG.info("Deleted the old data location: {} for the table: {}",
+deleteOldDataLoc, dbname + "." + name);
+ } catch (MetaException ex) {
+// Eat the exception as it doesn't affect the state of existing
tables.
+// Expect, user to manually drop this path when exception and so
logging a warning.
+LOG.warn("Unable to delete the old data location: {} for the
table: {}",
Review comment:
Doesn't matter, we ignore that exception here. REPL LOAD will succeed. Also,
previous run already archived in CM dir.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
sankarh commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268480382
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
##
@@ -209,7 +214,13 @@ public void alterTable(RawStore msdb, Warehouse wh,
String catName, String dbnam
boolean tableInSpecifiedLoc = !oldtRelativePath.equalsIgnoreCase(name)
&& !oldtRelativePath.equalsIgnoreCase(name + Path.SEPARATOR);
-if (!tableInSpecifiedLoc) {
+if (replDataLocationChanged) {
+ // If data location is changed in replication flow, then new path
was already set in
+ // the newt. Also, it is as good as the data is moved and set
dataWasMoved=true so that
+ // location in partitions are also updated accordingly.
+ destPath = new Path(newt.getSd().getLocation());
Review comment:
Yes. it is already like that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
sankarh commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268480328
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
##
@@ -192,12 +197,12 @@ public void alterTable(RawStore msdb, Warehouse wh,
String catName, String dbnam
// 2) the table is not an external table, and
// 3) the user didn't change the default location (or new location is
empty), and
// 4) the table was not initially created with a specified location
- if (rename
- && !oldt.getTableType().equals(TableType.VIRTUAL_VIEW.toString())
- && (oldt.getSd().getLocation().compareTo(newt.getSd().getLocation())
== 0
-|| StringUtils.isEmpty(newt.getSd().getLocation()))
- && !MetaStoreUtils.isExternalTable(oldt)) {
-Database olddb = msdb.getDatabase(catName, dbname);
+ if (replDataLocationChanged
+ || (rename
Review comment:
It cannot be same because the behaviour is different. If location is set for
normal flow, we shouldn't update for partitions. Also, it is risky to modify
any of current behaviour. It is not in scope of this ticket.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
sankarh commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268480242
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/ReplConst.java
##
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.common;
+
+/**
+ * A class that defines the constant strings used by the replication
implementation.
+ */
+
+public class ReplConst {
Review comment:
I will keep it ReplConst for now. We have ReplUtils for common methods and
if needed we can add another class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268476396
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
##
@@ -400,7 +405,26 @@ public void alterTable(RawStore msdb, Warehouse wh,
String catName, String dbnam
"Unable to change partition or table. Object " + e.getMessage() + "
does not exist."
+ " Check metastore logs for detailed stack.");
} finally {
- if (!success) {
+ if (success) {
+// Txn was committed successfully.
+// If data location is changed in replication flow, then need to
delete the old path.
+if (replDataLocationChanged) {
+ assert(olddb != null);
+ assert(oldt != null);
+ Path deleteOldDataLoc = new Path(oldt.getSd().getLocation());
+ boolean isAutoPurge =
"true".equalsIgnoreCase(oldt.getParameters().get("auto.purge"));
+ try {
+wh.deleteDir(deleteOldDataLoc, true, isAutoPurge, olddb);
+LOG.info("Deleted the old data location: {} for the table: {}",
+deleteOldDataLoc, dbname + "." + name);
+ } catch (MetaException ex) {
+// Eat the exception as it doesn't affect the state of existing
tables.
+// Expect, user to manually drop this path when exception and so
logging a warning.
+LOG.warn("Unable to delete the old data location: {} for the
table: {}",
Review comment:
if the delete directory succeeds and the event reply fails for some other
reason, then the event will be replayed again. In the next replay, if it does
not finds the directory, cm copy might fail.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target. URL: https://github.com/apache/hive/pull/578#discussion_r268479992 ## File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java ## @@ -99,9 +100,12 @@ public void alterTable(RawStore msdb, Warehouse wh, String catName, String dbnam dbname = dbname.toLowerCase(); final boolean cascade = environmentContext != null -&& environmentContext.isSetProperties() -&& StatsSetupConst.TRUE.equals(environmentContext.getProperties().get( -StatsSetupConst.CASCADE)); +&& environmentContext.isSetProperties() +&& StatsSetupConst.TRUE.equals(environmentContext.getProperties().get(StatsSetupConst.CASCADE)); +final boolean replDataLocationChanged = environmentContext != null Review comment: I think flag is needed as we shouldn't update partition locations if user set location for a table. That is current behaviour and shouldn't be changed. Only for repl flow, this is needed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
sankarh commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268480113
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -673,6 +674,59 @@ public void
retryIncBootstrapExternalTablesFromDifferentDumpWithoutCleanTablesCo
ErrorMsg.REPL_BOOTSTRAP_LOAD_PATH_NOT_VALID.getErrorCode());
}
+ @Test
+ public void dynamicallyConvertManagedToExternalTable() throws Throwable {
+List dumpWithClause = Collections.singletonList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='true'"
+);
+List loadWithClause = externalTableBasePathWithClause();
+
+WarehouseInstance.Tuple tupleBootstrapManagedTable = primary.run("use " +
primaryDbName)
+.run("create table t1 (id int)")
+.run("insert into table t1 values (1)")
+.run("create table t2 (id int) partitioned by (key int)")
Review comment:
It is not acid table. It is non-acid managed table.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268475619
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/ReplConst.java
##
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.common;
+
+/**
+ * A class that defines the constant strings used by the replication
implementation.
+ */
+
+public class ReplConst {
Review comment:
The name can be repl common as its in common folder
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268474981
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -673,6 +674,59 @@ public void
retryIncBootstrapExternalTablesFromDifferentDumpWithoutCleanTablesCo
ErrorMsg.REPL_BOOTSTRAP_LOAD_PATH_NOT_VALID.getErrorCode());
}
+ @Test
+ public void dynamicallyConvertManagedToExternalTable() throws Throwable {
+List dumpWithClause = Collections.singletonList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='true'"
+);
+List loadWithClause = externalTableBasePathWithClause();
+
+WarehouseInstance.Tuple tupleBootstrapManagedTable = primary.run("use " +
primaryDbName)
+.run("create table t1 (id int)")
+.run("insert into table t1 values (1)")
+.run("create table t2 (id int) partitioned by (key int)")
Review comment:
the case does not exist that a managed acid table is converted to external
..it should be always using migration that an acid table will be converted to
external ..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268475390
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
##
@@ -209,7 +214,13 @@ public void alterTable(RawStore msdb, Warehouse wh,
String catName, String dbnam
boolean tableInSpecifiedLoc = !oldtRelativePath.equalsIgnoreCase(name)
&& !oldtRelativePath.equalsIgnoreCase(name + Path.SEPARATOR);
-if (!tableInSpecifiedLoc) {
+if (replDataLocationChanged) {
+ // If data location is changed in replication flow, then new path
was already set in
+ // the newt. Also, it is as good as the data is moved and set
dataWasMoved=true so that
+ // location in partitions are also updated accordingly.
+ destPath = new Path(newt.getSd().getLocation());
Review comment:
if for a partition the location is not within the table location and that
table is altered to a external table ..the partition location need not be
changed to base path at target ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268475390
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
##
@@ -209,7 +214,13 @@ public void alterTable(RawStore msdb, Warehouse wh,
String catName, String dbnam
boolean tableInSpecifiedLoc = !oldtRelativePath.equalsIgnoreCase(name)
&& !oldtRelativePath.equalsIgnoreCase(name + Path.SEPARATOR);
-if (!tableInSpecifiedLoc) {
+if (replDataLocationChanged) {
+ // If data location is changed in replication flow, then new path
was already set in
+ // the newt. Also, it is as good as the data is moved and set
dataWasMoved=true so that
+ // location in partitions are also updated accordingly.
+ destPath = new Path(newt.getSd().getLocation());
Review comment:
if for a partition the location is not within the table location and that
table is altered to a external table ..the partition location need not be
changed to base path ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268475152
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
##
@@ -192,12 +197,12 @@ public void alterTable(RawStore msdb, Warehouse wh,
String catName, String dbnam
// 2) the table is not an external table, and
// 3) the user didn't change the default location (or new location is
empty), and
// 4) the table was not initially created with a specified location
- if (rename
- && !oldt.getTableType().equals(TableType.VIRTUAL_VIEW.toString())
- && (oldt.getSd().getLocation().compareTo(newt.getSd().getLocation())
== 0
-|| StringUtils.isEmpty(newt.getSd().getLocation()))
- && !MetaStoreUtils.isExternalTable(oldt)) {
-Database olddb = msdb.getDatabase(catName, dbname);
+ if (replDataLocationChanged
+ || (rename
Review comment:
what i meant is the flow can be kept same for replication and normal flow to
avoid adding extra complexity ..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268474981
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -673,6 +674,59 @@ public void
retryIncBootstrapExternalTablesFromDifferentDumpWithoutCleanTablesCo
ErrorMsg.REPL_BOOTSTRAP_LOAD_PATH_NOT_VALID.getErrorCode());
}
+ @Test
+ public void dynamicallyConvertManagedToExternalTable() throws Throwable {
+List dumpWithClause = Collections.singletonList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='true'"
+);
+List loadWithClause = externalTableBasePathWithClause();
+
+WarehouseInstance.Tuple tupleBootstrapManagedTable = primary.run("use " +
primaryDbName)
+.run("create table t1 (id int)")
+.run("insert into table t1 values (1)")
+.run("create table t2 (id int) partitioned by (key int)")
Review comment:
the case does not exist that a managed acid table is converted to external
..it should be always in using migration that an acid table will be converted
to external ..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.
maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target. URL: https://github.com/apache/hive/pull/578#discussion_r268474795 ## File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java ## @@ -99,9 +100,12 @@ public void alterTable(RawStore msdb, Warehouse wh, String catName, String dbnam dbname = dbname.toLowerCase(); final boolean cascade = environmentContext != null -&& environmentContext.isSetProperties() -&& StatsSetupConst.TRUE.equals(environmentContext.getProperties().get( -StatsSetupConst.CASCADE)); +&& environmentContext.isSetProperties() +&& StatsSetupConst.TRUE.equals(environmentContext.getProperties().get(StatsSetupConst.CASCADE)); +final boolean replDataLocationChanged = environmentContext != null Review comment: yes ..i think the flag sent from hive server to meta store using environment context is not required ...the code changes are redundant .. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HIVE-21499) should not remove the function if create command failed with AlreadyExistsException
Rajkumar Singh created HIVE-21499: - Summary: should not remove the function if create command failed with AlreadyExistsException Key: HIVE-21499 URL: https://issues.apache.org/jira/browse/HIVE-21499 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 3.1.0 Environment: Hive-3.1 Reporter: Rajkumar Singh As a part of HIVE-20953 we are removing the function if creation for same failed with any reason, this will yield into the following situation. 1. create function failed since function already exists 2. on #1 failure hive will clear the permanent function from the registry 3. this function will be of no use until hiveserver2 restarted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 70286: HIVE-21305: LLAP: Option to skip cache for ETL queries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70286/#review213954 --- Ship it! Ship It! - Gopal V On March 23, 2019, 2:47 a.m., prasanthj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70286/ > --- > > (Updated March 23, 2019, 2:47 a.m.) > > > Review request for hive and Gopal V. > > > Bugs: HIVE-21305 > https://issues.apache.org/jira/browse/HIVE-21305 > > > Repository: hive-git > > > Description > --- > > HIVE-21305: LLAP: Option to skip cache for ETL queries > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0c783e144d > itests/src/test/resources/testconfiguration.properties 8c4d9b7de7 > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java > 6252013335 > ql/src/test/queries/clientpositive/llap_io_etl.q PRE-CREATION > ql/src/test/results/clientpositive/llap/llap_io_etl.q.out PRE-CREATION > > > Diff: https://reviews.apache.org/r/70286/diff/2/ > > > Testing > --- > > > Thanks, > > prasanthj > >
[jira] [Created] (HIVE-21498) Upgrade Thrift to 0.12.0
Ajith S created HIVE-21498: -- Summary: Upgrade Thrift to 0.12.0 Key: HIVE-21498 URL: https://issues.apache.org/jira/browse/HIVE-21498 Project: Hive Issue Type: Bug Reporter: Ajith S Upgrade to consider security fixes. Especially https://issues.apache.org/jira/browse/THRIFT-4506 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
