[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

sankarh commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268381343
 
 

 ##
 File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ##
 @@ -357,6 +368,13 @@ public void alterTable(RawStore msdb, Warehouse wh, 
String catName, String dbnam
 }
   }
 
+  // If data location is changed in replication flow, then need to delete 
the old path.
+  if (replDataLocationChanged) {
+Path deleteOldDataLoc = new Path(oldt.getSd().getLocation());
+boolean isAutoPurge = 
"true".equalsIgnoreCase(oldt.getParameters().get("auto.purge"));
+wh.deleteDir(deleteOldDataLoc, true, isAutoPurge, olddb);
 
 Review comment:
   Good catch. Will move it to finally block only when success=true and eat any 
exception by DeleteDir and just log a warn instead.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

sankarh commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268381321
 
 

 ##
 File path: 
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/ReplConst.java
 ##
 @@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.common;
+
+/**
+ * A class that defines the constant strings used by the replication 
implementation.
+ */
+
+public class ReplConst {
 
 Review comment:
   ReplUtils was already taken. I used similar naming as StatsSetupConst. I 
think ReplConst is good enough for now. Will change if any utility methods 
comes up for metastore common.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

sankarh commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268381295
 
 

 ##
 File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ##
 @@ -192,12 +197,12 @@ public void alterTable(RawStore msdb, Warehouse wh, 
String catName, String dbnam
   // 2) the table is not an external table, and
   // 3) the user didn't change the default location (or new location is 
empty), and
   // 4) the table was not initially created with a specified location
-  if (rename
-  && !oldt.getTableType().equals(TableType.VIRTUAL_VIEW.toString())
-  && (oldt.getSd().getLocation().compareTo(newt.getSd().getLocation()) 
== 0
-|| StringUtils.isEmpty(newt.getSd().getLocation()))
-  && !MetaStoreUtils.isExternalTable(oldt)) {
-Database olddb = msdb.getDatabase(catName, dbname);
+  if (replDataLocationChanged
+  || (rename
 
 Review comment:
   This patch is not dealing with set location at source. It deals with 
changing table type to external at source via ALTER table EXTERNAL=true in 
table properties. So, this scenario is not valid and not in scope.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

sankarh commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268381269
 
 

 ##
 File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ##
 @@ -209,7 +214,13 @@ public void alterTable(RawStore msdb, Warehouse wh, 
String catName, String dbnam
 boolean tableInSpecifiedLoc = !oldtRelativePath.equalsIgnoreCase(name)
 
 Review comment:
   OK


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

sankarh commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268381258
 
 

 ##
 File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ##
 @@ -209,7 +214,13 @@ public void alterTable(RawStore msdb, Warehouse wh, 
String catName, String dbnam
 boolean tableInSpecifiedLoc = !oldtRelativePath.equalsIgnoreCase(name)
 && !oldtRelativePath.equalsIgnoreCase(name + Path.SEPARATOR);
 
-if (!tableInSpecifiedLoc) {
+if (replDataLocationChanged) {
+  // If data location is changed in replication flow, then new path 
was already set in
+  // the newt. Also, it is as good as the data is moved and set 
dataWasMoved=true so that
+  // location in partitions are also updated accordingly.
+  destPath = new Path(newt.getSd().getLocation());
 
 Review comment:
   Already handled. If you notice the line 279, the partition location is 
changed only if the old location is a sub-dir of old table location.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

sankarh commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268381242
 
 

 ##
 File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ##
 @@ -192,12 +197,12 @@ public void alterTable(RawStore msdb, Warehouse wh, 
String catName, String dbnam
   // 2) the table is not an external table, and
 
 Review comment:
   OK


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

sankarh commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268381212
 
 

 ##
 File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ##
 @@ -99,9 +100,12 @@ public void alterTable(RawStore msdb, Warehouse wh, String 
catName, String dbnam
 dbname = dbname.toLowerCase();
 
 final boolean cascade = environmentContext != null
-&& environmentContext.isSetProperties()
-&& StatsSetupConst.TRUE.equals(environmentContext.getProperties().get(
-StatsSetupConst.CASCADE));
+&& environmentContext.isSetProperties()
+&& 
StatsSetupConst.TRUE.equals(environmentContext.getProperties().get(StatsSetupConst.CASCADE));
+final boolean replDataLocationChanged = environmentContext != null
 
 Review comment:
   OK.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

sankarh commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268381204
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/CreateTableOperation.java
 ##
 @@ -108,6 +116,12 @@ private void createTableReplaceMode(Table tbl) throws 
HiveException {
   }
 }
 
+// If table's data location is moved, then set the corresponding flag in 
environment context to
 
 Review comment:
   Nope. The scenario is location is changed at target not at source. At source 
only the table type is changed from managed to external not location. I will 
update the comment to say that in repl flow.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

sankarh commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268381165
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/CreateTableOperation.java
 ##
 @@ -54,17 +55,24 @@ public int execute() throws HiveException {
 Table tbl = desc.toTable(context.getConf());
 LOG.debug("creating table {} on {}", tbl.getFullyQualifiedName(), 
tbl.getDataLocation());
 
-if (desc.getReplicationSpec().isInReplicationScope() && 
(!desc.getReplaceMode())){
-  // if this is a replication spec, then replace-mode semantics might 
apply.
-  // if we're already asking for a table replacement, then we can skip 
this check.
-  // however, otherwise, if in replication scope, and we've not been 
explicitly asked
-  // to replace, we should check if the object we're looking at exists, 
and if so,
+boolean dataLocationChanged = false;
+if (desc.getReplicationSpec().isInReplicationScope()) {
+  // If in replication scope, we should check if the object we're looking 
at exists, and if so,
   // trigger replace-mode semantics.
   Table existingTable = context.getDb().getTable(tbl.getDbName(), 
tbl.getTableName(), false);
-  if (existingTable != null){
+  if (existingTable != null) {
 if 
(desc.getReplicationSpec().allowEventReplacementInto(existingTable.getParameters()))
 {
   desc.setReplaceMode(true); // we replace existing table.
   ReplicationSpec.copyLastReplId(existingTable.getParameters(), 
tbl.getParameters());
+
+  // If location of an existing managed table is changed, then need to 
delete the old location if exists.
+  // This scenario occurs when a managed table is converted into 
external table at source. In this case,
+  // at target, the table data would be moved to different location 
under base directory for external tables.
+  if (existingTable.getTableType().equals(TableType.MANAGED_TABLE)
+  && tbl.getTableType().equals(TableType.EXTERNAL_TABLE)
+  && 
(!existingTable.getDataLocation().equals(tbl.getDataLocation( {
 
 Review comment:
   It's not possible in replication flow. But I kept the check explicitly for 
better readability. There is no harm in keeping it. If you insist, let me 
change it to assert.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

sankarh commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268381147
 
 

 ##
 File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
 ##
 @@ -673,6 +674,59 @@ public void 
retryIncBootstrapExternalTablesFromDifferentDumpWithoutCleanTablesCo
 ErrorMsg.REPL_BOOTSTRAP_LOAD_PATH_NOT_VALID.getErrorCode());
   }
 
+  @Test
+  public void dynamicallyConvertManagedToExternalTable() throws Throwable {
+List dumpWithClause = Collections.singletonList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname + 
"'='true'"
+);
+List loadWithClause = externalTableBasePathWithClause();
+
+WarehouseInstance.Tuple tupleBootstrapManagedTable = primary.run("use " + 
primaryDbName)
+.run("create table t1 (id int)")
+.run("insert into table t1 values (1)")
+.run("create table t2 (id int) partitioned by (key int)")
 
 Review comment:
   Nope. It is not a case of migration. It is external table replication 
specific where the table is converted to external at source not at target. So, 
here location is changed only at target not in source.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

sankarh commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r26838
 
 

 ##
 File path: 
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/ReplConst.java
 ##
 @@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.common;
+
+/**
+ * A class that defines the constant strings used by the replication 
implementation.
+ */
+
+public class ReplConst {
+
+  /**
+   * The constant that denotes the table data location is changed to different 
path. This indicates
+   * Metastore to update corresponding path in Partitions and also need to 
delete old path.
+   */
+  public static final String DATA_LOCATION_CHANGED = "DATA_LOCATION_CHANGED";
 
 Review comment:
   I thought ReplConst is self explanatory that it is Repl specific. Anyways, I 
will prefix with REPL_


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] sankarh commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

sankarh commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268381082
 
 

 ##
 File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ##
 @@ -99,9 +100,12 @@ public void alterTable(RawStore msdb, Warehouse wh, String 
catName, String dbnam
 dbname = dbname.toLowerCase();
 
 final boolean cascade = environmentContext != null
-&& environmentContext.isSetProperties()
-&& StatsSetupConst.TRUE.equals(environmentContext.getProperties().get(
-StatsSetupConst.CASCADE));
+&& environmentContext.isSetProperties()
+&& 
StatsSetupConst.TRUE.equals(environmentContext.getProperties().get(StatsSetupConst.CASCADE));
+final boolean replDataLocationChanged = environmentContext != null
 
 Review comment:
   It is not possible in normal flow to have table type changed from managed to 
external and location set in same ALTER query. Also, it won't reach here as it 
is CreateTableTask flow. Only repl flow reaches here for this scenario.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: Review Request 70286: HIVE-21305: LLAP: Option to skip cache for ETL queries

[prasanthj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70286/
---

(Updated March 23, 2019, 2:47 a.m.)


Review request for hive and Gopal V.


Bugs: HIVE-21305
https://issues.apache.org/jira/browse/HIVE-21305


Repository: hive-git


Description
---

HIVE-21305: LLAP: Option to skip cache for ETL queries


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0c783e144d 
  itests/src/test/resources/testconfiguration.properties 8c4d9b7de7 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 6252013335 
  ql/src/test/queries/clientpositive/llap_io_etl.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/llap_io_etl.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/70286/diff/2/

Changes: https://reviews.apache.org/r/70286/diff/1-2/


Testing
---


Thanks,

prasanthj

Review Request 70286: HIVE-21305: LLAP: Option to skip cache for ETL queries

[prasanthj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70286/
---

Review request for hive and Gopal V.


Bugs: HIVE-21305
https://issues.apache.org/jira/browse/HIVE-21305


Repository: hive-git


Description
---

HIVE-21305: LLAP: Option to skip cache for ETL queries


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
0c783e144dd8cb0bc8f9368f553868759556 
  itests/src/test/resources/testconfiguration.properties 
8c4d9b7de71ed6762a2f918c163ecfb7499b4c03 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
6252013335f4e985e43d58227ffd7294bc0ebdf8 
  ql/src/test/queries/clientpositive/llap_io_etl.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/llap_io_etl.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/70286/diff/1/


Testing
---


Thanks,

prasanthj

[jira] [Created] (HIVE-21495) Calcite assertion error when UDF returns null

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-21495:


 Summary: Calcite assertion error when UDF returns null
 Key: HIVE-21495
 URL: https://issues.apache.org/jira/browse/HIVE-21495
 Project: Hive
  Issue Type: Bug
  Components: Logical Optimizer
Affects Versions: 4.0.0
Reporter: Prasanth Jayachandran
Assignee: Jesus Camacho Rodriguez


Calcite throws the following error when UDFs return null. 
"current_authorizer()" for example can return null if authorizer is disabled.
{code:java}
org.apache.hive.service.cli.HiveSQLException: Error running query: 
java.lang.AssertionError: Cannot add expression of different type to set:
set type is RecordType(CHAR(7) CHARACTER SET "UTF-16LE" COLLATE 
"ISO-8859-1$en_US$primary" NOT NULL $f0, VARCHAR(2147483647) CHARACTER SET 
"UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" $f1, VARCHAR(2147483647) 
CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" $f2, 
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" 
$f3, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE 
"ISO-8859-1$en_US$primary" $f4, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" 
COLLATE "ISO-8859-1$en_US$primary" $f5, VARCHAR(2147483647) CHARACTER SET 
"UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" $f6, VARCHAR(2147483647) 
CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" $f7, 
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" 
$f8, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE 
"ISO-8859-1$en_US$primary" $f9, CHAR(2) CHARACTER SET "UTF-16LE" COLLATE 
"ISO-8859-1$en_US$primary" NOT NULL $f10, VARCHAR(2147483647) CHARACTER SET 
"UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" $f11) NOT NULL
expression type is RecordType(CHAR(7) CHARACTER SET "UTF-16LE" COLLATE 
"ISO-8859-1$en_US$primary" NOT NULL $f0, CHAR(7) CHARACTER SET "UTF-16LE" 
COLLATE "ISO-8859-1$en_US$primary" NOT NULL $f1, VARCHAR(2147483647) CHARACTER 
SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" $f2, VARCHAR(2147483647) 
CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" $f3, 
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" 
$f4, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE 
"ISO-8859-1$en_US$primary" $f5, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" 
COLLATE "ISO-8859-1$en_US$primary" $f6, VARCHAR(2147483647) CHARACTER SET 
"UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" $f7, VARCHAR(2147483647) 
CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" $f8, 
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" 
$f9, CHAR(2) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT 
NULL $f10, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE 
"ISO-8859-1$en_US$primary" $f11) NOT NULL
set is rel#784:HiveAggregate.HIVE.[](input=HepRelVertex#783,group={0, 1, 2, 3, 
4, 5, 6, 7, 8, 9, 10, 11})
expression is HiveProject#829
at 
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:210)
at 
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at 
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:342)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.AssertionError: Cannot add expression of different type to 
set:
set type is RecordType(CHAR(7) CHARACTER SET "UTF-16LE" COLLATE 
"ISO-8859-1$en_US$primary" NOT NULL $f0, VARCHAR(2147483647) CHARACTER SET 
"UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" $f1, VARCHAR(2147483647) 
CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" $f2, 
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" 
$f3, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE 
"ISO-8859-1$en_US$primary" $f4, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" 
COLLATE "ISO-8859-1$en_US$primary" $f5, VARCHAR(2147483647) CHARACTER SET 
"UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" $f6, VARCHAR(2147483647) 
CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" $f7, 
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" 
$f8, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE 
"ISO-8859-1$en_US$primary" $f9, CHAR(2) CHARACTER SET "UTF-16LE" 

[jira] [Created] (HIVE-21494) Wrong CASE branch chosen seemingly dependent on contents of branch

[David Ross (JIRA)]

David Ross created HIVE-21494:
-

 Summary: Wrong CASE branch chosen seemingly dependent on contents 
of branch
 Key: HIVE-21494
 URL: https://issues.apache.org/jira/browse/HIVE-21494
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.1.0
Reporter: David Ross


The following query returns {{-100}} which is logically incorrect:

 
{code:java}
SELECT

    CASE

        WHEN COALESCE(

                CASE

                    WHEN COALESCE( CAST('Infinity' AS DOUBLE), 0 ) = 0

                    THEN -1

                    ELSE ( 5 / (CAST('Infinity' AS DOUBLE) * SIGN(5)) )

                END, 0 ) = 0

        THEN (

            CAST('Infinity' AS DOUBLE)

            -- -200

            )

        ELSE -100

    END full_query{code}
 

The expected behavior is for it to return {{Infinity}}, or the contents of the 
{{THEN}} branch.

 

If the {{THEN}} branch is edited like this, it returns -{{200}} as expected:

 
{code:java}
SELECT

    CASE

        WHEN COALESCE(

                CASE

                    WHEN COALESCE( CAST('Infinity' AS DOUBLE), 0 ) = 0

                    THEN -1

                    ELSE ( 5 / (CAST('Infinity' AS DOUBLE) * SIGN(5)) )

                END, 0 ) = 0

        THEN (

           -- CAST('Infinity' AS DOUBLE)

             -200

            )

        ELSE -100

    END full_query{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21493) BuddyAllocator - Metrics count for allocated arenas wrong if preallocation is done

[Oliver Draese (JIRA)]

Oliver Draese created HIVE-21493:


 Summary: BuddyAllocator - Metrics count for allocated arenas wrong 
if preallocation is done
 Key: HIVE-21493
 URL: https://issues.apache.org/jira/browse/HIVE-21493
 Project: Hive
  Issue Type: Bug
  Components: llap
Affects Versions: 3.1.1
Reporter: Oliver Draese
Assignee: Oliver Draese
 Fix For: 4.0.0


The (Hadoop/JMX) metrics are not correctly initialized if arena preallocation 
is done and the arena count is greater than 1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268226195
 
 

 ##
 File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ##
 @@ -209,7 +214,13 @@ public void alterTable(RawStore msdb, Warehouse wh, 
String catName, String dbnam
 boolean tableInSpecifiedLoc = !oldtRelativePath.equalsIgnoreCase(name)
 && !oldtRelativePath.equalsIgnoreCase(name + Path.SEPARATOR);
 
-if (!tableInSpecifiedLoc) {
+if (replDataLocationChanged) {
+  // If data location is changed in replication flow, then new path 
was already set in
+  // the newt. Also, it is as good as the data is moved and set 
dataWasMoved=true so that
+  // location in partitions are also updated accordingly.
+  destPath = new Path(newt.getSd().getLocation());
 
 Review comment:
   Need to handle the scenario where the partition is not within table location


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268128979
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/CreateTableOperation.java
 ##
 @@ -54,17 +55,24 @@ public int execute() throws HiveException {
 Table tbl = desc.toTable(context.getConf());
 LOG.debug("creating table {} on {}", tbl.getFullyQualifiedName(), 
tbl.getDataLocation());
 
-if (desc.getReplicationSpec().isInReplicationScope() && 
(!desc.getReplaceMode())){
-  // if this is a replication spec, then replace-mode semantics might 
apply.
-  // if we're already asking for a table replacement, then we can skip 
this check.
-  // however, otherwise, if in replication scope, and we've not been 
explicitly asked
-  // to replace, we should check if the object we're looking at exists, 
and if so,
+boolean dataLocationChanged = false;
+if (desc.getReplicationSpec().isInReplicationScope()) {
+  // If in replication scope, we should check if the object we're looking 
at exists, and if so,
   // trigger replace-mode semantics.
   Table existingTable = context.getDb().getTable(tbl.getDbName(), 
tbl.getTableName(), false);
-  if (existingTable != null){
+  if (existingTable != null) {
 if 
(desc.getReplicationSpec().allowEventReplacementInto(existingTable.getParameters()))
 {
   desc.setReplaceMode(true); // we replace existing table.
   ReplicationSpec.copyLastReplId(existingTable.getParameters(), 
tbl.getParameters());
+
+  // If location of an existing managed table is changed, then need to 
delete the old location if exists.
+  // This scenario occurs when a managed table is converted into 
external table at source. In this case,
+  // at target, the table data would be moved to different location 
under base directory for external tables.
+  if (existingTable.getTableType().equals(TableType.MANAGED_TABLE)
+  && tbl.getTableType().equals(TableType.EXTERNAL_TABLE)
+  && 
(!existingTable.getDataLocation().equals(tbl.getDataLocation( {
 
 Review comment:
   in what scenario the location will be same for conversion from managed to 
external in replication flow ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268222534
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/CreateTableOperation.java
 ##
 @@ -108,6 +116,12 @@ private void createTableReplaceMode(Table tbl) throws 
HiveException {
   }
 }
 
+// If table's data location is moved, then set the corresponding flag in 
environment context to
 
 Review comment:
   comment should be , if the location is changed at source. The comment gives 
wrong impression that it is done for normal flow also.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268225105
 
 

 ##
 File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ##
 @@ -192,12 +197,12 @@ public void alterTable(RawStore msdb, Warehouse wh, 
String catName, String dbnam
   // 2) the table is not an external table, and
 
 Review comment:
   update the comment for the extra condition added


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268235202
 
 

 ##
 File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ##
 @@ -209,7 +214,13 @@ public void alterTable(RawStore msdb, Warehouse wh, 
String catName, String dbnam
 boolean tableInSpecifiedLoc = !oldtRelativePath.equalsIgnoreCase(name)
 
 Review comment:
   these are not required if replDataLocationChanged is true 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268223879
 
 

 ##
 File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ##
 @@ -99,9 +100,12 @@ public void alterTable(RawStore msdb, Warehouse wh, String 
catName, String dbnam
 dbname = dbname.toLowerCase();
 
 final boolean cascade = environmentContext != null
-&& environmentContext.isSetProperties()
-&& StatsSetupConst.TRUE.equals(environmentContext.getProperties().get(
-StatsSetupConst.CASCADE));
+&& environmentContext.isSetProperties()
+&& 
StatsSetupConst.TRUE.equals(environmentContext.getProperties().get(StatsSetupConst.CASCADE));
+final boolean replDataLocationChanged = environmentContext != null
 
 Review comment:
   environmentContext != null and environmentContext.isSetProperties() is done 
twice


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268126938
 
 

 ##
 File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ##
 @@ -99,9 +100,12 @@ public void alterTable(RawStore msdb, Warehouse wh, String 
catName, String dbnam
 dbname = dbname.toLowerCase();
 
 final boolean cascade = environmentContext != null
-&& environmentContext.isSetProperties()
-&& StatsSetupConst.TRUE.equals(environmentContext.getProperties().get(
-StatsSetupConst.CASCADE));
+&& environmentContext.isSetProperties()
+&& 
StatsSetupConst.TRUE.equals(environmentContext.getProperties().get(StatsSetupConst.CASCADE));
+final boolean replDataLocationChanged = environmentContext != null
 
 Review comment:
   why only repl ? if the table is changed from managed to external and the 
location is not same ..the old path should be deleted if its owned by hive as 
we do for drop table


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268132700
 
 

 ##
 File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ##
 @@ -192,12 +197,12 @@ public void alterTable(RawStore msdb, Warehouse wh, 
String catName, String dbnam
   // 2) the table is not an external table, and
   // 3) the user didn't change the default location (or new location is 
empty), and
   // 4) the table was not initially created with a specified location
-  if (rename
-  && !oldt.getTableType().equals(TableType.VIRTUAL_VIEW.toString())
-  && (oldt.getSd().getLocation().compareTo(newt.getSd().getLocation()) 
== 0
-|| StringUtils.isEmpty(newt.getSd().getLocation()))
-  && !MetaStoreUtils.isExternalTable(oldt)) {
-Database olddb = msdb.getDatabase(catName, dbname);
+  if (replDataLocationChanged
+  || (rename
 
 Review comment:
   i think in case of non txn table ..if the location is changed then rename 
..if its txn table  then delete the directory in replication flow. For normal 
flow, txn table, control  should not come till here ... it should fail in hive 
server it self 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268126548
 
 

 ##
 File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ##
 @@ -357,6 +368,13 @@ public void alterTable(RawStore msdb, Warehouse wh, 
String catName, String dbnam
 }
   }
 
+  // If data location is changed in replication flow, then need to delete 
the old path.
+  if (replDataLocationChanged) {
+Path deleteOldDataLoc = new Path(oldt.getSd().getLocation());
+boolean isAutoPurge = 
"true".equalsIgnoreCase(oldt.getParameters().get("auto.purge"));
+wh.deleteDir(deleteOldDataLoc, true, isAutoPurge, olddb);
 
 Review comment:
   how to rollback the delete directory if txn fails ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268130334
 
 

 ##
 File path: 
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/ReplConst.java
 ##
 @@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.common;
+
+/**
+ * A class that defines the constant strings used by the replication 
implementation.
+ */
+
+public class ReplConst {
 
 Review comment:
   instead of replConst ..some thing more generic like repl common or util 
should be used ..so that it can be used for other purpose also 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268127478
 
 

 ##
 File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
 ##
 @@ -673,6 +674,59 @@ public void 
retryIncBootstrapExternalTablesFromDifferentDumpWithoutCleanTablesCo
 ErrorMsg.REPL_BOOTSTRAP_LOAD_PATH_NOT_VALID.getErrorCode());
   }
 
+  @Test
+  public void dynamicallyConvertManagedToExternalTable() throws Throwable {
+List dumpWithClause = Collections.singletonList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname + 
"'='true'"
+);
+List loadWithClause = externalTableBasePathWithClause();
+
+WarehouseInstance.Tuple tupleBootstrapManagedTable = primary.run("use " + 
primaryDbName)
+.run("create table t1 (id int)")
+.run("insert into table t1 values (1)")
+.run("create table t2 (id int) partitioned by (key int)")
 
 Review comment:
   The case is for migration, so the test should be there for migration case. 
Alter of acid table to external table should be avoided 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

maheshk114 commented on a change in pull request #578: HIVE-21471: Replicating 
conversion of managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578#discussion_r268127055
 
 

 ##
 File path: 
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/ReplConst.java
 ##
 @@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.common;
+
+/**
+ * A class that defines the constant strings used by the replication 
implementation.
+ */
+
+public class ReplConst {
+
+  /**
+   * The constant that denotes the table data location is changed to different 
path. This indicates
+   * Metastore to update corresponding path in Partitions and also need to 
delete old path.
+   */
+  public static final String DATA_LOCATION_CHANGED = "DATA_LOCATION_CHANGED";
 
 Review comment:
   its used only for repl ..so name should suggest that 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift

[Ganesha Shreedhara (JIRA)]

Ganesha Shreedhara created HIVE-21492:
-

 Summary: VectorizedParquetRecordReader can't to read parquet file 
generated using thrift
 Key: HIVE-21492
 URL: https://issues.apache.org/jira/browse/HIVE-21492
 Project: Hive
  Issue Type: Bug
Reporter: Ganesha Shreedhara
Assignee: Ganesha Shreedhara


Taking an example of a parquet table having array of integers as below. 

 
{code:java}
CREATE EXTERNAL TABLE ( list_of_ints` array)
STORED AS PARQUET 
LOCATION '{location}';
{code}
 

Parquet file generated using hive will have schema for Type as below:

 
{code:java}
group list_of_ints (LIST) { repeated group bag { optional int32 
array;\n};\n}{code}
 

 

Parquet file generated using thrift may have schema for Type as below:

 
{code:java}
required group list_of_ints (LIST) { repeated int32 list_of_tuple}{code}
 

 

VectorizedParquetRecordReader handles only parquet file generated using hive. 
It throws the following exception when parquet file generated using thrift is 
read because of the changes done as part of 
[HIVE-18553|https://issues.apache.org/jira/browse/HIVE-18553] .
{code:java}
Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is 
not a group
 at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
 at 
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
 at 
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
 at 
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
 at 
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
 at 
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
 at 
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
 at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
 

 I have done a small change to handle the case where the child type of group 
type can be PrimitiveType.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21491) Update Reserved Word Documentation

[Shawn Weeks (JIRA)]

Shawn Weeks created HIVE-21491:
--

 Summary: Update Reserved Word Documentation
 Key: HIVE-21491
 URL: https://issues.apache.org/jira/browse/HIVE-21491
 Project: Hive
  Issue Type: Bug
Reporter: Shawn Weeks


HIVE-15173 added "dec" as an alias for "decimal" but it's not documented 
anywhere. Only way I found it was by looking at the Hive Lexer file. There may 
be others missing I haven't done a search against keywords yet.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21490) Remove unused duplicate code added in HIVE-20506

[Brock Noland (JIRA)]

Brock Noland created HIVE-21490:
---

 Summary: Remove unused duplicate code added in HIVE-20506
 Key: HIVE-21490
 URL: https://issues.apache.org/jira/browse/HIVE-21490
 Project: Hive
  Issue Type: Improvement
Reporter: Brock Noland
Assignee: Brock Noland


HIVE-20506 added a small amount of unused duplicate code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70256: HIVE-21480: Fixed flaky and broken test TestHiveMetaStore.testJDOPersistenceManagerCleanup

[Peter Vary via Review Board]

> On márc. 21, 2019, 10:18 de, Peter Vary wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
> > Lines 3153-3155 (patched)
> > 
> >
> > My concern here is that we testing a different case with the patch.
> > 
> > Before we tested that we open/use/close a client and do not have 
> > lingering object. After the patch we test that closing the client will 
> > remove the 1 object - which says that the getAllDatabases will result in 
> > exactly 1 object.
> > 
> > How could that happen that there are lingering objects when we create a 
> > new client? Is there a way to get rid of the lingering object somehow, and 
> > then test the original usecase?
> 
> Morio Ramdenbourg wrote:
> My knowledge on the PersistenceManager code isn't that great, but before 
> the new client is even created, the object count returned from 
> getJDOPersistenceManagerCacheSize() is 1. I believe this object comes from 
> when the HMS is initializing the database schema. There seems to be a 
> lingering object from that, at least when I run this test as a standalone 
> without the other tests in the class.
> 
> Morio Ramdenbourg wrote:
> Do you know of other ways I can clear / wait for the PMF cache to empty 
> itself?

Me neither :(
Maybe 
https://docs.oracle.com/cd/E13189_01/kodo/docs303/jdo-javadoc/javax/jdo/PersistenceManager.html#evictAll()
 this? If not then after marking them to evict and some waiting?


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70256/#review213878
---


On márc. 20, 2019, 6:43 du, Morio Ramdenbourg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70256/
> ---
> 
> (Updated márc. 20, 2019, 6:43 du)
> 
> 
> Review request for hive, Adam Holley, Karthik Manamcheri, Karen Coppage, 
> Peter Vary, and Vihang Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This test was not correctly counting the number of
> objects in the PersistenceManager cache before and after 
> HiveMetaStoreClient.close(). The
> getJDOPersistenceManagerCacheSize() internal helper method did not use
> the updated fields present in the metastore classes, and was
> consistently returning -1. Additionally, there was a chance to cause
> flakiness since the object count before and after close() could
> differ depending on lingering objects from previous
> tests or setup.
> 
> Modified the helper method to use the new
> fields, and fixed the flakiness on this test.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
>  77e0c98265e7b561f2eb39536e3251dd92e9cab0 
> 
> 
> Diff: https://reviews.apache.org/r/70256/diff/1/
> 
> 
> Testing
> ---
> 
> Unit tests run
> 
> 
> Thanks,
> 
> Morio Ramdenbourg
> 
>

[GitHub] [hive] TopGunViper commented on issue #577: HIVE-21485: Add flag to turn off fetching partition stats in DESCRIBE…

[GitBox]

TopGunViper commented on issue #577: HIVE-21485: Add flag to turn off fetching 
partition stats in DESCRIBE…
URL: https://github.com/apache/hive/pull/577#issuecomment-475568072
 
 
   Hi @ashutoshc  
   Could you please take a look at this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] TopGunViper commented on issue #577: HIVE-21485: Add flag to turn off fetching partition stats in DESCRIBE…

[GitBox]

TopGunViper commented on issue #577: HIVE-21485: Add flag to turn off fetching 
partition stats in DESCRIBE…
URL: https://github.com/apache/hive/pull/577#issuecomment-475567449
 
 
   > 1. Create a patch file (patch file is diff between the latest commit as 
available in master with your proposed changes)
   > 2. Upload the patch file to Jira card
   >3 click the submit patch button
   > 
   > The automatic pre built testing will be triggered at Jenkins and the 
results will be uploaded to Jira card upon build completion.
   > 
   > For more details check how to contribute to hive wiki
   
   Thanks，already upload the patch.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (HIVE-21489) explain command throw ClassCastException

[Ping Lu (JIRA)]

Ping Lu created HIVE-21489:
--

 Summary: explain command throw ClassCastException
 Key: HIVE-21489
 URL: https://issues.apache.org/jira/browse/HIVE-21489
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.3.4
Reporter: Ping Lu
Assignee: Ping Lu


I'm trying to run commands like explain select * from src in hive-2.3.4,but it 
falls with the ClassCastException: 
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer
Steps to reproduce:
1）hive.execution.engine is the default value mr
2）hive.security.authorization.enabled is set to true, and 
hive.security.authorization.manager is set to 
org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider
3）start hivecli to run command：explain select * from src
I debug the code and find the issue HIVE-18778 causing the above 
ClassCastException.If I set hive.in.test to true，the explain command can be 
successfully executed。
Now,I have one question,due to hive.in.test cann't be modified at runtime.how 
to run explain command with using default authorization in hive-2.3.4,



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [hive] sankarh opened a new pull request #578: HIVE-21471: Replicating conversion of managed to external table leaks HDFS files at target.

[GitBox]

sankarh opened a new pull request #578: HIVE-21471: Replicating conversion of 
managed to external table leaks HDFS files at target.
URL: https://github.com/apache/hive/pull/578
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [hive] Fokko commented on issue #576: HIVE-21488 Update Apache Parquet to 1.10.1

[GitBox]

Fokko commented on issue #576: HIVE-21488 Update Apache Parquet to 1.10.1
URL: https://github.com/apache/hive/pull/576#issuecomment-475532120
 
 
   I've opened up a ticket: https://jira.apache.org/jira/browse/HIVE-21488


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (HIVE-21488) Update Apache Parquet 1.10.1

[Fokko Driesprong (JIRA)]

Fokko Driesprong created HIVE-21488:
---

 Summary: Update Apache Parquet 1.10.1
 Key: HIVE-21488
 URL: https://issues.apache.org/jira/browse/HIVE-21488
 Project: Hive
  Issue Type: Improvement
Reporter: Fokko Driesprong
Assignee: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 70270: HIVE-21465 Introduce a base abstract VectorExpression to eliminate further code duplications

[Laszlo Bodor]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70270/
---

Review request for hive.


Repository: hive-git


Description
---

HIVE-21465 Introduce a base abstract VectorExpression to eliminate further code 
duplications


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/BaseVectorExpression.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastDoubleToDecimal.java
 eb925af64c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CastLongToDecimal.java
 fa88e3ffb0 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/ConvertDecimal64ToDecimal.java
 04a2cba62a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FuncDecimalToDouble.java
 09215509de 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FuncDoubleToDecimal.java
 7fc8f6ab40 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FuncLongToDecimal.java
 d3136c3ed6 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FuncLongToString.java
 5805afcf23 


Diff: https://reviews.apache.org/r/70270/diff/1/


Testing
---

vectorization unit + qtests passed


Thanks,

Laszlo Bodor

[GitHub] [hive] rmsmani commented on issue #576: FLINK-11992 Update Apache Parquet to 1.10.1

[GitBox]

rmsmani commented on issue #576: FLINK-11992 Update Apache Parquet to 1.10.1
URL: https://github.com/apache/hive/pull/576#issuecomment-475505501
 
 
   Hi @Fokko 
   What's the hive Jira number for this...
   If Jira is not available, then please create Jira card, 
   1. Create git diff patch
   2. Upload the patch to Jira 
   3. Then submit patch


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services