[jira] [Updated] (HIVE-22090) Upgrade jetty to 9.3.27
[ https://issues.apache.org/jira/browse/HIVE-22090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22090: Status: Patch Available (was: Open) > Upgrade jetty to 9.3.27 > --- > > Key: HIVE-22090 > URL: https://issues.apache.org/jira/browse/HIVE-22090 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > Attachments: HIVE-22090.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22090) Upgrade jetty to 9.3.27
[ https://issues.apache.org/jira/browse/HIVE-22090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22090: Attachment: HIVE-22090.patch > Upgrade jetty to 9.3.27 > --- > > Key: HIVE-22090 > URL: https://issues.apache.org/jira/browse/HIVE-22090 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > Attachments: HIVE-22090.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-22090) Upgrade jetty to 9.3.27
[ https://issues.apache.org/jira/browse/HIVE-22090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reassigned HIVE-22090: --- > Upgrade jetty to 9.3.27 > --- > > Key: HIVE-22090 > URL: https://issues.apache.org/jira/browse/HIVE-22090 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22089) Upgrade jackson to 2.9.9
[ https://issues.apache.org/jira/browse/HIVE-22089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22089: Attachment: HIVE-22089.patch > Upgrade jackson to 2.9.9 > > > Key: HIVE-22089 > URL: https://issues.apache.org/jira/browse/HIVE-22089 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0, 3.1.0, 3.1.1 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > Attachments: HIVE-22089.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22089) Upgrade jackson to 2.9.9
[ https://issues.apache.org/jira/browse/HIVE-22089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22089: Status: Patch Available (was: Open) > Upgrade jackson to 2.9.9 > > > Key: HIVE-22089 > URL: https://issues.apache.org/jira/browse/HIVE-22089 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.1, 3.1.0, 3.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > Attachments: HIVE-22089.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-22089) Upgrade jackson to 2.9.9
[ https://issues.apache.org/jira/browse/HIVE-22089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reassigned HIVE-22089: --- > Upgrade jackson to 2.9.9 > > > Key: HIVE-22089 > URL: https://issues.apache.org/jira/browse/HIVE-22089 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.1, 3.1.0, 3.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22002) Insert into table partition fails partially with stats.autogather is on.
[
https://issues.apache.org/jira/browse/HIVE-22002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bencao updated HIVE-22002:
--
Attachment: HIVE-22002.patch
Status: Patch Available (was: Open)
> Insert into table partition fails partially with stats.autogather is on.
>
>
> Key: HIVE-22002
> URL: https://issues.apache.org/jira/browse/HIVE-22002
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: bencao
>Priority: Major
> Attachments: HIVE-22002.patch, image-2019-07-31-20-02-38-069.png
>
>
> create table test_double(id int) partitioned by (dbtest double);
> insert into test_double partition(dbtest) values (1,9.9); --> this works
> insert into test_double partition(dbtest) values (1,10); --> this fails
> But if we change it to
> insert into test_double partition(dbtest) values (1, cast (10 as double)); it
> succeeds
> -> the problem is only seen when trying to insert a whole number i.e. 10,
> 10.0, 15, 14.0 etc. The issue is not seen when inserting a number with
> decimal values other than 0. So insert of 10.1 goes though.
> The underlying from the HMS is
> {code}
> 2019-07-11T07:58:16,670 [pool-6-thread-196]: server.TThreadPoolServer
> (TThreadPoolServer.java:run(297)) - occurred during processing of message.
> java.lang.IndexOutOfBoundsException: Index: 0 at
> java.util.Collections$EmptyList.get(Collections.java:4454) ~[?:1.8.0_112] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:7808)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7769)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> {code}
> With {{hive.stats.column.autogather=false}}, this exception does not occur
> with or without the explicit casting.
> The issue stems from the fact that HS2 created a partition with value
> {{dbtest=10}} for the table and the stats processor is attempting to add
> column statistics for partition with value {{dbtest=10.0}}. Thus HMS
> {{getPartitionsByNames}} cannot find the partition with that value and thus
> fails to insert the stats. So while the failure initiates on HMS side, the
> cause in the HS2 query planning.
> It makes sense that turning off {{hive.stats.column.autogather}} resolves the
> issue because there is no StatsTask in a query plan.
> But {{SHOW PARTITIONS}} shows the partition as created while the query
> planner is not including it any plan because of the absence of stats on the
> partition.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22002) Insert into table partition fails partially with stats.autogather is on.
[
https://issues.apache.org/jira/browse/HIVE-22002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bencao updated HIVE-22002:
--
Attachment: (was: HIVE-22002.patch)
> Insert into table partition fails partially with stats.autogather is on.
>
>
> Key: HIVE-22002
> URL: https://issues.apache.org/jira/browse/HIVE-22002
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: bencao
>Priority: Major
> Attachments: image-2019-07-31-20-02-38-069.png
>
>
> create table test_double(id int) partitioned by (dbtest double);
> insert into test_double partition(dbtest) values (1,9.9); --> this works
> insert into test_double partition(dbtest) values (1,10); --> this fails
> But if we change it to
> insert into test_double partition(dbtest) values (1, cast (10 as double)); it
> succeeds
> -> the problem is only seen when trying to insert a whole number i.e. 10,
> 10.0, 15, 14.0 etc. The issue is not seen when inserting a number with
> decimal values other than 0. So insert of 10.1 goes though.
> The underlying from the HMS is
> {code}
> 2019-07-11T07:58:16,670 [pool-6-thread-196]: server.TThreadPoolServer
> (TThreadPoolServer.java:run(297)) - occurred during processing of message.
> java.lang.IndexOutOfBoundsException: Index: 0 at
> java.util.Collections$EmptyList.get(Collections.java:4454) ~[?:1.8.0_112] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:7808)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7769)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> {code}
> With {{hive.stats.column.autogather=false}}, this exception does not occur
> with or without the explicit casting.
> The issue stems from the fact that HS2 created a partition with value
> {{dbtest=10}} for the table and the stats processor is attempting to add
> column statistics for partition with value {{dbtest=10.0}}. Thus HMS
> {{getPartitionsByNames}} cannot find the partition with that value and thus
> fails to insert the stats. So while the failure initiates on HMS side, the
> cause in the HS2 query planning.
> It makes sense that turning off {{hive.stats.column.autogather}} resolves the
> issue because there is no StatsTask in a query plan.
> But {{SHOW PARTITIONS}} shows the partition as created while the query
> planner is not including it any plan because of the absence of stats on the
> partition.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work logged] (HIVE-22074) Slow compilation due to IN to OR transformation
[
https://issues.apache.org/jira/browse/HIVE-22074?focusedWorklogId=290872=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-290872
]
ASF GitHub Bot logged work on HIVE-22074:
-
Author: ASF GitHub Bot
Created on: 07/Aug/19 23:43
Start Date: 07/Aug/19 23:43
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #746: HIVE-22074:
Slow compilation due to IN to OR transformation
URL: https://github.com/apache/hive/pull/746#discussion_r311806267
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
##
@@ -1220,16 +1220,26 @@ protected ExprNodeDesc
getXpathOrFuncExprNodeDesc(ASTNode expr,
}
outputOpList.add(nullConst);
}
+
if (!ctx.isCBOExecuted()) {
-ArrayList orOperands =
TypeCheckProcFactoryUtils.rewriteInToOR(children);
-if (orOperands != null) {
- if (orOperands.size() == 1) {
-orOperands.add(new
ExprNodeConstantDesc(TypeInfoFactory.booleanTypeInfo, false));
+
+HiveConf conf;
+try {
+ conf = Hive.get().getConf();
Review comment:
I think it is better to pass this value from the callers in the context. You
would not need to change all callers; if value is not passed, rewriting could
be skipped. I see mainly two advantages of doing this:
1) if transformation is never happening, we will not be retrieving the conf
and this value for every IN clause in a query (note that `isCBOExecuted` method
is misleading, the value returned is `foldExpr` boolean which is `false`
sometimes even for calls coming from CBO cf. first line in `genFilterRelNode`
method in `CalcitePlanner`), and
2) removing the static call to Hive object from within the folding logic.
I see there are other calls to `Hive.get()` in the class, that information
should probably be moved to context too.
These can all be tackled together in a follow-up, but I think since we are
cleaning up this logic, it would make sense to do it at some point.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 290872)
Time Spent: 50m (was: 40m)
> Slow compilation due to IN to OR transformation
> ---
>
> Key: HIVE-22074
> URL: https://issues.apache.org/jira/browse/HIVE-22074
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22074.1.patch, HIVE-22074.2.patch,
> HIVE-22074.3.patch, HIVE-22074.4.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Currently Hive transform IN expressions to OR to apply various CBO rules.
> This incur significant performance hit if IN consist of large number of
> expressions.
> It is better to not transform IN expressions to OR in such cases because
> overall benefit of various optimizations/transformations is unrealized due to
> the compilation overhead
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work logged] (HIVE-22074) Slow compilation due to IN to OR transformation
[
https://issues.apache.org/jira/browse/HIVE-22074?focusedWorklogId=290873=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-290873
]
ASF GitHub Bot logged work on HIVE-22074:
-
Author: ASF GitHub Bot
Created on: 07/Aug/19 23:43
Start Date: 07/Aug/19 23:43
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #746: HIVE-22074:
Slow compilation due to IN to OR transformation
URL: https://github.com/apache/hive/pull/746#discussion_r311806267
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
##
@@ -1220,16 +1220,26 @@ protected ExprNodeDesc
getXpathOrFuncExprNodeDesc(ASTNode expr,
}
outputOpList.add(nullConst);
}
+
if (!ctx.isCBOExecuted()) {
-ArrayList orOperands =
TypeCheckProcFactoryUtils.rewriteInToOR(children);
-if (orOperands != null) {
- if (orOperands.size() == 1) {
-orOperands.add(new
ExprNodeConstantDesc(TypeInfoFactory.booleanTypeInfo, false));
+
+HiveConf conf;
+try {
+ conf = Hive.get().getConf();
Review comment:
I think it is better to pass this value from the callers in the context. You
would not need to change all callers; if value is not passed, rewriting could
be skipped. I see mainly two advantages of doing this:
1) if transformation is never happening, we will not be retrieving the conf
and this value for every IN clause in a query (note that `isCBOExecuted` method
is misleading, the value returned is `foldExpr` boolean which is `false`
sometimes even for calls coming from CBO cf. first line in `genFilterRelNode`
method in `CalcitePlanner`), and
2) removing the static call to Hive object from within the folding logic.
I see there are other calls to `Hive.get()` in the class, that information
should probably be moved to context too.
These can all be tackled together in a follow-up, but I think since we are
cleaning up this logic, it would make sense to do it at some point.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 290873)
Time Spent: 1h (was: 50m)
> Slow compilation due to IN to OR transformation
> ---
>
> Key: HIVE-22074
> URL: https://issues.apache.org/jira/browse/HIVE-22074
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22074.1.patch, HIVE-22074.2.patch,
> HIVE-22074.3.patch, HIVE-22074.4.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently Hive transform IN expressions to OR to apply various CBO rules.
> This incur significant performance hit if IN consist of large number of
> expressions.
> It is better to not transform IN expressions to OR in such cases because
> overall benefit of various optimizations/transformations is unrealized due to
> the compilation overhead
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22088) Dynamic partition insert problem on external table with "=" in location path spec
[
https://issues.apache.org/jira/browse/HIVE-22088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Puneet Khatod updated HIVE-22088:
-
Environment:
Hive 2.6.0.10-2 Executing on Tez.
OS: Ubuntu 16.04.4 LTS
Config settings used:
SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
was:
Hive 2.6.0.10-2 Executing on Tez.
Config settings used:
SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
> Dynamic partition insert problem on external table with "=" in location path
> spec
> -
>
> Key: HIVE-22088
> URL: https://issues.apache.org/jira/browse/HIVE-22088
> Project: Hive
> Issue Type: Bug
> Environment: Hive 2.6.0.10-2 Executing on Tez.
> OS: Ubuntu 16.04.4 LTS
> Config settings used:
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
>Reporter: Puneet Khatod
>Priority: Major
>
> If external table location spec has a '=' sign (coincidentally partition
> specifier) in it, then dynamic partition loading fails.
> *Use cases:*
> Quite often the same data is used in different contexts by creating different
> external tables on top of the data. Many times the tables have different
> partition depths depending on how data is organized.
> Like in below example, there are individual customer specific tables and
> queries/jobs to insert data partitioned by type. And there is another table
> to give the consolidated data view of all the customers, thus have two level
> partition customer and type.
> The job to insert customer specific data into customer specific table fails
> if we use dynamic partitioning. Static partition insert on same table works
> fine though.
> *Replication:*
> To replicate following simple setup could be done. Below execution is on
> 'Tez'.
> *Source table**-*
> CREATE EXTERNAL TABLE temp_dummy_table
> (id STRING, type STRING)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/home/source/';
>
> *Destination Table-*
> CREATE EXTERNAL TABLE temp_dummy_dest_table
> (id STRING)
> PARTITIONED BY (type string)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/home/destination/{color:#ff}customer=abc{color}/';
>
> *Insert into destination-*
> insert overwrite table temp_dummy_dest_table partition (type)
> select i.id as id, i.type as type
> from temp_dummy_table i
> where i.type in ('type1','type2');
>
> *Log and Error Msgs on CLI*-
> Loading data to table temp_dummy_dest_table partition (type=null)
> Failed with exception Partition spec \{type=type1, customer=abc} contains
> non-partition columns
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
>
> *Possible resolution:*
> The dynamic partitioning should consider only those partition specs which are
> under the defined table root/base path. If the path itself has partition
> style format (customer=abc in above example) then that should not be
> considered as partition as it is outside the scope of the table.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22088) Dynamic partition insert problem on external table with "=" in location path spec
[
https://issues.apache.org/jira/browse/HIVE-22088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Puneet Khatod updated HIVE-22088:
-
Description:
If external table location spec has a '=' sign (coincidentally partition
specifier) in it, then dynamic partition loading fails.
*Use cases:*
Quite often the same data is used in different contexts by creating different
external tables on top of the data. Many times the tables have different
partition depths depending on how data is organized.
Like in below example, there are individual customer specific tables and
queries/jobs to insert data partitioned by type. And there is another table to
give the consolidated data view of all the customers, thus have two level
partition customer and type.
The job to insert customer specific data into customer specific table fails if
we use dynamic partitioning. Static partition insert on same table works fine
though.
*Replication:*
To replicate following simple setup could be done. Below execution is on 'Tez'.
*Source table**-*
CREATE EXTERNAL TABLE temp_dummy_table
(id STRING, type STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/source/';
*Destination Table-*
CREATE EXTERNAL TABLE temp_dummy_dest_table
(id STRING)
PARTITIONED BY (type string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/destination/{color:#ff}customer=abc{color}/';
*Insert into destination-*
insert overwrite table temp_dummy_dest_table partition (type)
select i.id as id, i.type as type
from temp_dummy_table i
where i.type in ('type1','type2');
*Log and Error Msgs on CLI*-
Loading data to table temp_dummy_dest_table partition (type=null)
Failed with exception Partition spec \{type=type1, customer=abc} contains
non-partition columns
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
*Possible resolution:*
The dynamic partitioning should consider only those partition specs which are
under the defined table root/base path. If the path itself has partition style
format (customer=abc in above example) then that should not be considered as
partition as it is outside the scope of the table.
was:
If external table location spec has a '=' sign (coincidentally partition
specifier) in it, then dynamic partition loading fails.
*Use cases:*
Quite often the same data is used in different contexts by creating different
external tables on top of the data. Many times the tables have different
partition depths depending on how data is organized. Static partition insert on
external table with '=' (partition specifier) in path works fine but dynamic
partition fails.
*Replication:*
To replicate following simple setup could be done. Below execution is on 'Tez'.
*Source table**-*
CREATE EXTERNAL TABLE temp_dummy_table
(id STRING, type STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/source/';
*Destination Table-*
CREATE EXTERNAL TABLE temp_dummy_dest_table
(id STRING)
PARTITIONED BY (type string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/destination/{color:#ff}customer=abc{color}/';
*Insert into destination-*
insert overwrite table temp_dummy_dest_table partition (type)
select i.id as id, i.type as type
from temp_dummy_table i
where i.type in ('type1','type2');
*Log and Error Msgs on CLI*-
Loading data to table temp_dummy_dest_table partition (type=null)
Failed with exception Partition spec \{type=type1, customer=abc} contains
non-partition columns
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
*Possible resolution:*
The dynamic partitioning should consider only those partition specs which are
under the defined table root/base path. If the path itself has partition style
format (customer=abc in above example) then that should not be considered as
partition as it is outside the scope of the table.
> Dynamic partition insert problem on external table with "=" in location path
> spec
> -
>
> Key: HIVE-22088
> URL: https://issues.apache.org/jira/browse/HIVE-22088
> Project: Hive
> Issue Type: Bug
> Environment: Hive 2.6.0.10-2 Executing on Tez.
> Config settings used:
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
>Reporter: Puneet Khatod
>Priority: Major
>
> If external table location spec has a '=' sign (coincidentally partition
> specifier) in it, then dynamic partition loading fails.
> *Use cases:*
> Quite often the same data is used in different contexts by creating
[jira] [Updated] (HIVE-22088) Dynamic partition insert problem on external table with "=" in location path spec
[
https://issues.apache.org/jira/browse/HIVE-22088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Puneet Khatod updated HIVE-22088:
-
Description:
If external table location spec has a '=' sign (coincidentally partition
specifier) in it, then dynamic partition loading fails.
*Use cases:*
Quite often the same data is used in different contexts by creating different
external tables on top of the data. Many times the tables have different
partition depths depending on how data is organized. Static partition insert on
external table with '=' (partition specifier) in path works fine but dynamic
partition fails.
*Replication:*
To replicate following simple setup could be done. Below execution is on 'Tez'.
*Source table**-*
CREATE EXTERNAL TABLE temp_dummy_table
(id STRING, type STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/source/';
*Destination Table-*
CREATE EXTERNAL TABLE temp_dummy_dest_table
(id STRING)
PARTITIONED BY (type string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/destination/{color:#ff}customer=abc{color}/';
*Insert into destination-*
insert overwrite table temp_dummy_dest_table partition (type)
select i.id as id, i.type as type
from temp_dummy_table i
where i.type in ('type1','type2');
*Log and Error Msgs on CLI*-
Loading data to table temp_dummy_dest_table partition (type=null)
Failed with exception Partition spec \{type=type1, customer=abc} contains
non-partition columns
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
*Possible resolution:*
The dynamic partitioning should consider only those partition specs which are
under the defined table root/base path. If the path itself has partition style
format (customer=abc in above example) then that should not be considered as
partition as it is outside the scope of the table.
was:
If external table path has a '=' sign (coincidentally partition specifier) in
it, then dynamic partition loading fails.
*Use cases:*
Quite often the same data is used in different contexts by creating different
external tables on top of the data. Many times the tables have different
partition depths depending on how data is organized. Static partition insert on
external table with '=' (partition specifier) in path works fine but dynamic
partition fails.
*Replication:*
To replicate following simple setup could be done. Below execution is on 'Tez'.
*Source table**-*
CREATE EXTERNAL TABLE temp_dummy_table
(id STRING, type STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/source/';
*Destination Table-*
CREATE EXTERNAL TABLE temp_dummy_dest_table
(id STRING)
PARTITIONED BY (type string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/destination/{color:#ff}customer=abc{color}/';
*Insert into destination-*
insert overwrite table temp_dummy_dest_table partition (type)
select i.id as id, i.type as type
from temp_dummy_table i
where i.type in ('type1','type2');
*Log and Error Msgs on CLI*-
Loading data to table temp_dummy_dest_table partition (type=null)
Failed with exception Partition spec \{type=type1, customer=abc} contains
non-partition columns
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
*Possible resolution:*
The dynamic partitioning should consider only those partition specs which are
under the defined table root/base path. If the path itself has partition style
format (customer=abc in above example) then that should not be considered as
partition as it is outside the scope of the table.
> Dynamic partition insert problem on external table with "=" in location path
> spec
> -
>
> Key: HIVE-22088
> URL: https://issues.apache.org/jira/browse/HIVE-22088
> Project: Hive
> Issue Type: Bug
> Environment: Hive 2.6.0.10-2 Executing on Tez.
> Config settings used:
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
>Reporter: Puneet Khatod
>Priority: Major
>
> If external table location spec has a '=' sign (coincidentally partition
> specifier) in it, then dynamic partition loading fails.
> *Use cases:*
> Quite often the same data is used in different contexts by creating different
> external tables on top of the data. Many times the tables have different
> partition depths depending on how data is organized. Static partition insert
> on external table with '=' (partition specifier) in path works fine but
> dynamic partition fails.
> *Replication:*
> To replicate following
[jira] [Updated] (HIVE-22088) Dynamic partition insert problem on external table with "=" in path
[
https://issues.apache.org/jira/browse/HIVE-22088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Puneet Khatod updated HIVE-22088:
-
Description:
If external table path has a '=' sign (coincidentally partition specifier) in
it, then dynamic partition loading fails.
*Use cases:*
Quite often the same data is used in different contexts by creating different
external tables on top of the data. Many times the tables have different
partition depths depending on how data is organized. Static partition insert on
external table with '=' (partition specifier) in path works fine but dynamic
partition fails.
*Replication:*
To replicate following simple setup could be done. Below execution is on 'Tez'.
*Source table**-*
CREATE EXTERNAL TABLE temp_dummy_table
(id STRING, type STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/source/';
*Destination Table-*
CREATE EXTERNAL TABLE temp_dummy_dest_table
(id STRING)
PARTITIONED BY (type string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/destination/{color:#ff}customer=abc{color}/';
*Insert into destination-*
insert overwrite table temp_dummy_dest_table partition (type)
select i.id as id, i.type as type
from temp_dummy_table i
where i.type in ('type1','type2');
*Log and Error Msgs on CLI*-
Loading data to table temp_dummy_dest_table partition (type=null)
Failed with exception Partition spec \{type=type1, customer=abc} contains
non-partition columns
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
*Possible resolution:*
The dynamic partitioning should consider only those partition specs which are
under the defined table root/base path. If the path itself has partition style
format (customer=abc in above example) then that should not be considered as
partition as it is outside the scope of the table.
was:
If external table path has a '=' sign (coincidentally partition specifier) in
it, then dynamic partition loading fails.
*Use cases:*
Quite often the same data is used in different contexts by creating different
external tables on top of the data. Many times the tables have different
partition depths depending on how data is organized. Static partition insert on
external table with '=' (partition specifier) in path works fine but dynamic
partition fails.
*Replication:*
To replicate following simple setup could be done. Below execution is on 'Tez'.
*Source table**-*
CREATE EXTERNAL TABLE temp_dummy_table
(id STRING, type STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/source/';
*Destination Table-*
CREATE EXTERNAL TABLE temp_dummy_dest_table
(id STRING)
PARTITIONED BY (type string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/destination/{color:#FF}customer=abc{color}/';
*Insert into destination-*
insert overwrite table temp_dummy_dest_table partition (type)
select i.id as id, i.type as type
from temp_dummy_table i
where i.type in ('type1','type2');
*Log and Error Msgs on CLI*-
Loading data to table temp_dummy_dest_table partition (type=null)
Failed with exception Partition spec \{type=type1, customer=abc} contains
non-partition columns
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
*Possible resolution:*
The dynamic partitioning should consider only those partition specs which are
under teh defined table root path. If the path itself has partition style
format (customer=abc in above example) then that should not be considered as
partition as it is outside the scope of the table.
> Dynamic partition insert problem on external table with "=" in path
> ---
>
> Key: HIVE-22088
> URL: https://issues.apache.org/jira/browse/HIVE-22088
> Project: Hive
> Issue Type: Bug
> Environment: Hive 2.6.0.10-2 Executing on Tez.
> Config settings used:
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
>Reporter: Puneet Khatod
>Priority: Major
>
> If external table path has a '=' sign (coincidentally partition specifier) in
> it, then dynamic partition loading fails.
> *Use cases:*
> Quite often the same data is used in different contexts by creating different
> external tables on top of the data. Many times the tables have different
> partition depths depending on how data is organized. Static partition insert
> on external table with '=' (partition specifier) in path works fine but
> dynamic partition fails.
> *Replication:*
> To replicate following simple setup could be done. Below execution is on
> 'Tez'.
[jira] [Updated] (HIVE-22088) Dynamic partition insert problem on external table with "=" in location path spec
[
https://issues.apache.org/jira/browse/HIVE-22088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Puneet Khatod updated HIVE-22088:
-
Summary: Dynamic partition insert problem on external table with "=" in
location path spec (was: Dynamic partition insert problem on external table
with "=" in path)
> Dynamic partition insert problem on external table with "=" in location path
> spec
> -
>
> Key: HIVE-22088
> URL: https://issues.apache.org/jira/browse/HIVE-22088
> Project: Hive
> Issue Type: Bug
> Environment: Hive 2.6.0.10-2 Executing on Tez.
> Config settings used:
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
>Reporter: Puneet Khatod
>Priority: Major
>
> If external table path has a '=' sign (coincidentally partition specifier) in
> it, then dynamic partition loading fails.
> *Use cases:*
> Quite often the same data is used in different contexts by creating different
> external tables on top of the data. Many times the tables have different
> partition depths depending on how data is organized. Static partition insert
> on external table with '=' (partition specifier) in path works fine but
> dynamic partition fails.
> *Replication:*
> To replicate following simple setup could be done. Below execution is on
> 'Tez'.
> *Source table**-*
> CREATE EXTERNAL TABLE temp_dummy_table
> (id STRING, type STRING)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/home/source/';
>
> *Destination Table-*
> CREATE EXTERNAL TABLE temp_dummy_dest_table
> (id STRING)
> PARTITIONED BY (type string)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/home/destination/{color:#ff}customer=abc{color}/';
>
> *Insert into destination-*
> insert overwrite table temp_dummy_dest_table partition (type)
> select i.id as id, i.type as type
> from temp_dummy_table i
> where i.type in ('type1','type2');
>
> *Log and Error Msgs on CLI*-
> Loading data to table temp_dummy_dest_table partition (type=null)
> Failed with exception Partition spec \{type=type1, customer=abc} contains
> non-partition columns
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
>
> *Possible resolution:*
> The dynamic partitioning should consider only those partition specs which are
> under the defined table root/base path. If the path itself has partition
> style format (customer=abc in above example) then that should not be
> considered as partition as it is outside the scope of the table.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22088) Dynamic partition insert problem on external table with "=" in path
[
https://issues.apache.org/jira/browse/HIVE-22088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Puneet Khatod updated HIVE-22088:
-
Description:
If external table path has a '=' sign (coincidentally partition specifier) in
it, then dynamic partition loading fails.
*Use cases:*
Quite often the same data is used in different contexts by creating different
external tables on top of the data. Many times the tables have different
partition depths depending on how data is organized. Static partition insert on
external table with '=' (partition specifier) in path works fine but dynamic
partition fails.
*Replication:*
To replicate following simple setup could be done. Below execution is on 'Tez'.
*Source table**-*
CREATE EXTERNAL TABLE temp_dummy_table
(id STRING, type STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/source/';
*Destination Table-*
CREATE EXTERNAL TABLE temp_dummy_dest_table
(id STRING)
PARTITIONED BY (type string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/destination/{color:#FF}customer=abc{color}/';
*Insert into destination-*
insert overwrite table temp_dummy_dest_table partition (type)
select i.id as id, i.type as type
from temp_dummy_table i
where i.type in ('type1','type2');
*Log and Error Msgs on CLI*-
Loading data to table temp_dummy_dest_table partition (type=null)
Failed with exception Partition spec \{type=type1, customer=abc} contains
non-partition columns
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
*Possible resolution:*
The dynamic partitioning should consider only those partition specs which are
under teh defined table root path. If the path itself has partition style
format (customer=abc in above example) then that should not be considered as
partition as it is outside the scope of the table.
was:
If external table path has a '=' sign (coincidentally partition specifier) in
it, then dynamic partition loading fails.
*Use cases:*
Quite often the same data is used in different contexts by creating different
external tables on top of the data. Many times the tables have different
partition depths depending on how data is organized. Static partition insert on
external table with '=' (partition specifier) in path works fine but dynamic
partition fails.
*Replication:*
To replicate following simple setup could be done. Below execution is on 'Tez'.
*Source table**-*
CREATE EXTERNAL TABLE temp_dummy_table
(id STRING, type STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/source/';
*Destination Table-*
CREATE EXTERNAL TABLE temp_dummy_dest_table
(id STRING)
PARTITIONED BY (type string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED
AS TEXTFILE
LOCATION '/home/destination/customer=abc/';
*Insert into destination-*
insert overwrite table temp_dummy_dest_table partition (type)
select i.id as id, i.type as type
from temp_dummy_table i
where i.type in ('type1','type2');
*Log and Error Msgs on CLI*-
Loading data to table temp_dummy_dest_table partition (type=null)
Failed with exception Partition spec \{type=type1, customer=abc} contains
non-partition columns
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
*Possible resolution:*
The dynamic partitioning should consider only those partition specs which are
under teh defined table root path. If the path itself has partition style
format (customer=abc in above example) then that should not be considered as
partition as it is outside the scope of the table.
> Dynamic partition insert problem on external table with "=" in path
> ---
>
> Key: HIVE-22088
> URL: https://issues.apache.org/jira/browse/HIVE-22088
> Project: Hive
> Issue Type: Bug
> Environment: Hive 2.6.0.10-2 Executing on Tez.
> Config settings used:
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
>Reporter: Puneet Khatod
>Priority: Major
>
> If external table path has a '=' sign (coincidentally partition specifier) in
> it, then dynamic partition loading fails.
> *Use cases:*
> Quite often the same data is used in different contexts by creating different
> external tables on top of the data. Many times the tables have different
> partition depths depending on how data is organized. Static partition insert
> on external table with '=' (partition specifier) in path works fine but
> dynamic partition fails.
> *Replication:*
> To replicate following simple setup could be done. Below execution is on
> 'Tez'.
> *Source table**-*
> CREATE
[jira] [Work logged] (HIVE-22074) Slow compilation due to IN to OR transformation
[
https://issues.apache.org/jira/browse/HIVE-22074?focusedWorklogId=290787=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-290787
]
ASF GitHub Bot logged work on HIVE-22074:
-
Author: ASF GitHub Bot
Created on: 07/Aug/19 21:34
Start Date: 07/Aug/19 21:34
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #746:
HIVE-22074: Slow compilation due to IN to OR transformation
URL: https://github.com/apache/hive/pull/746#discussion_r311773334
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
##
@@ -1220,16 +1220,26 @@ protected ExprNodeDesc
getXpathOrFuncExprNodeDesc(ASTNode expr,
}
outputOpList.add(nullConst);
}
+
if (!ctx.isCBOExecuted()) {
-ArrayList orOperands =
TypeCheckProcFactoryUtils.rewriteInToOR(children);
-if (orOperands != null) {
- if (orOperands.size() == 1) {
-orOperands.add(new
ExprNodeConstantDesc(TypeInfoFactory.booleanTypeInfo, false));
+
+HiveConf conf;
+try {
+ conf = Hive.get().getConf();
Review comment:
@jcamachor `ctx` is passed on to `genExprNode` by the callers. There are
several callers which uses `genExprNode`. For this we will have to update all
of these callers to set the config in `TypeCheckCtx`.
Fetching HiveConf only occurs if the expression at hand is `GenericUDFIn`
and CBO wasn't executed therefore this shouldn't add much overhead.
Let me know if you think otherwise
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 290787)
Time Spent: 40m (was: 0.5h)
> Slow compilation due to IN to OR transformation
> ---
>
> Key: HIVE-22074
> URL: https://issues.apache.org/jira/browse/HIVE-22074
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22074.1.patch, HIVE-22074.2.patch,
> HIVE-22074.3.patch, HIVE-22074.4.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently Hive transform IN expressions to OR to apply various CBO rules.
> This incur significant performance hit if IN consist of large number of
> expressions.
> It is better to not transform IN expressions to OR in such cases because
> overall benefit of various optimizations/transformations is unrealized due to
> the compilation overhead
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902476#comment-16902476
]
Vineet Garg commented on HIVE-21550:
These tests are still failing albeit with NullPointerException now
https://builds.apache.org/job/PreCommit-HIVE-Build/18277/testReport/
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hive.metastore.TestObjectStore.setUp(TestObjectStore.java:147)
{code}
Not sure if it is related to this change at all.
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21550.01.patch, HIVE-21550.02.patch,
> HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png,
> surefire_derby_stacktrace.log
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? where
> "DBS"."CTLG_NAME" = ? order by "PART_NAME" asc". at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:61)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1882)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:759)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2677)
>
[jira] [Updated] (HIVE-22080) Prevent implicit conversion from String/char/varchar to double/decimal
[ https://issues.apache.org/jira/browse/HIVE-22080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-22080: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master > Prevent implicit conversion from String/char/varchar to double/decimal > -- > > Key: HIVE-22080 > URL: https://issues.apache.org/jira/browse/HIVE-22080 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22080.1.patch, HIVE-22080.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Implicit conversion from String family types to any non-string family types > are invalid. User can force the conversion by turning off the setting > hive.metastore.disallow.incompatible.col.type.changes. If not turned off, > such a conversion should throw error. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-22087) HMS Translation: Translate getDatabase() API to alter warehouse location
[ https://issues.apache.org/jira/browse/HIVE-22087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam reassigned HIVE-22087: > HMS Translation: Translate getDatabase() API to alter warehouse location > > > Key: HIVE-22087 > URL: https://issues.apache.org/jira/browse/HIVE-22087 > Project: Hive > Issue Type: Sub-task >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > > It makes sense to translate getDatabase() calls as well, to alter the > location for the Database based on whether or not the processor has > capabilities to write to the managed warehouse directory. Every DB has 2 > locations, one external and the other in the managed warehouse directory. If > the processor has any AcidWrite capability, then the location remains > unchanged for the database. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902156#comment-16902156
]
Naveen Gangam commented on HIVE-21241:
--
Thanks [~belugabehr]. New patch looks good. I am good with leaving it at debug
if failures are to be expected. +1 for me.
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch, HIVE-21241.2.patch,
> HIVE-21241.3.patch, HIVE-21241.4.patch, HIVE-21241.5.patch, HIVE-21241.6.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21543) Use FilterHooks for show compactions
[
https://issues.apache.org/jira/browse/HIVE-21543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902091#comment-16902091
]
Hive QA commented on HIVE-21543:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976931/HIVE-21543.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16723 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18279/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18279/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18279/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976931 - PreCommit-HIVE-Build
> Use FilterHooks for show compactions
>
>
> Key: HIVE-21543
> URL: https://issues.apache.org/jira/browse/HIVE-21543
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21543.01.patch, HIVE-21543.02.patch
>
>
> Use FilterHooks for checking dbs/tables/partitions for showCompactions
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902082#comment-16902082
]
David Mollitor edited comment on HIVE-21241 at 8/7/19 1:59 PM:
---
[~ngangam] I implemented your requests. Code is also available for review on
Github.
I did not however change the DEBUG logging. It is expected that there are
going to be cases where the timestamp fails to parse. For example, a dirty
data set may have different timestamp formats in the data. Such a data set
would generate a lot of INFO logging here as the parser would try multiple
different formats until it got it right.
was (Author: belugabehr):
[~ngangam] I implemented your requests. Code is also available for review on
Github.
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch, HIVE-21241.2.patch,
> HIVE-21241.3.patch, HIVE-21241.4.patch, HIVE-21241.5.patch, HIVE-21241.6.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902082#comment-16902082
]
David Mollitor commented on HIVE-21241:
---
[~ngangam] I implemented your requests. Code is also available for review on
Github.
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch, HIVE-21241.2.patch,
> HIVE-21241.3.patch, HIVE-21241.4.patch, HIVE-21241.5.patch, HIVE-21241.6.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21543) Use FilterHooks for show compactions
[
https://issues.apache.org/jira/browse/HIVE-21543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902055#comment-16902055
]
Hive QA commented on HIVE-21543:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
25s{color} | {color:blue} standalone-metastore/metastore-common in master has
31 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
14s{color} | {color:blue} standalone-metastore/metastore-server in master has
180 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
41s{color} | {color:red} standalone-metastore/metastore-common generated 1 new
+ 31 unchanged - 0 fixed = 32 total (was 31) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-common |
| |
org.apache.hadoop.hive.metastore.utils.FilterUtils.filterCompactionsIfEnabled(boolean,
MetaStoreFilterHook, String, List) makes inefficient use of keySet iterator
instead of entrySet iterator At FilterUtils.java:inefficient use of keySet
iterator instead of entrySet iterator At FilterUtils.java:[line 423] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18279/dev-support/hive-personality.sh
|
| git revision | master / 333264b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18279/yetus/new-findbugs-standalone-metastore_metastore-common.html
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18279/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use FilterHooks for show compactions
>
>
> Key: HIVE-21543
> URL: https://issues.apache.org/jira/browse/HIVE-21543
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21543.01.patch, HIVE-21543.02.patch
>
>
> Use FilterHooks for checking dbs/tables/partitions for showCompactions
--
This
[jira] [Updated] (HIVE-22002) Insert into table partition fails partially with stats.autogather is on.
[
https://issues.apache.org/jira/browse/HIVE-22002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bencao updated HIVE-22002:
--
Attachment: HIVE-22002.patch
> Insert into table partition fails partially with stats.autogather is on.
>
>
> Key: HIVE-22002
> URL: https://issues.apache.org/jira/browse/HIVE-22002
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: bencao
>Priority: Major
> Attachments: HIVE-22002.patch, image-2019-07-31-20-02-38-069.png
>
>
> create table test_double(id int) partitioned by (dbtest double);
> insert into test_double partition(dbtest) values (1,9.9); --> this works
> insert into test_double partition(dbtest) values (1,10); --> this fails
> But if we change it to
> insert into test_double partition(dbtest) values (1, cast (10 as double)); it
> succeeds
> -> the problem is only seen when trying to insert a whole number i.e. 10,
> 10.0, 15, 14.0 etc. The issue is not seen when inserting a number with
> decimal values other than 0. So insert of 10.1 goes though.
> The underlying from the HMS is
> {code}
> 2019-07-11T07:58:16,670 [pool-6-thread-196]: server.TThreadPoolServer
> (TThreadPoolServer.java:run(297)) - occurred during processing of message.
> java.lang.IndexOutOfBoundsException: Index: 0 at
> java.util.Collections$EmptyList.get(Collections.java:4454) ~[?:1.8.0_112] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:7808)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7769)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> {code}
> With {{hive.stats.column.autogather=false}}, this exception does not occur
> with or without the explicit casting.
> The issue stems from the fact that HS2 created a partition with value
> {{dbtest=10}} for the table and the stats processor is attempting to add
> column statistics for partition with value {{dbtest=10.0}}. Thus HMS
> {{getPartitionsByNames}} cannot find the partition with that value and thus
> fails to insert the stats. So while the failure initiates on HMS side, the
> cause in the HS2 query planning.
> It makes sense that turning off {{hive.stats.column.autogather}} resolves the
> issue because there is no StatsTask in a query plan.
> But {{SHOW PARTITIONS}} shows the partition as created while the query
> planner is not including it any plan because of the absence of stats on the
> partition.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22002) Insert into table partition fails partially with stats.autogather is on.
[
https://issues.apache.org/jira/browse/HIVE-22002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902039#comment-16902039
]
bencao commented on HIVE-22002:
---
I found this:
When I execute sql :
{code:java}
Insert into test double partition(dbtest) values (1,10);{code}
The partition saved by the metabase is dbtest=10.
When executed:
{code:java}
Insert into test_double partition(dbtest) values (1, cast (10 as double));{code}
The partition saved by the metabase is dbtest=10.0.
When statstask is executed, when the partition information is assembled, it
will perform partition type conversion such as 10 will change 10.0.
This will result in the partition information not being found in the metabase,
because the partition saved by the metabase is Dbtest=10.
Therefore, I think it can be solved by doing this. When saving the partition
information, perform partition type conversion to ensure the partition value is
consistent.
Finally, I submitted the modified patch
> Insert into table partition fails partially with stats.autogather is on.
>
>
> Key: HIVE-22002
> URL: https://issues.apache.org/jira/browse/HIVE-22002
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: bencao
>Priority: Major
> Attachments: HIVE-22002.patch, image-2019-07-31-20-02-38-069.png
>
>
> create table test_double(id int) partitioned by (dbtest double);
> insert into test_double partition(dbtest) values (1,9.9); --> this works
> insert into test_double partition(dbtest) values (1,10); --> this fails
> But if we change it to
> insert into test_double partition(dbtest) values (1, cast (10 as double)); it
> succeeds
> -> the problem is only seen when trying to insert a whole number i.e. 10,
> 10.0, 15, 14.0 etc. The issue is not seen when inserting a number with
> decimal values other than 0. So insert of 10.1 goes though.
> The underlying from the HMS is
> {code}
> 2019-07-11T07:58:16,670 [pool-6-thread-196]: server.TThreadPoolServer
> (TThreadPoolServer.java:run(297)) - occurred during processing of message.
> java.lang.IndexOutOfBoundsException: Index: 0 at
> java.util.Collections$EmptyList.get(Collections.java:4454) ~[?:1.8.0_112] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:7808)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7769)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> {code}
> With {{hive.stats.column.autogather=false}}, this exception does not occur
> with or without the explicit casting.
> The issue stems from the fact that HS2 created a partition with value
> {{dbtest=10}} for the table and the stats processor is attempting to add
> column statistics for partition with value {{dbtest=10.0}}. Thus HMS
> {{getPartitionsByNames}} cannot find the partition with that value and thus
> fails to insert the stats. So while the failure initiates on HMS side, the
> cause in the HS2 query planning.
> It makes sense that turning off {{hive.stats.column.autogather}} resolves the
> issue because there is no StatsTask in a query plan.
> But {{SHOW PARTITIONS}} shows the partition as created while the query
> planner is not including it any plan because of the absence of stats on the
> partition.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-21543) Use FilterHooks for show compactions
[ https://issues.apache.org/jira/browse/HIVE-21543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-21543: -- Attachment: HIVE-21543.02.patch > Use FilterHooks for show compactions > > > Key: HIVE-21543 > URL: https://issues.apache.org/jira/browse/HIVE-21543 > Project: Hive > Issue Type: Improvement >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21543.01.patch, HIVE-21543.02.patch > > > Use FilterHooks for checking dbs/tables/partitions for showCompactions -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-14298) NPE could be thrown in HMS when an ExpressionTree could not be made from a filter
[
https://issues.apache.org/jira/browse/HIVE-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901947#comment-16901947
]
xiepengjie commented on HIVE-14298:
---
Hi, [~ctang],
About your test case:
{code:java}
It is quite easy to reproduce the NPE issue with following steps:
set the HMS configurations:
hive.metastore.try.direct.sql to true
hive.metastore.limit.partition.request to a certain positive integer (-1 means
disabled which is default).
Run query like
select * from sample_pt where code in ('53-5022', '53-5023') and dummy like
'%1';
you will get "FAILED: SemanticException java.lang.NullPointerException"
{code}
How to create a table named sample_pt which can reproduce this?
> NPE could be thrown in HMS when an ExpressionTree could not be made from a
> filter
> -
>
> Key: HIVE-14298
> URL: https://issues.apache.org/jira/browse/HIVE-14298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
>Priority: Major
> Fix For: 2.1.1, 2.2.0
>
> Attachments: HIVE-14298.patch, HIVE-14298.patch, HIVE-14298.patch
>
>
> In many cases where an ExpressionTree could not be made from a filter (e.g.
> parser fails to parse a filter etc.) and its value is null. But this null is
> passed around and used by a couple of HMS methods which can cause
> NullPointerException.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22083) Values of tag order cannot be null, so it can be "byte" instead of "Byte"
[
https://issues.apache.org/jira/browse/HIVE-22083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901861#comment-16901861
]
Hive QA commented on HIVE-22083:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976896/HIVE-22083.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16723 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query23]
(batchId=298)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query88]
(batchId=298)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18278/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18278/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18278/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976896 - PreCommit-HIVE-Build
> Values of tag order cannot be null, so it can be "byte" instead of "Byte"
> -
>
> Key: HIVE-22083
> URL: https://issues.apache.org/jira/browse/HIVE-22083
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Minor
> Attachments: HIVE-22083.1.patch, HIVE-22083.2.patch
>
>
> Values of tag order cannot be null, so it can be "byte" instead of "Byte".
> Switching between Byte and byte is "cheap" - the Byte objects are cached by
> the JVM - but it still costs a bit more memory and CPU usage.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22083) Values of tag order cannot be null, so it can be "byte" instead of "Byte"
[
https://issues.apache.org/jira/browse/HIVE-22083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901807#comment-16901807
]
Hive QA commented on HIVE-22083:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 1 new + 599 unchanged - 53
fixed = 600 total (was 652) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
21s{color} | {color:red} ql generated 5 new + 2238 unchanged - 12 fixed = 2243
total (was 2250) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to evaluatorWindowFrameDefs in
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.createVectorPTFInfo(Operator,
PTFDesc, VectorizationContext, VectorPTFDesc) At
Vectorizer.java:org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.createVectorPTFInfo(Operator,
PTFDesc, VectorizationContext, VectorPTFDesc) At Vectorizer.java:[line 4972] |
| | org.apache.hadoop.hive.ql.plan.HashTableSinkDesc.getTagOrder() may expose
internal representation by returning HashTableSinkDesc.tagOrder At
HashTableSinkDesc.java:by returning HashTableSinkDesc.tagOrder At
HashTableSinkDesc.java:[line 263] |
| | org.apache.hadoop.hive.ql.plan.HashTableSinkDesc.setTagOrder(byte[]) may
expose internal representation by storing an externally mutable object into
HashTableSinkDesc.tagOrder At HashTableSinkDesc.java:by storing an externally
mutable object into HashTableSinkDesc.tagOrder At HashTableSinkDesc.java:[line
268] |
| | org.apache.hadoop.hive.ql.plan.JoinDesc.getTagOrder() may expose internal
representation by returning JoinDesc.tagOrder At JoinDesc.java:by returning
JoinDesc.tagOrder At JoinDesc.java:[line 419] |
| | org.apache.hadoop.hive.ql.plan.JoinDesc.setTagOrder(byte[]) may expose
internal representation by storing an externally mutable object into
JoinDesc.tagOrder At JoinDesc.java:by storing an externally mutable object
into JoinDesc.tagOrder At JoinDesc.java:[line 429] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18278/dev-support/hive-personality.sh
|
| git revision | master / 333264b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18278/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18278/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18278/yetus.txt |
| Powered by | Apache Yetus
[jira] [Updated] (HIVE-22083) Values of tag order cannot be null, so it can be "byte" instead of "Byte"
[ https://issues.apache.org/jira/browse/HIVE-22083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-22083: --- Attachment: HIVE-22083.2.patch > Values of tag order cannot be null, so it can be "byte" instead of "Byte" > - > > Key: HIVE-22083 > URL: https://issues.apache.org/jira/browse/HIVE-22083 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-22083.1.patch, HIVE-22083.2.patch > > > Values of tag order cannot be null, so it can be "byte" instead of "Byte". > Switching between Byte and byte is "cheap" - the Byte objects are cached by > the JVM - but it still costs a bit more memory and CPU usage. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
