Re: Review Request 35107: HIVE-6791 Support variable substition for Beeline shell command
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35107/ --- (Updated June 26, 2015, 2 p.m.) Review request for hive, chinna and Xuefu Zhang. Bugs: HIVE-6791 https://issues.apache.org/jira/browse/HIVE-6791 Repository: hive-git Description --- Summary: 1) move the beeline-cli convertor to the place where cli is executed(class **Commands**) 2) support substitution for source command 3) add some unit test for substitution 4) add one way to get the configuration from HS2 Diffs (updated) - beeline/src/java/org/apache/hive/beeline/BeeLine.java b7d2f2e beeline/src/java/org/apache/hive/beeline/Commands.java a42baa3 beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java 6cbb030 cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java d62fd5c common/src/java/org/apache/hadoop/hive/conf/HiveVariableSource.java PRE-CREATION common/src/java/org/apache/hadoop/hive/conf/VariableSubstitution.java PRE-CREATION common/src/test/org/apache/hadoop/hive/conf/TestVariableSubstitution.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/Driver.java 338e755 ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java a5f0a7f ql/src/java/org/apache/hadoop/hive/ql/parse/VariableSubstitution.java e8b1d96 ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java 0558c53 ql/src/java/org/apache/hadoop/hive/ql/processors/CompileProcessor.java 25ce168 ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java 9052c82 ql/src/java/org/apache/hadoop/hive/ql/processors/DfsProcessor.java cc0414d ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java bc9254c service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 33ee16b Diff: https://reviews.apache.org/r/35107/diff/ Testing --- Unit test passed Thanks, cheng xu
Re: Review Request 35107: HIVE-6791 Support variable substition for Beeline shell command
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35107/ --- (Updated June 26, 2015, 4:02 p.m.) Review request for hive, chinna and Xuefu Zhang. Changes --- Summary: 1) rebase code 2) add some java doc 3) do some code clean work Bugs: HIVE-6791 https://issues.apache.org/jira/browse/HIVE-6791 Repository: hive-git Description --- Summary: 1) move the beeline-cli convertor to the place where cli is executed(class **Commands**) 2) support substitution for source command 3) add some unit test for substitution 4) add one way to get the configuration from HS2 Diffs (updated) - beeline/src/java/org/apache/hive/beeline/BeeLine.java 66fe322 beeline/src/java/org/apache/hive/beeline/Commands.java aaf6aec beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java 6cbb030 cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java d62fd5c common/src/java/org/apache/hadoop/hive/conf/HiveVariableSource.java PRE-CREATION common/src/java/org/apache/hadoop/hive/conf/VariableSubstitution.java PRE-CREATION common/src/test/org/apache/hadoop/hive/conf/TestVariableSubstitution.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/Driver.java 669e6be ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java a5f0a7f ql/src/java/org/apache/hadoop/hive/ql/parse/VariableSubstitution.java e8b1d96 ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java 0558c53 ql/src/java/org/apache/hadoop/hive/ql/processors/CompileProcessor.java 25ce168 ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java 9052c82 ql/src/java/org/apache/hadoop/hive/ql/processors/DfsProcessor.java cc0414d ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java d271d6d service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java cc9df76 Diff: https://reviews.apache.org/r/35107/diff/ Testing --- Unit test passed Thanks, cheng xu
Hive-0.14 - Build # 996 - Still Failing
Changes for Build #980 Changes for Build #981 Changes for Build #982 Changes for Build #983 Changes for Build #984 Changes for Build #985 Changes for Build #986 Changes for Build #987 Changes for Build #988 Changes for Build #989 Changes for Build #990 Changes for Build #991 Changes for Build #992 Changes for Build #993 Changes for Build #994 Changes for Build #995 Changes for Build #996 No tests ran. The Apache Jenkins build system has built Hive-0.14 (build #996) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-0.14/996/ to view the results.
Re: [ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and Chinna Rao Lalam
Congratulations China and Jesus, and thanks for all your contributions! -- Lefty On Fri, Jun 26, 2015 at 7:01 PM, Sergio Pena sergio.p...@cloudera.com wrote: Congratulations China and Jesus !!!. - Sergio On Fri, Jun 26, 2015 at 1:57 PM, Carl Steinbach c...@apache.org wrote: On behalf of the Apache Hive PMC I am pleased to announce that Jesus Camacho Rodriguez and Chinna Rao Lalam have been voted in as committers. Please join me in congratulating Jesus and Chinna! Thanks. - Carl
Re: Review Request 35899: HIVE-11104
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35899/#review89523 --- ql/src/test/results/clientpositive/annotate_stats_select.q.out (line 1025) https://reviews.apache.org/r/35899/#comment142153 Stats shouldn't be 0 right? Select stats annotation can handle constants AFAIK. ql/src/test/results/clientpositive/input_part10.q.out (line 51) https://reviews.apache.org/r/35899/#comment142154 Same issue here. ql/src/test/results/clientpositive/insert_into5.q.out (line 45) https://reviews.apache.org/r/35899/#comment142155 We seem to drop stats everywhere. - Prasanth_J On June 26, 2015, 12:19 a.m., Ashutosh Chauhan wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35899/ --- (Updated June 26, 2015, 12:19 a.m.) Review request for hive. Bugs: HIVE-11104 https://issues.apache.org/jira/browse/HIVE-11104 Repository: hive-git Description --- Select operator doesn't propagate constants appearing in expressions Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java 3c8c0d6 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java f9df8e5 ql/src/test/queries/clientpositive/cp_sel.q PRE-CREATION ql/src/test/results/clientpositive/annotate_stats_select.q.out 306b870 ql/src/test/results/clientpositive/auto_join8.q.out 5b02597 ql/src/test/results/clientpositive/cluster.q.out eff3399 ql/src/test/results/clientpositive/cp_sel.q.out PRE-CREATION ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out bcc03cf ql/src/test/results/clientpositive/input7.q.out 3abfabb ql/src/test/results/clientpositive/input_part10.q.out 4f8a07f ql/src/test/results/clientpositive/insert_into5.q.out 490f737 ql/src/test/results/clientpositive/join8.q.out 71792c1 ql/src/test/results/clientpositive/join_cond_pushdown_1.q.out be83963 ql/src/test/results/clientpositive/join_cond_pushdown_3.q.out a3641d6 ql/src/test/results/clientpositive/list_bucket_dml_12.q.java1.7.out 0b518e4 ql/src/test/results/clientpositive/list_bucket_dml_13.q.java1.7.out f834787 ql/src/test/results/clientpositive/load_dyn_part14.q.out 5af8490 ql/src/test/results/clientpositive/ppd_udf_case.q.out 0c53532 ql/src/test/results/clientpositive/ppd_union_view.q.out f0f6614 ql/src/test/results/clientpositive/spark/auto_join8.q.out e77817a ql/src/test/results/clientpositive/spark/join8.q.out e3fac82 ql/src/test/results/clientpositive/spark/join_cond_pushdown_1.q.out ddfdc66 ql/src/test/results/clientpositive/spark/join_cond_pushdown_3.q.out d9ad78d ql/src/test/results/clientpositive/spark/load_dyn_part14.q.out 4df5e49 ql/src/test/results/clientpositive/spark/union_remove_12.q.out 62d9d7d ql/src/test/results/clientpositive/spark/union_remove_14.q.out b28bb75 ql/src/test/results/clientpositive/spark/union_top_level.q.out dede1ef ql/src/test/results/clientpositive/tez/tez_union_dynamic_partition.q.out 68a7531 ql/src/test/results/clientpositive/tez/vector_coalesce.q.out 1e01ab5 ql/src/test/results/clientpositive/tez/vector_decimal_2.q.out bbdd75f ql/src/test/results/clientpositive/tez/vector_decimal_round_2.q.out 103fd8a ql/src/test/results/clientpositive/tez/vector_interval_1.q.out 1cc7cb6 ql/src/test/results/clientpositive/union_remove_12.q.out 3ac1d44 ql/src/test/results/clientpositive/union_remove_14.q.out 94f2ff5 ql/src/test/results/clientpositive/union_top_level.q.out c64e792 ql/src/test/results/clientpositive/vector_coalesce.q.out eb5f204 ql/src/test/results/clientpositive/vector_decimal_2.q.out c0112a9 ql/src/test/results/clientpositive/vector_decimal_round_2.q.out 6947059 ql/src/test/results/clientpositive/vector_interval_1.q.out 5015916 Diff: https://reviews.apache.org/r/35899/diff/ Testing --- Added new test pre-existing ones. Thanks, Ashutosh Chauhan
Re: Review Request 35803: HIVE-10895 ObjectStore does not close Query objects in some calls, causing a potential leak in some metastore db resources
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35803/ --- (Updated June 26, 2015, 5:23 p.m.) Review request for hive. Repository: hive-git Description --- HIVE-10895 ObjectStore does not close Query objects in some calls, causing a potential leak in some metastore db resources Diffs (updated) - metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 417ecc8 metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java d0ff329 metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java PRE-CREATION Diff: https://reviews.apache.org/r/35803/diff/ Testing --- Testing has been done. Thanks, Aihua Xu
Re: Review Request 35930: HIVE-11119
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35930/#review89540 --- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java https://reviews.apache.org/r/35930/#comment142164 nit qq: should we keep the comments or remove it also from the ReducerRecordSource.java and put in the util class? - Xuefu Zhang On June 26, 2015, 5:17 p.m., Ashutosh Chauhan wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35930/ --- (Updated June 26, 2015, 5:17 p.m.) Review request for hive. Bugs: HIVE-9 https://issues.apache.org/jira/browse/HIVE-9 Repository: hive-git Description --- Spark reduce vectorization doesnt account for scratch columns Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java 23d6a5d ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java cdabe3a ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedBatchUtil.java 99cb620 Diff: https://reviews.apache.org/r/35930/diff/ Testing --- Regression suite. Thanks, Ashutosh Chauhan
Review Request 35930: HIVE-11119
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35930/ --- Review request for hive. Bugs: HIVE-9 https://issues.apache.org/jira/browse/HIVE-9 Repository: hive-git Description --- Spark reduce vectorization doesnt account for scratch columns Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java 23d6a5d ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java cdabe3a ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedBatchUtil.java 99cb620 Diff: https://reviews.apache.org/r/35930/diff/ Testing --- Regression suite. Thanks, Ashutosh Chauhan
[jira] [Created] (HIVE-11128) Stats annotation should consider select star same as select without column list
Ashutosh Chauhan created HIVE-11128: --- Summary: Stats annotation should consider select star same as select without column list Key: HIVE-11128 URL: https://issues.apache.org/jira/browse/HIVE-11128 Project: Hive Issue Type: Bug Components: Statistics Affects Versions: 1.2.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and Chinna Rao Lalam
Congrats! On Jun 26, 2015 1:36 PM, Hari Subramaniyan hsubramani...@hortonworks.com wrote: Congrats Jesús and Chinna! From: Chao Sun c...@cloudera.com Sent: Friday, June 26, 2015 12:06 PM To: dev@hive.apache.org Cc: jcama...@apache.org; Lalam, Chinna R Subject: Re: [ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and Chinna Rao Lalam Good job! Congratulations! On Fri, Jun 26, 2015 at 12:02 PM, Jimmy Xiang jxi...@cloudera.com wrote: Congrats!! On Fri, Jun 26, 2015 at 12:00 PM, Xuefu Zhang xzh...@cloudera.com wrote: Congratulations, Chinna and Jesus! Impressive! --Xuefu On Fri, Jun 26, 2015 at 11:57 AM, Carl Steinbach c...@apache.org wrote: On behalf of the Apache Hive PMC I am pleased to announce that Jesus Camacho Rodriguez and Chinna Rao Lalam have been voted in as committers. Please join me in congratulating Jesus and Chinna! Thanks. - Carl
Re: [ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and Chinna Rao Lalam
Congratulations guys! On Jun 26, 2015, at 12:00 PM, Xuefu Zhang xzh...@cloudera.com wrote: Congratulations, Chinna and Jesus! Impressive! --Xuefu On Fri, Jun 26, 2015 at 11:57 AM, Carl Steinbach c...@apache.org wrote: On behalf of the Apache Hive PMC I am pleased to announce that Jesus Camacho Rodriguez and Chinna Rao Lalam have been voted in as committers. Please join me in congratulating Jesus and Chinna! Thanks. - Carl
Re: [ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and Chinna Rao Lalam
Congrats!! On Fri, Jun 26, 2015 at 12:00 PM, Xuefu Zhang xzh...@cloudera.com wrote: Congratulations, Chinna and Jesus! Impressive! --Xuefu On Fri, Jun 26, 2015 at 11:57 AM, Carl Steinbach c...@apache.org wrote: On behalf of the Apache Hive PMC I am pleased to announce that Jesus Camacho Rodriguez and Chinna Rao Lalam have been voted in as committers. Please join me in congratulating Jesus and Chinna! Thanks. - Carl
Re: [ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and Chinna Rao Lalam
Good job! Congratulations! On Fri, Jun 26, 2015 at 12:02 PM, Jimmy Xiang jxi...@cloudera.com wrote: Congrats!! On Fri, Jun 26, 2015 at 12:00 PM, Xuefu Zhang xzh...@cloudera.com wrote: Congratulations, Chinna and Jesus! Impressive! --Xuefu On Fri, Jun 26, 2015 at 11:57 AM, Carl Steinbach c...@apache.org wrote: On behalf of the Apache Hive PMC I am pleased to announce that Jesus Camacho Rodriguez and Chinna Rao Lalam have been voted in as committers. Please join me in congratulating Jesus and Chinna! Thanks. - Carl
Re: [ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and Chinna Rao Lalam
Congrats Jesús and Chinna! From: Chao Sun c...@cloudera.com Sent: Friday, June 26, 2015 12:06 PM To: dev@hive.apache.org Cc: jcama...@apache.org; Lalam, Chinna R Subject: Re: [ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and Chinna Rao Lalam Good job! Congratulations! On Fri, Jun 26, 2015 at 12:02 PM, Jimmy Xiang jxi...@cloudera.com wrote: Congrats!! On Fri, Jun 26, 2015 at 12:00 PM, Xuefu Zhang xzh...@cloudera.com wrote: Congratulations, Chinna and Jesus! Impressive! --Xuefu On Fri, Jun 26, 2015 at 11:57 AM, Carl Steinbach c...@apache.org wrote: On behalf of the Apache Hive PMC I am pleased to announce that Jesus Camacho Rodriguez and Chinna Rao Lalam have been voted in as committers. Please join me in congratulating Jesus and Chinna! Thanks. - Carl
[ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and Chinna Rao Lalam
On behalf of the Apache Hive PMC I am pleased to announce that Jesus Camacho Rodriguez and Chinna Rao Lalam have been voted in as committers. Please join me in congratulating Jesus and Chinna! Thanks. - Carl
Re: [ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and Chinna Rao Lalam
Congratulations, Chinna and Jesus! Impressive! --Xuefu On Fri, Jun 26, 2015 at 11:57 AM, Carl Steinbach c...@apache.org wrote: On behalf of the Apache Hive PMC I am pleased to announce that Jesus Camacho Rodriguez and Chinna Rao Lalam have been voted in as committers. Please join me in congratulating Jesus and Chinna! Thanks. - Carl
Re: [ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and Chinna Rao Lalam
Congratulations Jesus, Chinna Rao. On 6/26/15, 11:57 AM, Carl Steinbach c...@apache.org wrote: On behalf of the Apache Hive PMC I am pleased to announce that Jesus Camacho Rodriguez and Chinna Rao Lalam have been voted in as committers. Please join me in congratulating Jesus and Chinna! Thanks. - Carl
[jira] [Created] (HIVE-11129) Issue a warning when copied from UTF-8 to ISO 8859-1
Aihua Xu created HIVE-11129:
---
Summary: Issue a warning when copied from UTF-8 to ISO 8859-1
Key: HIVE-11129
URL: https://issues.apache.org/jira/browse/HIVE-11129
Project: Hive
Issue Type: Bug
Components: File Formats
Reporter: Aihua Xu
Copying data from a table using UTF-8 encoding to one using ISO 8859-1 encoding
causes data corruption without warning.
{noformat}
CREATE TABLE person_utf8 (name STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('serialization.encoding'='UTF8');
{noformat}
Put the following data in the table:
Müller,Thomas
Jørgensen,Jørgen
Vega,Andrés
中村,浩人
אביה,נועם
{noformat}
CREATE TABLE person_2 ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('serialization.encoding'='ISO8859_1')
AS select * from person_utf8;
{noformat}
expected to get mangled data but we should give a warning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-11130) Refactoring the code so that HiveTxnManager interface will support lock/unlock table/database object
Aihua Xu created HIVE-11130: --- Summary: Refactoring the code so that HiveTxnManager interface will support lock/unlock table/database object Key: HIVE-11130 URL: https://issues.apache.org/jira/browse/HIVE-11130 Project: Hive Issue Type: Sub-task Components: Locking Affects Versions: 2.0.0 Reporter: Aihua Xu Assignee: Aihua Xu This is just a refactoring step which keeps the current logic, but it exposes the explicit lock/unlock table and database in HiveTxnManager which should be implemented differently by the subclasses ( currently it's not. e.g., for ZooKeeper implementation, we should lock table and database when we try to lock the table). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11125) when i run a sql use hive on spark, it seem like the hive cli finished, but the application is always running
JoneZhang created HIVE-11125: Summary: when i run a sql use hive on spark, it seem like the hive cli finished, but the application is always running Key: HIVE-11125 URL: https://issues.apache.org/jira/browse/HIVE-11125 Project: Hive Issue Type: Bug Components: spark-branch Affects Versions: 1.2.0 Environment: Hive1.2.0 Spark1.3.1 Hadoop2.5.1 Reporter: JoneZhang when i run a sql use hive on spark,. The hive cli has finished hive (default) select count(id) from t1 where id100; Query ID = mqq_20150626174732_9e18f0c9-7b56-46ab-bf90-3b66f1a51300 Total jobs = 1 Launching Job 1 out of 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=number In order to limit the maximum number of reducers: set hive.exec.reducers.max=number In order to set a constant number of reducers: set mapreduce.job.reduces=number Starting Spark Job = 7d34cb8c-eaad-4724-a99a-37e517db80d9 Query Hive on Spark job[0] stages: 0 1 Status: Running (Hive on Spark job[0]) Job Progress Format CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost] 2015-06-26 17:47:53,746 Stage-0_0: 0(+1)/5 Stage-1_0: 0/1 2015-06-26 17:47:56,771 Stage-0_0: 1(+0)/5 Stage-1_0: 0/1 2015-06-26 17:47:57,778 Stage-0_0: 4(+1)/5 Stage-1_0: 0/1 2015-06-26 17:47:59,791 Stage-0_0: 5/5 Finished Stage-1_0: 0(+1)/1 2015-06-26 17:48:00,797 Stage-0_0: 5/5 Finished Stage-1_0: 1/1 Finished Status: Finished successfully in 18.08 seconds OK 5 Time taken: 28.512 seconds, Fetched: 1 row(s) But the application is always running state on resourcemanager User: mqq Name: Hive on Spark Application Type: SPARK Application Tags: State: RUNNING FinalStatus:UNDEFINED Started:2015-06-26 17:47:38 Elapsed:24mins, 33sec Tracking URL: ApplicationMaster Diagnostics: the hive.log is 2015-06-26 18:12:26,878 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 15/06/26 18:12:26 main INFO org.apache.spark.deploy.yarn.Client Application report for application_1433328839160_0071 (state: RUNNING) 2015-06-26 18:12:27,879 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 15/06/26 18:12:27 main INFO org.apache.spark.deploy.yarn.Client Application report for application_1433328839160_0071 (state: RUNNING) 2015-06-26 18:12:28,880 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 15/06/26 18:12:28 main INFO org.apache.spark.deploy.yarn.Client Application report for application_1433328839160_0071 (state: RUNNING) ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11133) Support hive.explain.user for Spark
Mohit Sabharwal created HIVE-11133:
--
Summary: Support hive.explain.user for Spark
Key: HIVE-11133
URL: https://issues.apache.org/jira/browse/HIVE-11133
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Mohit Sabharwal
User friendly explain output ({{set hive.explain.user=true}}) should support
Spark as well.
Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-11132) Queries using join and group by produce incorrect output when hive.auto.convert.join=false and hive.optimize.reducededuplication=true
Rich Haase created HIVE-11132: - Summary: Queries using join and group by produce incorrect output when hive.auto.convert.join=false and hive.optimize.reducededuplication=true Key: HIVE-11132 URL: https://issues.apache.org/jira/browse/HIVE-11132 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Rich Haase Queries using join and group by produce multiple output rows with the same key when hive.auto.convert.join=false and hive.optimize.reducededuplication=true. This interaction between configuration parameters is unexpected and should be well documented at the very least and should likely be considered a bug. e.g. hive set hive.auto.convert.join = false; hive set hive.optimize.reducededuplication = true; hive SELECT foo.id, count(*) as factor FROM foo JOIN bar ON (foo.id = bar.id and foo.line_id = bar.line_id) JOIN split ON (foo.id = split.id and foo.line_id = split.line_id) JOIN forecast ON (foo.id = forecast.id AND foo.line_id = forecast.line_id) WHERE foo.order != ‘blah’ AND foo.id = ‘XYZ' GROUP BY foo.id; XYZ 79 XYZ 74 XYZ 297 XYZ 66 hive set hive.auto.convert.join = true; hive set hive.optimize.reducededuplication = true; hive SELECT foo.id, count(*) as factor FROM foo JOIN bar ON (foo.id = bar.id and foo.line_id = bar.line_id) JOIN split ON (foo.id = split.id and foo.line_id = split.line_id) JOIN forecast ON (foo.id = forecast.id AND foo.line_id = forecast.line_id) WHERE foo.order != ‘blah’ AND foo.id = ‘XYZ' GROUP BY foo.id; XYZ 516 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11131) Get row information on DataWritableWriter once for better writing performance
Sergio Peña created HIVE-11131: -- Summary: Get row information on DataWritableWriter once for better writing performance Key: HIVE-11131 URL: https://issues.apache.org/jira/browse/HIVE-11131 Project: Hive Issue Type: Sub-task Affects Versions: 1.2.0 Reporter: Sergio Peña Assignee: Sergio Peña DataWritableWriter is a class used to write Hive records to Parquet files. This class is getting all the information about how to parse a record, such as schema and object inspector, every time a record is written (or write() is called). We can make this class perform better by initializing some writers per data type once, and saving all object inspectors on each writer. The class expects that the next records written will have the same object inspectors and schema, so there is no need to have conditions for that. When a new schema is written, DataWritableWriter is created again by Parquet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 35950: HIVE-11131: Get row information on DataWritableWriter once for better writing performance
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35950/ --- Review request for hive, Ryan Blue, cheng xu, and Dong Chen. Bugs: HIVE-11131 https://issues.apache.org/jira/browse/HIVE-11131 Repository: hive-git Description --- Implemented data type writers that will be created before the first Hive row is written to Parquet. These writers contain information about object inspectors and schema of a specific data type, and calls the specific add() method used by Parquet for each data type. Diffs - ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java c195c3ec3ddae19bf255fc2c9633f8bf4390f428 Diff: https://reviews.apache.org/r/35950/diff/ Testing --- Tests from TestDataWritableWriter run OK. I run other tests with micro-becnhmarks, and I got some better results from this new implemntation: Using repeated rows across the file, the speed increased in: bigint boolean double float int string 33.42% 53.66% 35.62% 35.70% 36.02% 5.93% Using random rows across the file, the speed increased in: bigint boolean double float int string 18.38% 35.52% 44.73% 13.80% 10.68% 10.00% Thanks, Sergio Pena
Question regarding the use of TaskAttemptContext on ParquetOutputFormat
Hi, I see ParquetRecordWriterWrapper constructor is getting/initializing a TaskAttemptID object that will be passed to the getRecordWriter(TaskAttemptContext taskAttemptContext, Path file) method of ParquetOutputFormat. But this method only gets the Configuration and CompressionCodeName objects to pass to another constructor. My question is, if TaskAttempID links the Configuration object from the JobConf parameter of ParquetRecordWriterWrapper, and the Code name can be retrieved from the JobConf or Properties objects, is there another reason about using TaskAttempID? During some java profile tests, I noticed that ContextUtil.newTaskAttemptContext() takes some time to initialize, and we can save that time if we use the other constructor. - Sergio
Re: [ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and Chinna Rao Lalam
Congratulations China and Jesus !!!. - Sergio On Fri, Jun 26, 2015 at 1:57 PM, Carl Steinbach c...@apache.org wrote: On behalf of the Apache Hive PMC I am pleased to announce that Jesus Camacho Rodriguez and Chinna Rao Lalam have been voted in as committers. Please join me in congratulating Jesus and Chinna! Thanks. - Carl
Re: Review Request 35950: HIVE-11131: Get row information on DataWritableWriter once for better writing performance
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35950/ --- (Updated June 27, 2015, 2:51 a.m.) Review request for hive, Ryan Blue, cheng xu, and Dong Chen. Changes --- Added change on DataListWriter to loop into the list of values using a for() instead of for each. The ListObjectInspector.getList() is more expensive than getListLength() and getListElement() Bugs: HIVE-11131 https://issues.apache.org/jira/browse/HIVE-11131 Repository: hive-git Description --- Implemented data type writers that will be created before the first Hive row is written to Parquet. These writers contain information about object inspectors and schema of a specific data type, and calls the specific add() method used by Parquet for each data type. Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java c195c3ec3ddae19bf255fc2c9633f8bf4390f428 Diff: https://reviews.apache.org/r/35950/diff/ Testing --- Tests from TestDataWritableWriter run OK. I run other tests with micro-becnhmarks, and I got some better results from this new implemntation: Using repeated rows across the file, the speed increased in: bigint boolean double float int string 33.42% 53.66% 35.62% 35.70% 36.02% 5.93% Using random rows across the file, the speed increased in: bigint boolean double float int string 18.38% 35.52% 44.73% 13.80% 10.68% 10.00% Thanks, Sergio Pena
Review Request 35963: HIVE-7150 FileInputStream is not closed in HiveConnection class
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35963/ --- Review request for hive, Ashutosh Chauhan and Thejas Nair. Bugs: HIVE-7150 https://issues.apache.org/jira/browse/HIVE-7150 Repository: hive-git Description --- HIVE-7150 FileInputStream is not closed in HiveConnection class Diffs - jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 277f6d44d3b3472218526a2933380420676729a3 jdbc/src/java/org/apache/hive/jdbc/Utils.java 0e4693b34544fcc4845a7ed0b16dc5299a06455a Diff: https://reviews.apache.org/r/35963/diff/ Testing --- Thanks, Alexander Pivovarov
Review Request 35964: HIVE-7180 BufferedReader is not closed in MetaStoreSchemaInfo
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35964/ --- Review request for hive and Ashutosh Chauhan. Bugs: HIVE-7180 https://issues.apache.org/jira/browse/HIVE-7180 Repository: hive-git Description --- HIVE-7180 BufferedReader is not closed in MetaStoreSchemaInfo Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java a5eed6d3c0ff4a229f3209175b8dfcf8bde5e79e Diff: https://reviews.apache.org/r/35964/diff/ Testing --- Thanks, Alexander Pivovarov
[jira] [Created] (HIVE-11126) multiple insert fails when select with group by clause
Guodong Wang created HIVE-11126:
---
Summary: multiple insert fails when select with group by clause
Key: HIVE-11126
URL: https://issues.apache.org/jira/browse/HIVE-11126
Project: Hive
Issue Type: Bug
Components: Parser
Affects Versions: 0.12.0
Reporter: Guodong Wang
When the select statement contains group by clause, multiple insert fails.
Here is the sample sql.
{code}
from test_src_table
insert overwrite table test_target_table partition(p)
select src_id as id, lala as p
group by src_id
insert overwrite table test_target_table partition(p)
select id, p from
select src_id as id, papa as p
group by src_id
{code}
The exception is like this
{code}
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{_col0:1107625...@qq.com,_col1:lala}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row {_col0:1107625...@qq.com,_col1:lala}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:550)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(Ex
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-11127) Document time zone handling for current_date and current_timestamp
Punya Biswal created HIVE-11127:
---
Summary: Document time zone handling for current_date and
current_timestamp
Key: HIVE-11127
URL: https://issues.apache.org/jira/browse/HIVE-11127
Project: Hive
Issue Type: Improvement
Affects Versions: 1.2.0
Reporter: Punya Biswal
The new {{current_date}} and {{current_timestamp}} functions introduced in
HIVE-5472 emit dates/timestamps in the user's local timezone. This behavior
should be documented on [the
wiki|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
