[jira] [Commented] (HIVE-12257) Enhance ORC FileDump utility to handle flush_length files and recovery
[ https://issues.apache.org/jira/browse/HIVE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041231#comment-15041231 ] Lefty Leverenz commented on HIVE-12257: --- Does this need any documentation? * [ORC File Dump Utility | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-ORCFileDumpUtility] > Enhance ORC FileDump utility to handle flush_length files and recovery > -- > > Key: HIVE-12257 > URL: https://issues.apache.org/jira/browse/HIVE-12257 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-12257-branch-1.patch, HIVE-12257.1.patch, > HIVE-12257.10.patch, HIVE-12257.11.patch, HIVE-12257.12.patch, > HIVE-12257.2.patch, HIVE-12257.3.patch, HIVE-12257.4.patch, > HIVE-12257.6.patch, HIVE-12257.7.patch, HIVE-12257.8.patch, HIVE-12257.9.patch > > > ORC file dump utility currently does not handle delta directories that > contain *_flush_length files. These files contains offsets to footer in the > corresponding delta file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11110) Reorder applyPreJoinOrderingTransforms, add NotNULL/FilterMerge rules, improve Filter selectivity estimation
[
https://issues.apache.org/jira/browse/HIVE-0?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041209#comment-15041209
]
Hive QA commented on HIVE-0:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775477/HIVE-0.29.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6229/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6229/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6229/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6229/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 22fc397 HIVE-12444 Global Limit optimization on ACID table
without base directory may throw exception ADDENDUM (Wei Zheng via Eugene
Koifman)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 22fc397 HIVE-12444 Global Limit optimization on ACID table
without base directory may throw exception ADDENDUM (Wei Zheng via Eugene
Koifman)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775477 - PreCommit-HIVE-TRUNK-Build
> Reorder applyPreJoinOrderingTransforms, add NotNULL/FilterMerge rules,
> improve Filter selectivity estimation
>
>
> Key: HIVE-0
> URL: https://issues.apache.org/jira/browse/HIVE-0
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-0-10.patch, HIVE-0-11.patch,
> HIVE-0-12.patch, HIVE-0-branch-1.2.patch, HIVE-0.1.patch,
> HIVE-0.13.patch, HIVE-0.14.patch, HIVE-0.15.patch,
> HIVE-0.16.patch, HIVE-0.17.patch, HIVE-0.18.patch,
> HIVE-0.19.patch, HIVE-0.2.patch, HIVE-0.20.patch,
> HIVE-0.21.patch, HIVE-0.22.patch, HIVE-0.23.patch,
> HIVE-0.24.patch, HIVE-0.25.patch, HIVE-0.26.patch, HIVE-0.27,
> HIVE-0.27.patch, HIVE-0.28.patch, HIVE-0.29.patch,
> HIVE-0.4.patch, HIVE-0.5.patch, HIVE-0.6.patch,
> HIVE-0.7.patch, HIVE-0.8.patch, HIVE-0.9.patch,
> HIVE-0.91.patch, HIVE-0.92.patch, HIVE-0.patch
>
>
> Query
> {code}
> select count(*)
> from store_sales
> ,store_returns
> ,date_dim d1
> ,date_dim d2
> where d1.d_quarter_name = '2000Q1'
>and d1.d_date_sk = ss_sold_date_sk
>and ss_customer_sk = sr_customer_sk
>and ss_item_sk = sr
[jira] [Commented] (HIVE-10982) Customizable the value of java.sql.statement.setFetchSize in Hive JDBC Driver
[
https://issues.apache.org/jira/browse/HIVE-10982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041207#comment-15041207
]
Hive QA commented on HIVE-10982:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12774948/HIVE-10982.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6228/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6228/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6228/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6228/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 22fc397 HIVE-12444 Global Limit optimization on ACID table
without base directory may throw exception ADDENDUM (Wei Zheng via Eugene
Koifman)
+ git clean -f -d
Removing
ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java.orig
Removing ql/src/test/queries/clientnegative/invalid_select_fn.q
Removing ql/src/test/results/clientnegative/invalid_select_fn.q.out
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 22fc397 HIVE-12444 Global Limit optimization on ACID table
without base directory may throw exception ADDENDUM (Wei Zheng via Eugene
Koifman)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12774948 - PreCommit-HIVE-TRUNK-Build
> Customizable the value of java.sql.statement.setFetchSize in Hive JDBC Driver
> --
>
> Key: HIVE-10982
> URL: https://issues.apache.org/jira/browse/HIVE-10982
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Bing Li
>Assignee: Bing Li
>Priority: Critical
> Attachments: HIVE-10982.1.patch, HIVE-10982.2.patch
>
>
> The current JDBC driver for Hive hard-code the value of setFetchSize to 50,
> which will be a bottleneck for performance.
> Pentaho filed this issue as http://jira.pentaho.com/browse/PDI-11511, whose
> status is open.
> Also it has discussion in
> http://forums.pentaho.com/showthread.php?158381-Hive-JDBC-Query-too-slow-too-many-fetches-after-query-execution-Kettle-Xform
> http://mail-archives.apache.org/mod_mbox/hive-user/201307.mbox/%3ccacq46vevgrfqg5rwxnr1psgyz7dcf07mvlo8mm2qit3anm1...@mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12570) Incorrect error message Expression not in GROUP BY key thrown instead of Invalid function
[
https://issues.apache.org/jira/browse/HIVE-12570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041202#comment-15041202
]
Hive QA commented on HIVE-12570:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775463/HIVE-12570.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 328 failed/errored test(s), 9890 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_admin_almighty2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_cli_nonsql
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autogen_colalias
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_udfs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binarysortable_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_insert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_lineage2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_subq_exists
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_subq_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_subq_not_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_udf_percentile
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_windowing
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_windowing_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_exists
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_not_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_udf_max
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_windowing
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_varchar_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_compile_processor
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constant_prop
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_func1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_disallow_incompatible_type_change_off
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_rdd_cache
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explode_null
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_cond_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_eq_with_case_when
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby7_map
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby7_map_multi_single_reducer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby7_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby7_noskew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby7_noskew_multi_single_reducer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby8_map
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby8_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby8_noskew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_complex_types
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_complex_types_multi_single_reducer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_id1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_position
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_resolution
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_having2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_self_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15040812#comment-15040812 ] Lefty Leverenz commented on HIVE-11358: --- Yes, +1 for all the configuration parameters. Now the light dawns. Thanks for the explanation, I was just being dunderheaded. > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch, > HIVE-11358.03.patch, HIVE-11358.04.patch, HIVE-11358.05.patch, > HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11166) HiveHBaseTableOutputFormat can't call getFileExtension(JobConf jc, boolean isCompressed, HiveOutputFormat hiveOutputFormat)
[
https://issues.apache.org/jira/browse/HIVE-11166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Zhao reassigned HIVE-11166:
---
Assignee: Yun Zhao
> HiveHBaseTableOutputFormat can't call getFileExtension(JobConf jc, boolean
> isCompressed, HiveOutputFormat hiveOutputFormat)
> -
>
> Key: HIVE-11166
> URL: https://issues.apache.org/jira/browse/HIVE-11166
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler, Spark
>Reporter: meiyoula
>Assignee: Yun Zhao
> Attachments: HIVE-11166.2.patch, HIVE-11166.patch
>
>
> I create a hbase table with HBaseStorageHandler in JDBCServer of spark, then
> execute the *insert into* sql statement, ClassCastException occurs.
> {quote}
> Error: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in
> stage 3.0 (TID 12, vm-17): java.lang.ClassCastException:
> org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat cannot be cast to
> org.apache.hadoop.hive.ql.io.HiveOutputFormat
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.outputFormat$lzycompute(hiveWriterContainers.scala:72)
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.outputFormat(hiveWriterContainers.scala:71)
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.getOutputName(hiveWriterContainers.scala:91)
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.initWriters(hiveWriterContainers.scala:115)
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.executorSideSetup(hiveWriterContainers.scala:84)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.org$apache$spark$sql$hive$execution$InsertIntoHiveTable$$writeToFile$1(InsertIntoHiveTable.scala:112)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:93)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:93)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:56)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> {quote}
> It's because the code in spark below. To hbase table, the outputFormat is
> HiveHBaseTableOutputFormat, it isn't instanceOf[HiveOutputForm
> at].
> {quote}
> @transient private lazy val
> outputFormat=conf.value.getOutputFormat.asInstanceOf[HiveOutputForm
> at[AnyRef, Writable]]
> val extension = Utilities.getFileExtension(conf.value,
> fileSinkConf.getCompressed, outputFormat)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11166) HiveHBaseTableOutputFormat can't call getFileExtension(JobConf jc, boolean isCompressed, HiveOutputFormat hiveOutputFormat)
[
https://issues.apache.org/jira/browse/HIVE-11166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Zhao updated HIVE-11166:
Attachment: HIVE-11166.2.patch
I modify HiveHBaseTableOutputFormat implements HiveOutputFormat instead of
OutputFormat
> HiveHBaseTableOutputFormat can't call getFileExtension(JobConf jc, boolean
> isCompressed, HiveOutputFormat hiveOutputFormat)
> -
>
> Key: HIVE-11166
> URL: https://issues.apache.org/jira/browse/HIVE-11166
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler, Spark
>Reporter: meiyoula
> Attachments: HIVE-11166.2.patch, HIVE-11166.patch
>
>
> I create a hbase table with HBaseStorageHandler in JDBCServer of spark, then
> execute the *insert into* sql statement, ClassCastException occurs.
> {quote}
> Error: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in
> stage 3.0 (TID 12, vm-17): java.lang.ClassCastException:
> org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat cannot be cast to
> org.apache.hadoop.hive.ql.io.HiveOutputFormat
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.outputFormat$lzycompute(hiveWriterContainers.scala:72)
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.outputFormat(hiveWriterContainers.scala:71)
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.getOutputName(hiveWriterContainers.scala:91)
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.initWriters(hiveWriterContainers.scala:115)
> at
> org.apache.spark.sql.hive.SparkHiveWriterContainer.executorSideSetup(hiveWriterContainers.scala:84)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.org$apache$spark$sql$hive$execution$InsertIntoHiveTable$$writeToFile$1(InsertIntoHiveTable.scala:112)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:93)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:93)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:56)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> {quote}
> It's because the code in spark below. To hbase table, the outputFormat is
> HiveHBaseTableOutputFormat, it isn't instanceOf[HiveOutputForm
> at].
> {quote}
> @transient private lazy val
> outputFormat=conf.value.getOutputFormat.asInstanceOf[HiveOutputForm
> at[AnyRef, Writable]]
> val extension = Utilities.getFileExtension(conf.value,
> fileSinkConf.getCompressed, outputFormat)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12529) HiveTxnManager.acquireLocks() should not block forever
[
https://issues.apache.org/jira/browse/HIVE-12529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15040438#comment-15040438
]
Hive QA commented on HIVE-12529:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775438/HIVE-12529.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6226/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6226/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6226/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6226/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 22fc397 HIVE-12444 Global Limit optimization on ACID table
without base directory may throw exception ADDENDUM (Wei Zheng via Eugene
Koifman)
+ git clean -f -d
Removing ql/src/test/queries/clientpositive/semijoin2.q
Removing ql/src/test/results/clientpositive/semijoin2.q.out
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 22fc397 HIVE-12444 Global Limit optimization on ACID table
without base directory may throw exception ADDENDUM (Wei Zheng via Eugene
Koifman)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775438 - PreCommit-HIVE-TRUNK-Build
> HiveTxnManager.acquireLocks() should not block forever
> --
>
> Key: HIVE-12529
> URL: https://issues.apache.org/jira/browse/HIVE-12529
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-12529.2.patch, HIVE-12529.4.patch,
> HIVE-12529.5.patch, HIVE-12529.patch
>
>
> Currently, in DbTxnManager this method will block until all competing locks
> have gone away.
> This is not appropriate for all clients. There should be a way to specify a
> max-wait-time.
> It will throw an exception on timeout (given how current method signature is
> written).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12578) Hive query failing with error ClassCastException org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc cannot be cast to org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc
[
https://issues.apache.org/jira/browse/HIVE-12578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15040109#comment-15040109
]
Hive QA commented on HIVE-12578:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775439/HIVE-12578.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 9875 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-dynpart_sort_optimization2.q-skewjoin_mapjoin4.q-list_bucket_dml_6.q-and-12-more
- did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_udf_max
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_unqual_corr_expr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_leftsemi_mapjoin
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_leftsemi_mapjoin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_semijoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6225/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6225/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6225/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775439 - PreCommit-HIVE-TRUNK-Build
> Hive query failing with error ClassCastException
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc cannot be cast to
> org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc
>
>
> Key: HIVE-12578
> URL: https://issues.apache.org/jira/browse/HIVE-12578
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-12578.patch
>
>
> Two tables:
> {noformat}
> CREATE TABLE table_1 (boolean_col_1 BOOLEAN, float_col_2 FLOAT, bigint_col_3
> BIGINT, varchar0111_col_4 VARCHAR(111), bigint_col_5 BIGINT, float_col_6
> FLOAT, boolean_col_7 BOOLEAN, decimal0101_col_8 DECIMAL(1, 1),
> decimal0904_col_9 DECIMAL(9, 4), char0112_col_10 CHAR(112), double_col_11
> DOUBLE, boolean_col_12 BOOLEAN, double_col_13 DOUBLE, varchar0142_col_14
> VARCHAR(142), timestamp_col_15 TIMESTAMP, decimal0502_col_16 DECIMAL(5, 2),
> smallint_col_25 SMALLINT, decimal3222_col_18 DECIMAL(32, 22), boolean_col_19
> BOOLEAN, decimal2012_col_20 DECIMAL(20, 12), char0204_col_21 CHAR(204),
> double_col_61 DOUBLE, timestamp_col_23 TIMESTAMP, int_col_24 INT,
> float_col_25 FLOAT, smallint_col_26 SMALLINT, double_col_27 DOUBLE,
> char0180_col_28 CHAR(180), decimal1503_col_29 DECIMAL(15, 3),
> timestamp_col_30 TIMESTAMP, smallint_col_31 SMALLINT, decimal2020_col_32
> DECIMAL(20, 20), timestamp_col_33 TIMESTAMP, boolean_col_34 BOOLEAN,
> decimal3025_col_35 DECIMAL(30, 25), decimal3117_col_36 DECIMAL(31, 17),
> timestamp_col_37 TIMESTAMP, varchar0146_col_38 VARCHAR(146), boolean_col_39
> BOOLEAN, double_col_40 DOUBLE, float_col_41 FLOAT, timestamp_col_42
> TIMESTAMP, double_col_43 DOUBLE, boolean_col_44 BOOLEAN, timestamp_col_45
> TIMESTAMP, tinyint_col_8 TINYINT, int_col_47 INT, decimal0401_col_48
> DECIMAL(4, 1), varchar0064_col_49 VARCHAR(64), string_col_50 STRING,
> double_col_51 DOUBLE, string_col_52 STRING, boolean_col_53 BOOLEAN,
> int_col_54 INT, boolean_col_55 BOOLEAN, string_col_56 STRING, double_col_57
> DOUBLE, varchar0131_col_58 VARCHAR(131), boolean_col_59 BOOLEAN,
> bigint_col_22 BIGINT, char0184_c
[jira] [Commented] (HIVE-12477) CBO: Left Semijoins are incompatible with a cross-product
[
https://issues.apache.org/jira/browse/HIVE-12477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15039720#comment-15039720
]
Hive QA commented on HIVE-12477:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775428/HIVE-12477.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 9859 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_decimal_10_0.q-vector_acid3.q-vector_decimal_trailing.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_decimal_round.q-cbo_windowing.q-tez_schema_evolution.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_udf_max
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.jdbc.miniHS2.TestHs2Metrics.testMetrics
org.apache.tez.dag.app.rm.TestLlapTaskSchedulerService.testSimpleLocalAllocation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6224/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6224/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6224/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775428 - PreCommit-HIVE-TRUNK-Build
> CBO: Left Semijoins are incompatible with a cross-product
> -
>
> Key: HIVE-12477
> URL: https://issues.apache.org/jira/browse/HIVE-12477
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.0.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-12477.01.patch, HIVE-12477.patch
>
>
> with HIVE-12017 in place, a few queries generate left sem-joins without a key.
> This is an invalid plan and can be produced by doing.
> {code}
> explain logical select count(1) from store_sales where ss_sold_date_sk in
> (select d_date_sk from date_dim where d_date_sk = 1);
> LOGICAL PLAN:
> $hdt$_0:$hdt$_0:$hdt$_0:store_sales
> TableScan (TS_0)
> alias: store_sales
> filterExpr: (ss_sold_date_sk = 1) (type: boolean)
> Filter Operator (FIL_20)
> predicate: (ss_sold_date_sk = 1) (type: boolean)
> Select Operator (SEL_2)
> Reduce Output Operator (RS_9)
> sort order:
> Join Operator (JOIN_11)
> condition map:
> Left Semi Join 0 to 1
> keys:
> 0
> 1
> Group By Operator (GBY_14)
> aggregations: count(1)
> mode: hash
> {code}
> without CBO
> {code}
> sq_1:date_d
[jira] [Updated] (HIVE-12591) LLAP cache counters displays -ve value for CacheCapacityUsed
[ https://issues.apache.org/jira/browse/HIVE-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-12591: Attachment: HIVE-12591.png > LLAP cache counters displays -ve value for CacheCapacityUsed > > > Key: HIVE-12591 > URL: https://issues.apache.org/jira/browse/HIVE-12591 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan > Attachments: HIVE-12591.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11526) LLAP: implement LLAP UI as a separate service
[ https://issues.apache.org/jira/browse/HIVE-11526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15039639#comment-15039639 ] Yuya OZAWA commented on HIVE-11526: --- Now status, running Monitoring web ui and Metrics Collector. Collector can get jmx metrics and it stored memory, but I will it use hadoop metrics2. Web service using hadoop webapp and diplay this metrics HTML table. And now I choosing plot library. > LLAP: implement LLAP UI as a separate service > - > > Key: HIVE-11526 > URL: https://issues.apache.org/jira/browse/HIVE-11526 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Sergey Shelukhin >Assignee: Yuya OZAWA > Attachments: llap_monitor_design.pdf > > > The specifics are vague at this point. > Hadoop metrics can be output, as well as metrics we collect and output in > jmx, as well as those we collect per fragment and log right now. > This service can do LLAP-specific views, and per-query aggregation. > [~gopalv] may have some information on how to reuse existing solutions for > part of the work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12499) Add HMS metrics for number of tables and partitions
[ https://issues.apache.org/jira/browse/HIVE-12499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-12499: - Target Version/s: 2.1.0 > Add HMS metrics for number of tables and partitions > --- > > Key: HIVE-12499 > URL: https://issues.apache.org/jira/browse/HIVE-12499 > Project: Hive > Issue Type: Sub-task > Components: Diagnosability >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-12499.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12499) Add HMS metrics for number of tables and partitions
[ https://issues.apache.org/jira/browse/HIVE-12499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15039570#comment-15039570 ] Szehon Ho commented on HIVE-12499: -- Setting target version. Review board : [https://reviews.apache.org/r/40948/] > Add HMS metrics for number of tables and partitions > --- > > Key: HIVE-12499 > URL: https://issues.apache.org/jira/browse/HIVE-12499 > Project: Hive > Issue Type: Sub-task > Components: Diagnosability >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-12499.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12499) Add HMS metrics for number of tables and partitions
[ https://issues.apache.org/jira/browse/HIVE-12499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-12499: - Attachment: HIVE-12499.patch > Add HMS metrics for number of tables and partitions > --- > > Key: HIVE-12499 > URL: https://issues.apache.org/jira/browse/HIVE-12499 > Project: Hive > Issue Type: Sub-task > Components: Diagnosability >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-12499.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9820) LLAP: Use a share-nothing scoreboard /status implementation
[ https://issues.apache.org/jira/browse/HIVE-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15039549#comment-15039549 ] Sergey Shelukhin commented on HIVE-9820: Assigned. Thank you! > LLAP: Use a share-nothing scoreboard /status implementation > --- > > Key: HIVE-9820 > URL: https://issues.apache.org/jira/browse/HIVE-9820 > Project: Hive > Issue Type: Sub-task > Components: llap >Affects Versions: llap >Reporter: Gopal V >Assignee: Yohei Abe > > To prevent thread-conflicts in executor information, the Apache HTTP servers > use a share-nothing data structure known as a scoreboard. > This is read by various systems like mod_status to read out the current state > of executors available for PHP (and similar mod_* engines). > The /status output is traditionally periodically read by the load-balancers > to route requests away from busy machines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9820) LLAP: Use a share-nothing scoreboard /status implementation
[ https://issues.apache.org/jira/browse/HIVE-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-9820: --- Assignee: Yohei Abe > LLAP: Use a share-nothing scoreboard /status implementation > --- > > Key: HIVE-9820 > URL: https://issues.apache.org/jira/browse/HIVE-9820 > Project: Hive > Issue Type: Sub-task > Components: llap >Affects Versions: llap >Reporter: Gopal V >Assignee: Yohei Abe > > To prevent thread-conflicts in executor information, the Apache HTTP servers > use a share-nothing data structure known as a scoreboard. > This is read by various systems like mod_status to read out the current state > of executors available for PHP (and similar mod_* engines). > The /status output is traditionally periodically read by the load-balancers > to route requests away from busy machines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12444) Global Limit optimization on ACID table without base directory may throw exception
[
https://issues.apache.org/jira/browse/HIVE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-12444:
-
Attachment: HIVE-12444.branch-1.ADDENDUM.patch
HIVE-12444.ADDENDUM.patch
Attaching two addendum patches for master and branch-1 to fix the Precommit
mismatches
> Global Limit optimization on ACID table without base directory may throw
> exception
> --
>
> Key: HIVE-12444
> URL: https://issues.apache.org/jira/browse/HIVE-12444
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 1.2.1
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-12444.1.patch, HIVE-12444.2.patch,
> HIVE-12444.3.patch, HIVE-12444.4.patch, HIVE-12444.5.patch,
> HIVE-12444.ADDENDUM.patch, HIVE-12444.branch-1.ADDENDUM.patch,
> HIVE-12444.branch-1.patch
>
>
> Steps to reproduce:
> set hive.fetch.task.conversion=minimal;
> set hive.limit.optimize.enable=true;
> create table acidtest1(
> c_custkey int,
> c_name string,
> c_nationkey int,
> c_acctbal double)
> clustered by (c_nationkey) into 3 buckets
> stored as orc
> tblproperties("transactional"="true");
> insert into table acidtest1
> select c_custkey, c_name, c_nationkey, c_acctbal from tpch_text_10.customer;
> select cast (c_nationkey as string) from acidtest.acidtest1 limit 10;
> {code}
> DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
> vertexId=vertex_1447362491939_0020_1_00, diagnostics=[Vertex
> vertex_1447362491939_0020_1_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: acidtest1 initializer failed,
> vertex=vertex_1447362491939_0020_1_00 [Map 1], java.lang.RuntimeException:
> serious problem
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1035)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1062)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:308)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:410)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:155)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:246)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:240)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:240)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:227)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.IllegalArgumentException: delta_017_017 does not start with
> base_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1012)
> ... 15 more
> Caused by: java.lang.IllegalArgumentException: delta_017_017 does not
> start with base_
> at org.apache.hadoop.hive.ql.io.AcidUtils.parseBase(AcidUtils.java:144)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.parseBaseBucketFilename(AcidUtils.java:172)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:667)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:625)
> ... 4 more
> ]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12574) windowing function returns incorrect result when the window size is larger than the partition size
[
https://issues.apache.org/jira/browse/HIVE-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15039540#comment-15039540
]
Hive QA commented on HIVE-12574:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775583/HIVE-12574.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 9863 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_udf_max
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetricsFile
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccessWithReadOnly
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6223/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6223/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6223/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775583 - PreCommit-HIVE-TRUNK-Build
> windowing function returns incorrect result when the window size is larger
> than the partition size
> --
>
> Key: HIVE-12574
> URL: https://issues.apache.org/jira/browse/HIVE-12574
> Project: Hive
> Issue Type: Sub-task
> Components: PTF-Windowing
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-12574.2.patch, HIVE-12574.patch
>
>
> In PTF windowing, when the partition is small and the window size is larger
> than the partition size, we are seeing incorrect result. It happens for max,
> min, first_value, last_value and sum functions.
> {noformat}
> CREATE TABLE sdy1(
> ord int,
> type string);
> {noformat}
> The data is:
> {noformat}
> 2 a
> 3 a
> 1 a
> {noformat}
> The result is as follows for the query {{select ord, min(ord) over (partition
> by type order by ord rows between 1 preceding and 7 following)}}
> {noformat}
> 1 1
> 2 1
> 3 1
> {noformat}
> The expected result is:
> {noformat}
> 1 1
> 2 1
> 3 2
> {noformat}
--
This message was sent
[jira] [Updated] (HIVE-11531) Add mysql-style LIMIT support to Hive, or improve ROW_NUMBER performance-wise
[ https://issues.apache.org/jira/browse/HIVE-11531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11531: Attachment: HIVE-11531.04.patch Uploading the same patch for HiveQA > Add mysql-style LIMIT support to Hive, or improve ROW_NUMBER performance-wise > - > > Key: HIVE-11531 > URL: https://issues.apache.org/jira/browse/HIVE-11531 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Sergey Shelukhin >Assignee: Hui Zheng > Attachments: HIVE-11531.02.patch, HIVE-11531.03.patch, > HIVE-11531.04.patch, HIVE-11531.WIP.1.patch, HIVE-11531.WIP.2.patch, > HIVE-11531.patch > > > For any UIs that involve pagination, it is useful to issue queries in the > form SELECT ... LIMIT X,Y where X,Y are coordinates inside the result to be > paginated (which can be extremely large by itself). At present, ROW_NUMBER > can be used to achieve this effect, but optimizations for LIMIT such as TopN > in ReduceSink do not apply to ROW_NUMBER. We can add first class support for > "skip" to existing limit, or improve ROW_NUMBER for better performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11107) Support for Performance regression test suite with TPCDS
[
https://issues.apache.org/jira/browse/HIVE-11107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15039525#comment-15039525
]
Ashutosh Chauhan commented on HIVE-11107:
-
1. Why do we have sql scripts for stats copied in? They should be sourced from
original location which is in metastore/scripts/derby/
2. Please add comments in TestPerfCliDriver.vm on how is it different from
TestCliDriver.vm
3. I also dont see any changes in ptest2/ How are we making sure QA will pick
this new Driver.
> Support for Performance regression test suite with TPCDS
>
>
> Key: HIVE-11107
> URL: https://issues.apache.org/jira/browse/HIVE-11107
> Project: Hive
> Issue Type: Sub-task
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-11107.1.patch, HIVE-11107.2.patch,
> HIVE-11107.3.patch, HIVE-11107.4.patch, HIVE-11107.5.patch
>
>
> Support to add TPCDS queries to the performance regression test suite with
> Hive CBO turned on.
> This benchmark is intended to make sure that subsequent changes to the
> optimizer or any hive code do not yield any unexpected plan changes. i.e.
> the intention is to not run the entire TPCDS query set, but just "explain
> plan" for the TPCDS queries.
> As part of this jira, we will manually verify that expected hive
> optimizations kick in for the queries (for given stats/dataset). If there is
> a difference in plan within this test suite due to a future commit, it needs
> to be analyzed and we need to make sure that it is not a regression.
> The test suite can be run in master branch from itests by
> {code}
> mvn test -Dtest=TestPerfCliDriver
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12566) Incorrect result returns when using COALESCE in WHERE condition with LEFT JOIN
[
https://issues.apache.org/jira/browse/HIVE-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15039518#comment-15039518
]
Ashutosh Chauhan commented on HIVE-12566:
-
[~jcamachorodriguez] was looking at this piece of code recently. Jesus, would
you also like to review this one?
> Incorrect result returns when using COALESCE in WHERE condition with LEFT JOIN
> --
>
> Key: HIVE-12566
> URL: https://issues.apache.org/jira/browse/HIVE-12566
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 0.13.0
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-12566.patch
>
>
> The left join query with on/where clause returns incorrect result (more rows
> are returned). See the reproducible sample below.
> Left table with data:
> {code}
> CREATE TABLE ltable (i int, la int, lk1 string, lk2 string) ROW FORMAT
> DELIMITED FIELDS TERMINATED BY ',';
> ---
> 1,\N,CD5415192314304,00071
> 2,\N,CD5415192225530,00071
> {code}
> Right table with data:
> {code}
> CREATE TABLE rtable (ra int, rk1 string, rk2 string) ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ',';
> ---
> 1,CD5415192314304,00071
> 45,CD5415192314304,00072
> {code}
> Query:
> {code}
> SELECT * FROM ltable l LEFT OUTER JOIN rtable r on (l.lk1 = r.rk1 AND l.lk2 =
> r.rk2) WHERE COALESCE(l.la,'EMPTY')=COALESCE(r.ra,'EMPTY');
> {code}
> Result returns:
> {code}
> 1 NULLCD5415192314304 00071 NULLNULLNULL
> 2 NULLCD5415192225530 00071 NULLNULLNULL
> {code}
> The correct result should be
> {code}
> 2 NULLCD5415192225530 00071 NULLNULLNULL
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12589) LLAP service gen is broken for log4j2
[ https://issues.apache.org/jira/browse/HIVE-12589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15039514#comment-15039514 ] Prasanth Jayachandran commented on HIVE-12589: -- +1 > LLAP service gen is broken for log4j2 > - > > Key: HIVE-12589 > URL: https://issues.apache.org/jira/browse/HIVE-12589 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12589.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12589) LLAP service gen is broken for log4j2
[ https://issues.apache.org/jira/browse/HIVE-12589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12589: Attachment: HIVE-12589.patch [~prasanth_j] can you take a look? log4j file name is incorrect. I also added error logging to console since logs don't appear to be generated for this service. > LLAP service gen is broken for log4j2 > - > > Key: HIVE-12589 > URL: https://issues.apache.org/jira/browse/HIVE-12589 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12589.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12589) LLAP service gen is broken for log4j2
[ https://issues.apache.org/jira/browse/HIVE-12589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12589: Target Version/s: 2.0.0 > LLAP service gen is broken for log4j2 > - > > Key: HIVE-12589 > URL: https://issues.apache.org/jira/browse/HIVE-12589 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12589.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HIVE-12444) Global Limit optimization on ACID table without base directory may throw exception
[
https://issues.apache.org/jira/browse/HIVE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng reopened HIVE-12444:
--
There's some Precommit test issue. The golden files need to be updated.
> Global Limit optimization on ACID table without base directory may throw
> exception
> --
>
> Key: HIVE-12444
> URL: https://issues.apache.org/jira/browse/HIVE-12444
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 1.2.1
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-12444.1.patch, HIVE-12444.2.patch,
> HIVE-12444.3.patch, HIVE-12444.4.patch, HIVE-12444.5.patch,
> HIVE-12444.branch-1.patch
>
>
> Steps to reproduce:
> set hive.fetch.task.conversion=minimal;
> set hive.limit.optimize.enable=true;
> create table acidtest1(
> c_custkey int,
> c_name string,
> c_nationkey int,
> c_acctbal double)
> clustered by (c_nationkey) into 3 buckets
> stored as orc
> tblproperties("transactional"="true");
> insert into table acidtest1
> select c_custkey, c_name, c_nationkey, c_acctbal from tpch_text_10.customer;
> select cast (c_nationkey as string) from acidtest.acidtest1 limit 10;
> {code}
> DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
> vertexId=vertex_1447362491939_0020_1_00, diagnostics=[Vertex
> vertex_1447362491939_0020_1_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: acidtest1 initializer failed,
> vertex=vertex_1447362491939_0020_1_00 [Map 1], java.lang.RuntimeException:
> serious problem
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1035)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1062)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:308)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:410)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:155)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:246)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:240)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:240)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:227)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.IllegalArgumentException: delta_017_017 does not start with
> base_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1012)
> ... 15 more
> Caused by: java.lang.IllegalArgumentException: delta_017_017 does not
> start with base_
> at org.apache.hadoop.hive.ql.io.AcidUtils.parseBase(AcidUtils.java:144)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.parseBaseBucketFilename(AcidUtils.java:172)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:667)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:625)
> ... 4 more
> ]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12586) Support for Performance regression test suite
[
https://issues.apache.org/jira/browse/HIVE-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-12586:
-
Description:
Umbrella jira to add TestPerfCliDriver to the existing test framework
The test suite can be run in master branch from itests by
{code}
mvn test -Dtest=TestPerfCliDriver
{code}
was:Umbrella jira to add TestPerfCliDriver to the existing test framework
> Support for Performance regression test suite
> -
>
> Key: HIVE-12586
> URL: https://issues.apache.org/jira/browse/HIVE-12586
> Project: Hive
> Issue Type: Task
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
>
> Umbrella jira to add TestPerfCliDriver to the existing test framework
> The test suite can be run in master branch from itests by
> {code}
> mvn test -Dtest=TestPerfCliDriver
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11107) Support for Performance regression test suite with TPCDS
[
https://issues.apache.org/jira/browse/HIVE-11107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-11107:
-
Issue Type: Sub-task (was: Task)
Parent: HIVE-12586
> Support for Performance regression test suite with TPCDS
>
>
> Key: HIVE-11107
> URL: https://issues.apache.org/jira/browse/HIVE-11107
> Project: Hive
> Issue Type: Sub-task
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-11107.1.patch, HIVE-11107.2.patch,
> HIVE-11107.3.patch, HIVE-11107.4.patch, HIVE-11107.5.patch
>
>
> Support to add TPCDS queries to the performance regression test suite with
> Hive CBO turned on.
> This benchmark is intended to make sure that subsequent changes to the
> optimizer or any hive code do not yield any unexpected plan changes. i.e.
> the intention is to not run the entire TPCDS query set, but just "explain
> plan" for the TPCDS queries.
> As part of this jira, we will manually verify that expected hive
> optimizations kick in for the queries (for given stats/dataset). If there is
> a difference in plan within this test suite due to a future commit, it needs
> to be analyzed and we need to make sure that it is not a regression.
> The test suite can be run in master branch from itests by
> {code}
> mvn test -Dtest=TestPerfCliDriver
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11107) Support for Performance regression test suite with TPCDS
[
https://issues.apache.org/jira/browse/HIVE-11107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-11107:
-
Description:
Support to add TPCDS queries to the performance regression test suite with Hive
CBO turned on.
This benchmark is intended to make sure that subsequent changes to the
optimizer or any hive code do not yield any unexpected plan changes. i.e. the
intention is to not run the entire TPCDS query set, but just "explain plan" for
the TPCDS queries.
As part of this jira, we will manually verify that expected hive optimizations
kick in for the queries (for given stats/dataset). If there is a difference in
plan within this test suite due to a future commit, it needs to be analyzed and
we need to make sure that it is not a regression.
The test suite can be run in master branch from itests by
{code}
mvn test -Dtest=TestPerfCliDriver
{code}
was:
Support to add TPCDS queries to the performance regression test suite with Hive
CBO turned on.
This benchmark is intended to make sure that subsequent changes to the
optimizer or any hive code do not yield any unexpected plan changes. i.e. the
intention is to not run the entire TPCDS query set, but just "explain plan" for
the TPCDS queries.
As part of this jira, we will manually verify that expected hive optimizations
kick in for the queries (for given stats/dataset). If there is a difference in
plan within this test suite due to a future commit, it needs to be analyzed and
we need to make sure that it is not a regression.
The test suite can be run in master branch from itests by
{code}
mvn test -Dtest=TestPerfCliDriver -Phadoop-2
{code}
> Support for Performance regression test suite with TPCDS
>
>
> Key: HIVE-11107
> URL: https://issues.apache.org/jira/browse/HIVE-11107
> Project: Hive
> Issue Type: Task
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-11107.1.patch, HIVE-11107.2.patch,
> HIVE-11107.3.patch, HIVE-11107.4.patch, HIVE-11107.5.patch
>
>
> Support to add TPCDS queries to the performance regression test suite with
> Hive CBO turned on.
> This benchmark is intended to make sure that subsequent changes to the
> optimizer or any hive code do not yield any unexpected plan changes. i.e.
> the intention is to not run the entire TPCDS query set, but just "explain
> plan" for the TPCDS queries.
> As part of this jira, we will manually verify that expected hive
> optimizations kick in for the queries (for given stats/dataset). If there is
> a difference in plan within this test suite due to a future commit, it needs
> to be analyzed and we need to make sure that it is not a regression.
> The test suite can be run in master branch from itests by
> {code}
> mvn test -Dtest=TestPerfCliDriver
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11107) Support for Performance regression test suite with TPCDS
[
https://issues.apache.org/jira/browse/HIVE-11107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-11107:
-
Attachment: (was: HIVE-11107.5.patch)
> Support for Performance regression test suite with TPCDS
>
>
> Key: HIVE-11107
> URL: https://issues.apache.org/jira/browse/HIVE-11107
> Project: Hive
> Issue Type: Task
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-11107.1.patch, HIVE-11107.2.patch,
> HIVE-11107.3.patch, HIVE-11107.4.patch, HIVE-11107.5.patch
>
>
> Support to add TPCDS queries to the performance regression test suite with
> Hive CBO turned on.
> This benchmark is intended to make sure that subsequent changes to the
> optimizer or any hive code do not yield any unexpected plan changes. i.e.
> the intention is to not run the entire TPCDS query set, but just "explain
> plan" for the TPCDS queries.
> As part of this jira, we will manually verify that expected hive
> optimizations kick in for the queries (for given stats/dataset). If there is
> a difference in plan within this test suite due to a future commit, it needs
> to be analyzed and we need to make sure that it is not a regression.
> The test suite can be run in master branch from itests by
> {code}
> mvn test -Dtest=TestPerfCliDriver -Phadoop-2
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11107) Support for Performance regression test suite with TPCDS
[
https://issues.apache.org/jira/browse/HIVE-11107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-11107:
-
Attachment: HIVE-11107.5.patch
> Support for Performance regression test suite with TPCDS
>
>
> Key: HIVE-11107
> URL: https://issues.apache.org/jira/browse/HIVE-11107
> Project: Hive
> Issue Type: Task
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-11107.1.patch, HIVE-11107.2.patch,
> HIVE-11107.3.patch, HIVE-11107.4.patch, HIVE-11107.5.patch
>
>
> Support to add TPCDS queries to the performance regression test suite with
> Hive CBO turned on.
> This benchmark is intended to make sure that subsequent changes to the
> optimizer or any hive code do not yield any unexpected plan changes. i.e.
> the intention is to not run the entire TPCDS query set, but just "explain
> plan" for the TPCDS queries.
> As part of this jira, we will manually verify that expected hive
> optimizations kick in for the queries (for given stats/dataset). If there is
> a difference in plan within this test suite due to a future commit, it needs
> to be analyzed and we need to make sure that it is not a regression.
> The test suite can be run in master branch from itests by
> {code}
> mvn test -Dtest=TestPerfCliDriver -Phadoop-2
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12566) Incorrect result returns when using COALESCE in WHERE condition with LEFT JOIN
[
https://issues.apache.org/jira/browse/HIVE-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038861#comment-15038861
]
Xuefu Zhang commented on HIVE-12566:
+1, but let's see what the precommit-test will say.
> Incorrect result returns when using COALESCE in WHERE condition with LEFT JOIN
> --
>
> Key: HIVE-12566
> URL: https://issues.apache.org/jira/browse/HIVE-12566
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 0.13.0
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
>Priority: Critical
> Attachments: HIVE-12566.patch
>
>
> The left join query with on/where clause returns incorrect result (more rows
> are returned). See the reproducible sample below.
> Left table with data:
> {code}
> CREATE TABLE ltable (i int, la int, lk1 string, lk2 string) ROW FORMAT
> DELIMITED FIELDS TERMINATED BY ',';
> ---
> 1,\N,CD5415192314304,00071
> 2,\N,CD5415192225530,00071
> {code}
> Right table with data:
> {code}
> CREATE TABLE rtable (ra int, rk1 string, rk2 string) ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ',';
> ---
> 1,CD5415192314304,00071
> 45,CD5415192314304,00072
> {code}
> Query:
> {code}
> SELECT * FROM ltable l LEFT OUTER JOIN rtable r on (l.lk1 = r.rk1 AND l.lk2 =
> r.rk2) WHERE COALESCE(l.la,'EMPTY')=COALESCE(r.ra,'EMPTY');
> {code}
> Result returns:
> {code}
> 1 NULLCD5415192314304 00071 NULLNULLNULL
> 2 NULLCD5415192225530 00071 NULLNULLNULL
> {code}
> The correct result should be
> {code}
> 2 NULLCD5415192225530 00071 NULLNULLNULL
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12547) VectorMapJoinFastBytesHashTable fails during expansion
[
https://issues.apache.org/jira/browse/HIVE-12547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038855#comment-15038855
]
Sergey Shelukhin commented on HIVE-12547:
-
[~mmccline] is there a reason why 3 longs are used per key in the Fast
hashtable? Full hashcode and valueref are in separate longs, I wonder if
combining them to have last few bits of the hash will improve perf overall for
the cases where HT is big enough to fit with 2 longs but not 3? The only loss
will be some extra lookups when last hash bits match but the rest don't.
> VectorMapJoinFastBytesHashTable fails during expansion
> --
>
> Key: HIVE-12547
> URL: https://issues.apache.org/jira/browse/HIVE-12547
> Project: Hive
> Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
>Priority: Critical
>
> {code}
> 2015-11-30 20:55:30,361 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1448429572030_1224_7][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=Map 2, taskAttemptId=attempt_1448429572030_1224_7_03_05_0,
> creationTime=1448934722881, allocationTime=1448934726552,
> startTime=1448934726553, finishTime=1448934930360, timeTaken=203807,
> status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while
> running task:
> attempt_1448429572030_1224_7_03_05_0:java.lang.RuntimeException:
> java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:289)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Async
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:424)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:394)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:519)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:472)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:274)
> ... 15 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NegativeArraySizeException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:414)
> ... 20 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:106)
> at
> org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache$1.call(LlapObjectCache.java:131)
> ... 4 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoade
[jira] [Commented] (HIVE-12573) some DPP tests are broken
[ https://issues.apache.org/jira/browse/HIVE-12573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038848#comment-15038848 ] Sergey Shelukhin commented on HIVE-12573: - Test failures are unrelated and DPP tests no longer fail. [~hagleitn] ping? > some DPP tests are broken > - > > Key: HIVE-12573 > URL: https://issues.apache.org/jira/browse/HIVE-12573 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12573.patch > > > -It looks like LLAP out files were not updated in some DPP JIRA because the > test was entirely broken in HiveQA at the time- actually looks like out files > have explain output with a glitch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12547) VectorMapJoinFastBytesHashTable fails during expansion
[
https://issues.apache.org/jira/browse/HIVE-12547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-12547:
---
Assignee: Sergey Shelukhin
> VectorMapJoinFastBytesHashTable fails during expansion
> --
>
> Key: HIVE-12547
> URL: https://issues.apache.org/jira/browse/HIVE-12547
> Project: Hive
> Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
>Priority: Critical
>
> {code}
> 2015-11-30 20:55:30,361 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1448429572030_1224_7][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=Map 2, taskAttemptId=attempt_1448429572030_1224_7_03_05_0,
> creationTime=1448934722881, allocationTime=1448934726552,
> startTime=1448934726553, finishTime=1448934930360, timeTaken=203807,
> status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while
> running task:
> attempt_1448429572030_1224_7_03_05_0:java.lang.RuntimeException:
> java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:289)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Async
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:424)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:394)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:519)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:472)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:274)
> ... 15 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NegativeArraySizeException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:414)
> ... 20 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:106)
> at
> org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache$1.call(LlapObjectCache.java:131)
> ... 4 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:110)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:293)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:174)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:170)
> at
> org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:10
[jira] [Commented] (HIVE-12573) some DPP tests are broken
[
https://issues.apache.org/jira/browse/HIVE-12573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038835#comment-15038835
]
Hive QA commented on HIVE-12573:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775423/HIVE-12573.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 9885 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_udf_max
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testSparkQuery
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.jdbc.miniHS2.TestHs2Metrics.testMetrics
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6222/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6222/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6222/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775423 - PreCommit-HIVE-TRUNK-Build
> some DPP tests are broken
> -
>
> Key: HIVE-12573
> URL: https://issues.apache.org/jira/browse/HIVE-12573
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12573.patch
>
>
> -It looks like LLAP out files were not updated in some DPP JIRA because the
> test was entirely broken in HiveQA at the time- actually looks like out files
> have explain output with a glitch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12257) Enhance ORC FileDump utility to handle flush_length files and recovery
[ https://issues.apache.org/jira/browse/HIVE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038811#comment-15038811 ] Prasanth Jayachandran commented on HIVE-12257: -- Committed to master, branch-2.0 and branch-1. > Enhance ORC FileDump utility to handle flush_length files and recovery > -- > > Key: HIVE-12257 > URL: https://issues.apache.org/jira/browse/HIVE-12257 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-12257-branch-1.patch, HIVE-12257.1.patch, > HIVE-12257.10.patch, HIVE-12257.11.patch, HIVE-12257.12.patch, > HIVE-12257.2.patch, HIVE-12257.3.patch, HIVE-12257.4.patch, > HIVE-12257.6.patch, HIVE-12257.7.patch, HIVE-12257.8.patch, HIVE-12257.9.patch > > > ORC file dump utility currently does not handle delta directories that > contain *_flush_length files. These files contains offsets to footer in the > corresponding delta file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12257) Enhance ORC FileDump utility to handle flush_length files and recovery
[ https://issues.apache.org/jira/browse/HIVE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12257: - Attachment: (was: HIVE-12257-branch-1.patch) > Enhance ORC FileDump utility to handle flush_length files and recovery > -- > > Key: HIVE-12257 > URL: https://issues.apache.org/jira/browse/HIVE-12257 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12257-branch-1.patch, HIVE-12257.1.patch, > HIVE-12257.10.patch, HIVE-12257.11.patch, HIVE-12257.12.patch, > HIVE-12257.2.patch, HIVE-12257.3.patch, HIVE-12257.4.patch, > HIVE-12257.6.patch, HIVE-12257.7.patch, HIVE-12257.8.patch, HIVE-12257.9.patch > > > ORC file dump utility currently does not handle delta directories that > contain *_flush_length files. These files contains offsets to footer in the > corresponding delta file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12257) Enhance ORC FileDump utility to handle flush_length files and recovery
[ https://issues.apache.org/jira/browse/HIVE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12257: - Attachment: HIVE-12257-branch-1.patch > Enhance ORC FileDump utility to handle flush_length files and recovery > -- > > Key: HIVE-12257 > URL: https://issues.apache.org/jira/browse/HIVE-12257 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12257-branch-1.patch, HIVE-12257-branch-1.patch, > HIVE-12257.1.patch, HIVE-12257.10.patch, HIVE-12257.11.patch, > HIVE-12257.12.patch, HIVE-12257.2.patch, HIVE-12257.3.patch, > HIVE-12257.4.patch, HIVE-12257.6.patch, HIVE-12257.7.patch, > HIVE-12257.8.patch, HIVE-12257.9.patch > > > ORC file dump utility currently does not handle delta directories that > contain *_flush_length files. These files contains offsets to footer in the > corresponding delta file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12341) LLAP: add security to daemon protocol endpoint (excluding shuffle)
[ https://issues.apache.org/jira/browse/HIVE-12341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038799#comment-15038799 ] Sergey Shelukhin commented on HIVE-12341: - [~sseth] https://reviews.apache.org/r/40315/diff/2-8/ are the changes since you last reviewed I think... or maybe https://reviews.apache.org/r/40315/diff/5-8/ Main changes are - LlapIoProxy was renamed, protocols were split, and the change to how the token is propagated to Tez AM (via credentials rather than a config string). > LLAP: add security to daemon protocol endpoint (excluding shuffle) > -- > > Key: HIVE-12341 > URL: https://issues.apache.org/jira/browse/HIVE-12341 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12341.01.patch, HIVE-12341.02.patch, > HIVE-12341.03.patch, HIVE-12341.03.patch, HIVE-12341.04.patch, > HIVE-12341.05.patch, HIVE-12341.06.patch, HIVE-12341.07.patch, > HIVE-12341.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12431) Support timeout for compile lock
[ https://issues.apache.org/jira/browse/HIVE-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-12431: --- Attachment: HIVE-12431.2.patch > Support timeout for compile lock > > > Key: HIVE-12431 > URL: https://issues.apache.org/jira/browse/HIVE-12431 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Query Processor >Affects Versions: 1.2.1 >Reporter: Lenni Kuff >Assignee: Mohit Sabharwal > Attachments: HIVE-12431.1.patch, HIVE-12431.2.patch, HIVE-12431.patch > > > To help with HiveServer2 scalability, it would be useful to allow users to > configure a timeout value for queries waiting to be compiled. If the timeout > value is reached then the query would abort. One option to achieve this would > be to update the compile lock to use a try-lock with the timeout value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12431) Support timeout for compile lock
[ https://issues.apache.org/jira/browse/HIVE-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-12431: --- Summary: Support timeout for compile lock (was: Support timeout for global compile lock) > Support timeout for compile lock > > > Key: HIVE-12431 > URL: https://issues.apache.org/jira/browse/HIVE-12431 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Query Processor >Affects Versions: 1.2.1 >Reporter: Lenni Kuff >Assignee: Mohit Sabharwal > Attachments: HIVE-12431.1.patch, HIVE-12431.patch > > > To help with HiveServer2 scalability, it would be useful to allow users to > configure a timeout value for queries waiting to be compiled. If the timeout > value is reached then the query would abort. One option to achieve this would > be to update the compile lock to use a try-lock with the timeout value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9642) Hive metastore client retries don't happen consistently for all api calls
[ https://issues.apache.org/jira/browse/HIVE-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-9642: - Attachment: HIVE-9642.6.patch > Hive metastore client retries don't happen consistently for all api calls > - > > Key: HIVE-9642 > URL: https://issues.apache.org/jira/browse/HIVE-9642 > Project: Hive > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Xiaobing Zhou >Assignee: Daniel Dai > Attachments: HIVE-9642.1.patch, HIVE-9642.2.patch, HIVE-9642.3.patch, > HIVE-9642.4.patch, HIVE-9642.5.patch, HIVE-9642.5.patch, HIVE-9642.6.patch > > > When org.apache.thrift.transport.TTransportException is thrown for issues > like socket timeout, the retry via RetryingMetaStoreClient happens only in > certain cases. > Retry happens for the getDatabase call in but not for getAllDatabases(). > The reason is RetryingMetaStoreClient checks for TTransportException being > the cause for InvocationTargetException. But in case of some calls such as > getAllDatabases in HiveMetastoreClient, all exceptions get wrapped in a > MetaException. We should remove this unnecessary wrapping of exceptions for > certain functions in HMC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12431) Support timeout for global compile lock
[ https://issues.apache.org/jira/browse/HIVE-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038789#comment-15038789 ] Mohit Sabharwal commented on HIVE-12431: Also, changed the patch to apply the timeout to both global and session-level locks. > Support timeout for global compile lock > --- > > Key: HIVE-12431 > URL: https://issues.apache.org/jira/browse/HIVE-12431 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Query Processor >Affects Versions: 1.2.1 >Reporter: Lenni Kuff >Assignee: Mohit Sabharwal > Attachments: HIVE-12431.1.patch, HIVE-12431.patch > > > To help with HiveServer2 scalability, it would be useful to allow users to > configure a timeout value for queries waiting to be compiled. If the timeout > value is reached then the query would abort. One option to achieve this would > be to update the compile lock to use a try-lock with the timeout value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12431) Support timeout for global compile lock
[ https://issues.apache.org/jira/browse/HIVE-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038786#comment-15038786 ] Mohit Sabharwal commented on HIVE-12431: Thanks, [~sershe]! Changed patch to use compile-time hook instead. > Support timeout for global compile lock > --- > > Key: HIVE-12431 > URL: https://issues.apache.org/jira/browse/HIVE-12431 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Query Processor >Affects Versions: 1.2.1 >Reporter: Lenni Kuff >Assignee: Mohit Sabharwal > Attachments: HIVE-12431.1.patch, HIVE-12431.patch > > > To help with HiveServer2 scalability, it would be useful to allow users to > configure a timeout value for queries waiting to be compiled. If the timeout > value is reached then the query would abort. One option to achieve this would > be to update the compile lock to use a try-lock with the timeout value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12341) LLAP: add security to daemon protocol endpoint (excluding shuffle)
[ https://issues.apache.org/jira/browse/HIVE-12341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12341: Attachment: HIVE-12341.07.patch Fixing the bugs in the latest patch based on cluster testing. > LLAP: add security to daemon protocol endpoint (excluding shuffle) > -- > > Key: HIVE-12341 > URL: https://issues.apache.org/jira/browse/HIVE-12341 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12341.01.patch, HIVE-12341.02.patch, > HIVE-12341.03.patch, HIVE-12341.03.patch, HIVE-12341.04.patch, > HIVE-12341.05.patch, HIVE-12341.06.patch, HIVE-12341.07.patch, > HIVE-12341.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12431) Support timeout for global compile lock
[ https://issues.apache.org/jira/browse/HIVE-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-12431: --- Attachment: HIVE-12431.1.patch > Support timeout for global compile lock > --- > > Key: HIVE-12431 > URL: https://issues.apache.org/jira/browse/HIVE-12431 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Query Processor >Affects Versions: 1.2.1 >Reporter: Lenni Kuff >Assignee: Mohit Sabharwal > Attachments: HIVE-12431.1.patch, HIVE-12431.patch > > > To help with HiveServer2 scalability, it would be useful to allow users to > configure a timeout value for queries waiting to be compiled. If the timeout > value is reached then the query would abort. One option to achieve this would > be to update the compile lock to use a try-lock with the timeout value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11530) push limit thru outer join
[ https://issues.apache.org/jira/browse/HIVE-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038775#comment-15038775 ] Jesus Camacho Rodriguez commented on HIVE-11530: We have enabled CBO path even when stats are not present for Hive-2.0.0. Thus, now that HIVE-11684 went in, I think there is no need to implement the same rule in the non-CBO path, unless somebody feels the urge to do it. > push limit thru outer join > -- > > Key: HIVE-11530 > URL: https://issues.apache.org/jira/browse/HIVE-11530 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Reporter: Sergey Shelukhin >Assignee: Yohei Abe > > When the query has a left or right outer join with limit, we can push the > limit into the left/right side of the join. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12257) Enhance ORC FileDump utility to handle flush_length files and recovery
[ https://issues.apache.org/jira/browse/HIVE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12257: - Attachment: HIVE-12257.12.patch Patch committed to master and branch-2.0. Still working on branch-1 patch. > Enhance ORC FileDump utility to handle flush_length files and recovery > -- > > Key: HIVE-12257 > URL: https://issues.apache.org/jira/browse/HIVE-12257 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12257-branch-1.patch, HIVE-12257.1.patch, > HIVE-12257.10.patch, HIVE-12257.11.patch, HIVE-12257.12.patch, > HIVE-12257.2.patch, HIVE-12257.3.patch, HIVE-12257.4.patch, > HIVE-12257.6.patch, HIVE-12257.7.patch, HIVE-12257.8.patch, HIVE-12257.9.patch > > > ORC file dump utility currently does not handle delta directories that > contain *_flush_length files. These files contains offsets to footer in the > corresponding delta file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038725#comment-15038725 ] Jimmy Xiang commented on HIVE-12538: Currently, each hive session can have just one spark session. Yeah, if a hive session is allowed to have more than one spark session, this should work. > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-12538.1.patch, HIVE-12538.2.patch, > HIVE-12538.3.patch, HIVE-12538.4.patch, HIVE-12538.patch > > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11107) Support for Performance regression test suite with TPCDS
[
https://issues.apache.org/jira/browse/HIVE-11107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-11107:
-
Attachment: HIVE-11107.5.patch
> Support for Performance regression test suite with TPCDS
>
>
> Key: HIVE-11107
> URL: https://issues.apache.org/jira/browse/HIVE-11107
> Project: Hive
> Issue Type: Task
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-11107.1.patch, HIVE-11107.2.patch,
> HIVE-11107.3.patch, HIVE-11107.4.patch, HIVE-11107.5.patch
>
>
> Support to add TPCDS queries to the performance regression test suite with
> Hive CBO turned on.
> This benchmark is intended to make sure that subsequent changes to the
> optimizer or any hive code do not yield any unexpected plan changes. i.e.
> the intention is to not run the entire TPCDS query set, but just "explain
> plan" for the TPCDS queries.
> As part of this jira, we will manually verify that expected hive
> optimizations kick in for the queries (for given stats/dataset). If there is
> a difference in plan within this test suite due to a future commit, it needs
> to be analyzed and we need to make sure that it is not a regression.
> The test suite can be run in master branch from itests by
> {code}
> mvn test -Dtest=TestPerfCliDriver -Phadoop-2
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12471) Secure HS2 web UI with SSL
[ https://issues.apache.org/jira/browse/HIVE-12471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038689#comment-15038689 ] Jimmy Xiang commented on HIVE-12471: Backported to branch 2. > Secure HS2 web UI with SSL > -- > > Key: HIVE-12471 > URL: https://issues.apache.org/jira/browse/HIVE-12471 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 2.0.0, 2.1.0 > > Attachments: HIVE-12471.1.patch, HIVE-12471.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12471) Secure HS2 web UI with SSL
[ https://issues.apache.org/jira/browse/HIVE-12471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-12471: --- Fix Version/s: 2.0.0 > Secure HS2 web UI with SSL > -- > > Key: HIVE-12471 > URL: https://issues.apache.org/jira/browse/HIVE-12471 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 2.0.0, 2.1.0 > > Attachments: HIVE-12471.1.patch, HIVE-12471.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12585) fix TxnHandler connection leak
[ https://issues.apache.org/jira/browse/HIVE-12585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038685#comment-15038685 ] Sergey Shelukhin commented on HIVE-12585: - +1 pending tests > fix TxnHandler connection leak > -- > > Key: HIVE-12585 > URL: https://issues.apache.org/jira/browse/HIVE-12585 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-12585.patch > > > checkLock(CheckLockRequest rqst) is leaking connection -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12257) Enhance ORC FileDump utility to handle flush_length files and recovery
[ https://issues.apache.org/jira/browse/HIVE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038680#comment-15038680 ] Prasanth Jayachandran commented on HIVE-12257: -- The orc_merge test failures are related. It just misses a line in golden file. I will update the golden files on commit. Other test failures are unrelated. > Enhance ORC FileDump utility to handle flush_length files and recovery > -- > > Key: HIVE-12257 > URL: https://issues.apache.org/jira/browse/HIVE-12257 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12257-branch-1.patch, HIVE-12257.1.patch, > HIVE-12257.10.patch, HIVE-12257.11.patch, HIVE-12257.2.patch, > HIVE-12257.3.patch, HIVE-12257.4.patch, HIVE-12257.6.patch, > HIVE-12257.7.patch, HIVE-12257.8.patch, HIVE-12257.9.patch > > > ORC file dump utility currently does not handle delta directories that > contain *_flush_length files. These files contains offsets to footer in the > corresponding delta file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12585) fix TxnHandler connection leak
[ https://issues.apache.org/jira/browse/HIVE-12585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-12585: -- Attachment: HIVE-12585.patch > fix TxnHandler connection leak > -- > > Key: HIVE-12585 > URL: https://issues.apache.org/jira/browse/HIVE-12585 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-12585.patch > > > checkLock(CheckLockRequest rqst) is leaking connection -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038662#comment-15038662 ] Sergey Shelukhin commented on HIVE-11527: - Left some comments on RB. Later, it would be good if [~vgumashta] can also take a look > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12257) Enhance ORC FileDump utility to handle flush_length files and recovery
[
https://issues.apache.org/jira/browse/HIVE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038654#comment-15038654
]
Hive QA commented on HIVE-12257:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775394/HIVE-12257.11.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 9886 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_udf_max
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge11
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge10
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge11
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.jdbc.miniHS2.TestHs2Metrics.testMetrics
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6221/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6221/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6221/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775394 - PreCommit-HIVE-TRUNK-Build
> Enhance ORC FileDump utility to handle flush_length files and recovery
> --
>
> Key: HIVE-12257
> URL: https://issues.apache.org/jira/browse/HIVE-12257
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12257-branch-1.patch, HIVE-12257.1.patch,
> HIVE-12257.10.patch, HIVE-12257.11.patch, HIVE-12257.2.patch,
> HIVE-12257.3.patch, HIVE-12257.4.patch, HIVE-12257.6.patch,
> HIVE-12257.7.patch, HIVE-12257.8.patch, HIVE-12257.9.patch
>
>
> ORC file dump utility currently does not handle delta directories that
> contain *_flush_length files. These files contains offsets to footer in the
> corresponding delta file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11107) Support for Performance regression test suite with TPCDS
[
https://issues.apache.org/jira/browse/HIVE-11107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038659#comment-15038659
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-11107:
--
[~ashutoshc] Thanks for the review comments.
1. Modified the tests to run on MiniTezCluster
2. Moved to QTestUtil
3. This is something I will look at once I add support to use Hbase metastore
to run these queries. As discuss, I will need to modify
setupMetaStoreTableColumnStatsFor30TBTPCDSWorkload() to a more common function
which can be used with different metastore db flavors.
Thanks
Hari
> Support for Performance regression test suite with TPCDS
>
>
> Key: HIVE-11107
> URL: https://issues.apache.org/jira/browse/HIVE-11107
> Project: Hive
> Issue Type: Task
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-11107.1.patch, HIVE-11107.2.patch,
> HIVE-11107.3.patch, HIVE-11107.4.patch, HIVE-11107.5.patch
>
>
> Support to add TPCDS queries to the performance regression test suite with
> Hive CBO turned on.
> This benchmark is intended to make sure that subsequent changes to the
> optimizer or any hive code do not yield any unexpected plan changes. i.e.
> the intention is to not run the entire TPCDS query set, but just "explain
> plan" for the TPCDS queries.
> As part of this jira, we will manually verify that expected hive
> optimizations kick in for the queries (for given stats/dataset). If there is
> a difference in plan within this test suite due to a future commit, it needs
> to be analyzed and we need to make sure that it is not a regression.
> The test suite can be run in master branch from itests by
> {code}
> mvn test -Dtest=TestPerfCliDriver -Phadoop-2
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12435) SELECT COUNT(CASE WHEN...) GROUPBY returns 1 for 'NULL' in a case of ORC and vectorization is enabled.
[
https://issues.apache.org/jira/browse/HIVE-12435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-12435:
Attachment: vector_select_null2.q
> SELECT COUNT(CASE WHEN...) GROUPBY returns 1 for 'NULL' in a case of ORC and
> vectorization is enabled.
> --
>
> Key: HIVE-12435
> URL: https://issues.apache.org/jira/browse/HIVE-12435
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.0.0
>Reporter: Takahiko Saito
>Assignee: Gopal V
>Priority: Critical
> Attachments: vector_select_null2.q
>
>
> Run the following query:
> {noformat}
> create table count_case_groupby (key string, bool boolean) STORED AS orc;
> insert into table count_case_groupby values ('key1', true),('key2',
> false),('key3', NULL),('key4', false),('key5',NULL);
> {noformat}
> The table contains the following:
> {noformat}
> key1 true
> key2 false
> key3 NULL
> key4 false
> key5 NULL
> {noformat}
> The below query returns:
> {noformat}
> SELECT key, COUNT(CASE WHEN bool THEN 1 WHEN NOT bool THEN 0 ELSE NULL END)
> AS cnt_bool0_ok FROM count_case_groupby GROUP BY key;
> key1 1
> key2 1
> key3 1
> key4 1
> key5 1
> {noformat}
> while it expects the following results:
> {noformat}
> key1 1
> key2 1
> key3 0
> key4 1
> key5 0
> {noformat}
> The query works with hive ver 1.2. Also it works when a table is not orc
> format.
> Also even if it's an orc table, when vectorization is disabled, the query
> works.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12435) SELECT COUNT(CASE WHEN...) GROUPBY returns 1 for 'NULL' in a case of ORC and vectorization is enabled.
[
https://issues.apache.org/jira/browse/HIVE-12435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038646#comment-15038646
]
Matt McCline commented on HIVE-12435:
-
I've done some looking at this. I have a Q file that repros created from
description. I don't understand how the CASE WHEN is being executed via
VectorUDFAdaptor yet.
> SELECT COUNT(CASE WHEN...) GROUPBY returns 1 for 'NULL' in a case of ORC and
> vectorization is enabled.
> --
>
> Key: HIVE-12435
> URL: https://issues.apache.org/jira/browse/HIVE-12435
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.0.0
>Reporter: Takahiko Saito
>Assignee: Gopal V
>Priority: Critical
>
> Run the following query:
> {noformat}
> create table count_case_groupby (key string, bool boolean) STORED AS orc;
> insert into table count_case_groupby values ('key1', true),('key2',
> false),('key3', NULL),('key4', false),('key5',NULL);
> {noformat}
> The table contains the following:
> {noformat}
> key1 true
> key2 false
> key3 NULL
> key4 false
> key5 NULL
> {noformat}
> The below query returns:
> {noformat}
> SELECT key, COUNT(CASE WHEN bool THEN 1 WHEN NOT bool THEN 0 ELSE NULL END)
> AS cnt_bool0_ok FROM count_case_groupby GROUP BY key;
> key1 1
> key2 1
> key3 1
> key4 1
> key5 1
> {noformat}
> while it expects the following results:
> {noformat}
> key1 1
> key2 1
> key3 0
> key4 1
> key5 0
> {noformat}
> The query works with hive ver 1.2. Also it works when a table is not orc
> format.
> Also even if it's an orc table, when vectorization is disabled, the query
> works.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11372) join with between predicate comparing integer types returns no rows when ORC format used
[ https://issues.apache.org/jira/browse/HIVE-11372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038618#comment-15038618 ] Prasanth Jayachandran commented on HIVE-11372: -- LGTM, +1 > join with between predicate comparing integer types returns no rows when ORC > format used > > > Key: HIVE-11372 > URL: https://issues.apache.org/jira/browse/HIVE-11372 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: N Campbell >Assignee: Matt McCline > Attachments: HIVE-11372.01.patch, TINT, TSINT > > > getDatabaseProductNameApache Hive > getDatabaseProductVersion 1.2.1.2.3.0.0-2557 > getDriverName Hive JDBC > getDriverVersion 1.2.1.2.3.0.0-2557 > getDriverMajorVersion 1 > getDriverMinorVersion 2 > select tint.rnum, tsint.rnum from tint , tsint where tint.cint between > tsint.csint and tsint.csint > when ORC used no rows returned versus TEXT > create table if not exists TSINT ( RNUM int , CSINT smallint ) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; > create table if not exists TINT ( RNUM int , CINT int ) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12585) fix TxnHandler connection leak
[ https://issues.apache.org/jira/browse/HIVE-12585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-12585: -- Priority: Critical (was: Major) > fix TxnHandler connection leak > -- > > Key: HIVE-12585 > URL: https://issues.apache.org/jira/browse/HIVE-12585 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > > checkLock(CheckLockRequest rqst) is leaking connection -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12584) Vectorized join with partition column of type char does not trim spaces
[
https://issues.apache.org/jira/browse/HIVE-12584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038614#comment-15038614
]
Matt McCline commented on HIVE-12584:
-
+1 LGTM
> Vectorized join with partition column of type char does not trim spaces
>
>
> Key: HIVE-12584
> URL: https://issues.apache.org/jira/browse/HIVE-12584
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 1.3.0, 2.0.0, 2.1.0
>Reporter: Jagruti Varia
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-12584.1.patch
>
>
> When a table is partitioned on a column of type char and if join is performed
> on partitioned column then following exception gets thrown from hashtable
> loader
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: Unexpected tag: 52 reserialized
> to 5
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:82)
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:92)
> ... 4 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: Unexpected tag: 52 reserialized
> to 5
> at
> org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:216)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:293)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:174)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:170)
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:75)
> ... 5 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException: Unexpected tag: 52
> reserialized to 5
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.sanityCheckKeyForTag(MapJoinBytesTableContainer.java:276)
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.getHashFromKey(MapJoinBytesTableContainer.java:247)
> at
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.internalPutRow(HybridHashTableContainer.java:451)
> at
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.putRow(HybridHashTableContainer.java:444)
> at
> org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:210)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12565) VectorUDAFCount.aggregateInputSelection does not handle isRepeated case
[ https://issues.apache.org/jira/browse/HIVE-12565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038610#comment-15038610 ] Prasanth Jayachandran commented on HIVE-12565: -- +1, pending tests > VectorUDAFCount.aggregateInputSelection does not handle isRepeated case > --- > > Key: HIVE-12565 > URL: https://issues.apache.org/jira/browse/HIVE-12565 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline > Attachments: HIVE-12565.01.patch, HIVE-12565.02.patch > > > Not handling the isRepeated case causes wrong results when column is null. > (Native Vector MapJoin OUTER JOIN operators will generate isRepeated null > columns). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038609#comment-15038609 ] Xuefu Zhang commented on HIVE-12538: My understanding is that a hive session may "own" more than one spark session in case of asynchronous queries. If a spark session is live (used to run a spark job), that spark session will not be used to run the next job. Therefore, whenever whenever a spark configuration change is detected in Hive session, we need to mark all the live Spark sessions as outdated. When we are getting a session from the pool and check if the flag is set, then we destroy it and get a new one. Hope this will simply things a bit. > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-12538.1.patch, HIVE-12538.2.patch, > HIVE-12538.3.patch, HIVE-12538.4.patch, HIVE-12538.patch > > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12584) Vectorized join with partition column of type char does not trim spaces
[
https://issues.apache.org/jira/browse/HIVE-12584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-12584:
-
Reporter: Jagruti Varia (was: Prasanth Jayachandran)
> Vectorized join with partition column of type char does not trim spaces
>
>
> Key: HIVE-12584
> URL: https://issues.apache.org/jira/browse/HIVE-12584
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 1.3.0, 2.0.0, 2.1.0
>Reporter: Jagruti Varia
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-12584.1.patch
>
>
> When a table is partitioned on a column of type char and if join is performed
> on partitioned column then following exception gets thrown from hashtable
> loader
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: Unexpected tag: 52 reserialized
> to 5
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:82)
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:92)
> ... 4 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: Unexpected tag: 52 reserialized
> to 5
> at
> org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:216)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:293)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:174)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:170)
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:75)
> ... 5 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException: Unexpected tag: 52
> reserialized to 5
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.sanityCheckKeyForTag(MapJoinBytesTableContainer.java:276)
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.getHashFromKey(MapJoinBytesTableContainer.java:247)
> at
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.internalPutRow(HybridHashTableContainer.java:451)
> at
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.putRow(HybridHashTableContainer.java:444)
> at
> org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:210)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12584) Vectorized join with partition column of type char does not trim spaces
[
https://issues.apache.org/jira/browse/HIVE-12584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-12584:
-
Attachment: HIVE-12584.1.patch
[~mmccline] Can you please take a look at this patch? The code change is
trimming partition value if column type is char. Other changes are all tests.
> Vectorized join with partition column of type char does not trim spaces
>
>
> Key: HIVE-12584
> URL: https://issues.apache.org/jira/browse/HIVE-12584
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 1.3.0, 2.0.0, 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-12584.1.patch
>
>
> When a table is partitioned on a column of type char and if join is performed
> on partitioned column then following exception gets thrown from hashtable
> loader
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: Unexpected tag: 52 reserialized
> to 5
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:82)
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:92)
> ... 4 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: Unexpected tag: 52 reserialized
> to 5
> at
> org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:216)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:293)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:174)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:170)
> at
> org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:75)
> ... 5 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException: Unexpected tag: 52
> reserialized to 5
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.sanityCheckKeyForTag(MapJoinBytesTableContainer.java:276)
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.getHashFromKey(MapJoinBytesTableContainer.java:247)
> at
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.internalPutRow(HybridHashTableContainer.java:451)
> at
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.putRow(HybridHashTableContainer.java:444)
> at
> org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:210)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-12581) Concurrent calls to BoneCP.getConnection() lock up
[
https://issues.apache.org/jira/browse/HIVE-12581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman resolved HIVE-12581.
---
Resolution: Not A Problem
case of premature blaming of bonecp
> Concurrent calls to BoneCP.getConnection() lock up
> --
>
> Key: HIVE-12581
> URL: https://issues.apache.org/jira/browse/HIVE-12581
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> {noformat}
> Thread 31078: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
> - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
> @bci=20, line=226 (Interpreted frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
> @bci=68, line=2082 (Interpreted frame)
> - java.util.concurrent.LinkedBlockingQueue.poll(long,
> java.util.concurrent.TimeUnit) @bci=62, line=467 (Interpreted frame)
> - com.jolbox.bonecp.DefaultConnectionStrategy.getConnectionInternal()
> @bci=50, line=82 (Interpreted frame)
> - com.jolbox.bonecp.AbstractConnectionStrategy.getConnection() @bci=6,
> line=90 (Interpreted frame)
> - com.jolbox.bonecp.BoneCP.getConnection() @bci=4, line=553 (Interpreted
> frame)
> - com.jolbox.bonecp.BoneCPDataSource.getConnection() @bci=108, line=131
> (Interpreted frame)
> - org.apache.hadoop.hive.metastore.txn.TxnHandler.getDbConn(int) @bci=16,
> line=973 (Interpreted frame)
> -
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)
> @bci=88, line=530 (Interpreted frame)
> {noformat}
> Sometimes these lock up, i.e. some sort of deadlock happens inside the
> connection pool
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12525) Cleanup unused metrics in HMS
[ https://issues.apache.org/jira/browse/HIVE-12525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-12525: - Fix Version/s: 2.0.0 OK missed that, backported to 2.0 as well. > Cleanup unused metrics in HMS > - > > Key: HIVE-12525 > URL: https://issues.apache.org/jira/browse/HIVE-12525 > Project: Hive > Issue Type: Sub-task >Reporter: Szehon Ho >Assignee: Szehon Ho > Fix For: 2.0.0, 2.1.0 > > Attachments: HIVE-12525.patch > > > I had added these without much thought when writing the metrics-framework to > test out the concept. > Looking back, these actually need of more investigation, as some are actually > wrong or at least do not add much value. Wrong is the active-transaction, as > actually each ObjectStore is a thread-local, and an aggregate number is what > was meant. Open/committed/rollback need some investigation what really helps. > Goal is to remove these before the release to reduce confusion to users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12266) When client exists abnormally, it doesn't release ACID locks
[ https://issues.apache.org/jira/browse/HIVE-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-12266: - Fix Version/s: 2.0.0 > When client exists abnormally, it doesn't release ACID locks > > > Key: HIVE-12266 > URL: https://issues.apache.org/jira/browse/HIVE-12266 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-12266.1.patch, HIVE-12266.2.patch, > HIVE-12266.3.patch, HIVE-12266.branch-1.patch > > > if you start Hive CLI (locking enabled) and run some command that acquires > locks and ^C the shell before command completes the locks for the command > remain until they timeout. > I believe Beeline has the same issue. > Need to add proper hooks to release locks when command dies. (As much as > possible) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12266) When client exists abnormally, it doesn't release ACID locks
[ https://issues.apache.org/jira/browse/HIVE-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038558#comment-15038558 ] Thejas M Nair commented on HIVE-12266: -- Adding 2.0.0 as fix version as it was committed to master as well. > When client exists abnormally, it doesn't release ACID locks > > > Key: HIVE-12266 > URL: https://issues.apache.org/jira/browse/HIVE-12266 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-12266.1.patch, HIVE-12266.2.patch, > HIVE-12266.3.patch, HIVE-12266.branch-1.patch > > > if you start Hive CLI (locking enabled) and run some command that acquires > locks and ^C the shell before command completes the locks for the command > remain until they timeout. > I believe Beeline has the same issue. > Need to add proper hooks to release locks when command dies. (As much as > possible) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12583) HS2 ShutdownHookManager holds extra of Driver instance
[ https://issues.apache.org/jira/browse/HIVE-12583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038545#comment-15038545 ] Thejas M Nair commented on HIVE-12583: -- FYI [~wzheng] > HS2 ShutdownHookManager holds extra of Driver instance > --- > > Key: HIVE-12583 > URL: https://issues.apache.org/jira/browse/HIVE-12583 > Project: Hive > Issue Type: Bug > Components: Locking >Affects Versions: 1.3.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-12583.1.patch, HIVE-12583.2.patch > > > HIVE-12266 add a shutdown hook for every Driver instance to release the lock > th session holds in case Driver does not exist elegantly. However, that holds > all Driver instances and HS2 may run out of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11372) join with between predicate comparing integer types returns no rows when ORC format used
[ https://issues.apache.org/jira/browse/HIVE-11372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-11372: Attachment: HIVE-11372.01.patch > join with between predicate comparing integer types returns no rows when ORC > format used > > > Key: HIVE-11372 > URL: https://issues.apache.org/jira/browse/HIVE-11372 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: N Campbell >Assignee: Matt McCline > Attachments: HIVE-11372.01.patch, TINT, TSINT > > > getDatabaseProductNameApache Hive > getDatabaseProductVersion 1.2.1.2.3.0.0-2557 > getDriverName Hive JDBC > getDriverVersion 1.2.1.2.3.0.0-2557 > getDriverMajorVersion 1 > getDriverMinorVersion 2 > select tint.rnum, tsint.rnum from tint , tsint where tint.cint between > tsint.csint and tsint.csint > when ORC used no rows returned versus TEXT > create table if not exists TSINT ( RNUM int , CSINT smallint ) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; > create table if not exists TINT ( RNUM int , CINT int ) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11372) join with between predicate comparing integer types returns no rows when ORC format used
[ https://issues.apache.org/jira/browse/HIVE-11372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-11372: Attachment: (was: vector_select_int.q) > join with between predicate comparing integer types returns no rows when ORC > format used > > > Key: HIVE-11372 > URL: https://issues.apache.org/jira/browse/HIVE-11372 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: N Campbell >Assignee: Matt McCline > Attachments: TINT, TSINT > > > getDatabaseProductNameApache Hive > getDatabaseProductVersion 1.2.1.2.3.0.0-2557 > getDriverName Hive JDBC > getDriverVersion 1.2.1.2.3.0.0-2557 > getDriverMajorVersion 1 > getDriverMinorVersion 2 > select tint.rnum, tsint.rnum from tint , tsint where tint.cint between > tsint.csint and tsint.csint > when ORC used no rows returned versus TEXT > create table if not exists TSINT ( RNUM int , CSINT smallint ) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; > create table if not exists TINT ( RNUM int , CINT int ) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11372) join with between predicate comparing integer types returns no rows when ORC format used
[ https://issues.apache.org/jira/browse/HIVE-11372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-11372: Attachment: (was: vector_select_int.q.out) > join with between predicate comparing integer types returns no rows when ORC > format used > > > Key: HIVE-11372 > URL: https://issues.apache.org/jira/browse/HIVE-11372 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: N Campbell >Assignee: Matt McCline > Attachments: TINT, TSINT > > > getDatabaseProductNameApache Hive > getDatabaseProductVersion 1.2.1.2.3.0.0-2557 > getDriverName Hive JDBC > getDriverVersion 1.2.1.2.3.0.0-2557 > getDriverMajorVersion 1 > getDriverMinorVersion 2 > select tint.rnum, tsint.rnum from tint , tsint where tint.cint between > tsint.csint and tsint.csint > when ORC used no rows returned versus TEXT > create table if not exists TSINT ( RNUM int , CSINT smallint ) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; > create table if not exists TINT ( RNUM int , CINT int ) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12055) Create row-by-row shims for the write path
[
https://issues.apache.org/jira/browse/HIVE-12055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038519#comment-15038519
]
Hive QA commented on HIVE-12055:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775393/HIVE-12055.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6220/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6220/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6220/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6220/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
4c8e47e..afd7b93 branch-2.0 -> origin/branch-2.0
a603ed8..bdec08f master -> origin/master
+ git reset --hard HEAD
HEAD is now at a603ed8 HIVE-12537: RLEv2 doesn't seem to work (Prasanth
Jayachandran reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 3 commits, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at bdec08f HIVE-12525 : Cleanup unused metrics in HMS (Szehon,
reviewed by Jimmy Xiang)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775393 - PreCommit-HIVE-TRUNK-Build
> Create row-by-row shims for the write path
> ---
>
> Key: HIVE-12055
> URL: https://issues.apache.org/jira/browse/HIVE-12055
> Project: Hive
> Issue Type: Sub-task
> Components: ORC, Shims
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-12055.patch, HIVE-12055.patch, HIVE-12055.patch,
> HIVE-12055.patch
>
>
> As part of removing the row-by-row writer, we'll need to shim out the higher
> level API (OrcSerde and OrcOutputFormat) so that we maintain backwards
> compatibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12581) Concurrent calls to BoneCP.getConnection() lock up
[
https://issues.apache.org/jira/browse/HIVE-12581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038517#comment-15038517
]
Eugene Koifman commented on HIVE-12581:
---
This bug maybe a red herring.
but if it isn't there is a way to tell bonecp to timeout getConnection() after
specified time.
I agree that it is time to upgrade - last bonecp release was in 2013.
> Concurrent calls to BoneCP.getConnection() lock up
> --
>
> Key: HIVE-12581
> URL: https://issues.apache.org/jira/browse/HIVE-12581
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> {noformat}
> Thread 31078: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
> - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
> @bci=20, line=226 (Interpreted frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
> @bci=68, line=2082 (Interpreted frame)
> - java.util.concurrent.LinkedBlockingQueue.poll(long,
> java.util.concurrent.TimeUnit) @bci=62, line=467 (Interpreted frame)
> - com.jolbox.bonecp.DefaultConnectionStrategy.getConnectionInternal()
> @bci=50, line=82 (Interpreted frame)
> - com.jolbox.bonecp.AbstractConnectionStrategy.getConnection() @bci=6,
> line=90 (Interpreted frame)
> - com.jolbox.bonecp.BoneCP.getConnection() @bci=4, line=553 (Interpreted
> frame)
> - com.jolbox.bonecp.BoneCPDataSource.getConnection() @bci=108, line=131
> (Interpreted frame)
> - org.apache.hadoop.hive.metastore.txn.TxnHandler.getDbConn(int) @bci=16,
> line=973 (Interpreted frame)
> -
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)
> @bci=88, line=530 (Interpreted frame)
> {noformat}
> Sometimes these lock up, i.e. some sort of deadlock happens inside the
> connection pool
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12525) Cleanup unused metrics in HMS
[ https://issues.apache.org/jira/browse/HIVE-12525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12525: Fix Version/s: (was: 2.0.0) 2.1.0 > Cleanup unused metrics in HMS > - > > Key: HIVE-12525 > URL: https://issues.apache.org/jira/browse/HIVE-12525 > Project: Hive > Issue Type: Sub-task >Reporter: Szehon Ho >Assignee: Szehon Ho > Fix For: 2.1.0 > > Attachments: HIVE-12525.patch > > > I had added these without much thought when writing the metrics-framework to > test out the concept. > Looking back, these actually need of more investigation, as some are actually > wrong or at least do not add much value. Wrong is the active-transaction, as > actually each ObjectStore is a thread-local, and an aggregate number is what > was meant. Open/committed/rollback need some investigation what really helps. > Goal is to remove these before the release to reduce confusion to users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12525) Cleanup unused metrics in HMS
[ https://issues.apache.org/jira/browse/HIVE-12525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038514#comment-15038514 ] Sergey Shelukhin commented on HIVE-12525: - Master is now 2.1... please backport if needed :) > Cleanup unused metrics in HMS > - > > Key: HIVE-12525 > URL: https://issues.apache.org/jira/browse/HIVE-12525 > Project: Hive > Issue Type: Sub-task >Reporter: Szehon Ho >Assignee: Szehon Ho > Fix For: 2.1.0 > > Attachments: HIVE-12525.patch > > > I had added these without much thought when writing the metrics-framework to > test out the concept. > Looking back, these actually need of more investigation, as some are actually > wrong or at least do not add much value. Wrong is the active-transaction, as > actually each ObjectStore is a thread-local, and an aggregate number is what > was meant. Open/committed/rollback need some investigation what really helps. > Goal is to remove these before the release to reduce confusion to users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12471) Secure HS2 web UI with SSL
[
https://issues.apache.org/jira/browse/HIVE-12471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038511#comment-15038511
]
Hive QA commented on HIVE-12471:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775377/HIVE-12471.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 9885 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_udf_max
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6219/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6219/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6219/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775377 - PreCommit-HIVE-TRUNK-Build
> Secure HS2 web UI with SSL
> --
>
> Key: HIVE-12471
> URL: https://issues.apache.org/jira/browse/HIVE-12471
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Attachments: HIVE-12471.1.patch, HIVE-12471.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11530) push limit thru outer join
[ https://issues.apache.org/jira/browse/HIVE-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038501#comment-15038501 ] Sergey Shelukhin commented on HIVE-11530: - [~jcamachorodriguez] is this still needed for non-CBO path? > push limit thru outer join > -- > > Key: HIVE-11530 > URL: https://issues.apache.org/jira/browse/HIVE-11530 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Reporter: Sergey Shelukhin >Assignee: Yohei Abe > > When the query has a left or right outer join with limit, we can push the > limit into the left/right side of the join. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12471) Secure HS2 web UI with SSL
[ https://issues.apache.org/jira/browse/HIVE-12471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038499#comment-15038499 ] Szehon Ho commented on HIVE-12471: -- Looks good to me, +1, thanks Mohit for additional review. > Secure HS2 web UI with SSL > -- > > Key: HIVE-12471 > URL: https://issues.apache.org/jira/browse/HIVE-12471 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: HIVE-12471.1.patch, HIVE-12471.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12581) Concurrent calls to BoneCP.getConnection() lock up
[
https://issues.apache.org/jira/browse/HIVE-12581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038491#comment-15038491
]
Sergey Shelukhin commented on HIVE-12581:
-
We are upgrading to DN 4 somewhere, so maybe we can also switch to a new fancy
connection pool after that. Syncing around the connection pool is not really an
option, so I don't know what a workaround would be... connection pool
threadpool where we kill stuck threads?
> Concurrent calls to BoneCP.getConnection() lock up
> --
>
> Key: HIVE-12581
> URL: https://issues.apache.org/jira/browse/HIVE-12581
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> {noformat}
> Thread 31078: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
> - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
> @bci=20, line=226 (Interpreted frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
> @bci=68, line=2082 (Interpreted frame)
> - java.util.concurrent.LinkedBlockingQueue.poll(long,
> java.util.concurrent.TimeUnit) @bci=62, line=467 (Interpreted frame)
> - com.jolbox.bonecp.DefaultConnectionStrategy.getConnectionInternal()
> @bci=50, line=82 (Interpreted frame)
> - com.jolbox.bonecp.AbstractConnectionStrategy.getConnection() @bci=6,
> line=90 (Interpreted frame)
> - com.jolbox.bonecp.BoneCP.getConnection() @bci=4, line=553 (Interpreted
> frame)
> - com.jolbox.bonecp.BoneCPDataSource.getConnection() @bci=108, line=131
> (Interpreted frame)
> - org.apache.hadoop.hive.metastore.txn.TxnHandler.getDbConn(int) @bci=16,
> line=973 (Interpreted frame)
> -
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)
> @bci=88, line=530 (Interpreted frame)
> {noformat}
> Sometimes these lock up, i.e. some sort of deadlock happens inside the
> connection pool
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12583) HS2 ShutdownHookManager holds extra of Driver instance
[ https://issues.apache.org/jira/browse/HIVE-12583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038483#comment-15038483 ] Sergey Shelukhin commented on HIVE-12583: - +1 pending tests > HS2 ShutdownHookManager holds extra of Driver instance > --- > > Key: HIVE-12583 > URL: https://issues.apache.org/jira/browse/HIVE-12583 > Project: Hive > Issue Type: Bug > Components: Locking >Affects Versions: 1.3.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-12583.1.patch, HIVE-12583.2.patch > > > HIVE-12266 add a shutdown hook for every Driver instance to release the lock > th session holds in case Driver does not exist elegantly. However, that holds > all Driver instances and HS2 may run out of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12505) Insert overwrite in same encrypted zone silently fails to remove some existing files
[ https://issues.apache.org/jira/browse/HIVE-12505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038480#comment-15038480 ] Chaoyu Tang commented on HIVE-12505: Failed tests are not related to this patch. > Insert overwrite in same encrypted zone silently fails to remove some > existing files > > > Key: HIVE-12505 > URL: https://issues.apache.org/jira/browse/HIVE-12505 > Project: Hive > Issue Type: Bug > Components: Encryption >Affects Versions: 1.2.1 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12505.1.patch, HIVE-12505.2.patch, > HIVE-12505.3.patch, HIVE-12505.patch > > > With HDFS Trash enabled but its encryption zone lower than Hive data > directory, insert overwrite command silently fails to trash the existing > files during overwrite, which could lead to unexpected incorrect results > (more rows returned than expected) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12583) HS2 ShutdownHookManager holds extra of Driver instance
[ https://issues.apache.org/jira/browse/HIVE-12583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-12583: -- Attachment: HIVE-12583.2.patch 1) Yes, it should not be inside the try-catch block. Attach a new patch. 2) The lock list is dynamically maintained inside Driver, it cannot be exacted out at the moment of adding to shutdown hook. > HS2 ShutdownHookManager holds extra of Driver instance > --- > > Key: HIVE-12583 > URL: https://issues.apache.org/jira/browse/HIVE-12583 > Project: Hive > Issue Type: Bug > Components: Locking >Affects Versions: 1.3.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-12583.1.patch, HIVE-12583.2.patch > > > HIVE-12266 add a shutdown hook for every Driver instance to release the lock > th session holds in case Driver does not exist elegantly. However, that holds > all Driver instances and HS2 may run out of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12528) don't start HS2 Tez sessions in a single thread
[ https://issues.apache.org/jira/browse/HIVE-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038468#comment-15038468 ] Sergey Shelukhin commented on HIVE-12528: - Not sure if MiniTez timeout is related, it could be. I will eventually test this on a cluster, and probably resubmit for HiveQA after review feedback. > don't start HS2 Tez sessions in a single thread > --- > > Key: HIVE-12528 > URL: https://issues.apache.org/jira/browse/HIVE-12528 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12528.patch > > > Starting sessions in parallel would improve the startup time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12532) LLAP Cache: Uncompressed data cache has NPE
[
https://issues.apache.org/jira/browse/HIVE-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038438#comment-15038438
]
Sergey Shelukhin commented on HIVE-12532:
-
Test failures are unrelated... we are getting a lot of random failures with
metastore connection timeouts lately.
> LLAP Cache: Uncompressed data cache has NPE
> ---
>
> Key: HIVE-12532
> URL: https://issues.apache.org/jira/browse/HIVE-12532
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.0.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-12435.patch, HIVE-12532.01.patch
>
>
> {code}
> 2015-11-26 08:28:45,232
> [TezTaskRunner_attempt_1448429572030_0255_2_02_19_2(attempt_1448429572030_0255_2_02_19_2)]
> WARN org.apache.tez.runtime.LogicalIOProcessorRuntimeTask: Ignoring
> exception when closing input a(cleanup). Exception class=java.io.IOException,
> message=java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:283)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.close(LlapInputFormat.java:275)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doClose(HiveRecordReader.java:50)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.close(HiveContextAwareRecordReader.java:104)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.close(TezGroupedSplitsInputFormat.java:177)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.close(MRReaderMapred.java:96)
> at org.apache.tez.mapreduce.input.MRInput.close(MRInput.java:559)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.cleanup(LogicalIOProcessorRuntimeTask.java:872)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:104)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.preReadUncompressedStream(EncodedReaderImpl.java:795)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:320)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:413)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:194)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:191)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:191)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:74)
> ... 5 more
> {code}
> Not clear if current.next can set it to null before the continue;
> {code}
> assert partOffset <= current.getOffset();
> if (partOffset == current.getOffset() && current instanceof CacheChunk)
> {
> // We assume cache chunks would always match the way we read, so
> check and skip it.
> assert current.getOffset() == partOffset && current.getEnd() ==
> partEnd;
> lastUncompressed = (CacheChunk)current;
> current = current.next;
> continue;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12431) Support timeout for global compile lock
[ https://issues.apache.org/jira/browse/HIVE-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038436#comment-15038436 ] Sergey Shelukhin commented on HIVE-12431: - Also the testing approach is not pretty ;) I wonder if some pre-compile hook could be added instead that would do the job without public test globals? > Support timeout for global compile lock > --- > > Key: HIVE-12431 > URL: https://issues.apache.org/jira/browse/HIVE-12431 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Query Processor >Affects Versions: 1.2.1 >Reporter: Lenni Kuff >Assignee: Mohit Sabharwal > Attachments: HIVE-12431.patch > > > To help with HiveServer2 scalability, it would be useful to allow users to > configure a timeout value for queries waiting to be compiled. If the timeout > value is reached then the query would abort. One option to achieve this would > be to update the compile lock to use a try-lock with the timeout value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12431) Support timeout for global compile lock
[
https://issues.apache.org/jira/browse/HIVE-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038432#comment-15038432
]
Sergey Shelukhin commented on HIVE-12431:
-
{noformat}
+Thread.currentThread().interrupt();
+if (LOG.isDebugEnabled()) {
+ LOG.debug("Interrupted Exception ignored", e);
+}
+ }
{noformat}
This path will return the lock without having acquired it
> Support timeout for global compile lock
> ---
>
> Key: HIVE-12431
> URL: https://issues.apache.org/jira/browse/HIVE-12431
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Query Processor
>Affects Versions: 1.2.1
>Reporter: Lenni Kuff
>Assignee: Mohit Sabharwal
> Attachments: HIVE-12431.patch
>
>
> To help with HiveServer2 scalability, it would be useful to allow users to
> configure a timeout value for queries waiting to be compiled. If the timeout
> value is reached then the query would abort. One option to achieve this would
> be to update the compile lock to use a try-lock with the timeout value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12583) HS2 ShutdownHookManager holds extra of Driver instance
[ https://issues.apache.org/jira/browse/HIVE-12583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038421#comment-15038421 ] Sergey Shelukhin commented on HIVE-12583: - 1) Can it be in a separate try-catch so if the method prior fails, it still gets released? 2) Optional: is it easy to extract part of the driver (e.g. just extract the locks if they are all known or if the List is final in the ctx) in advance to pass to the hook, so it doesn't have to reference the entire driver? > HS2 ShutdownHookManager holds extra of Driver instance > --- > > Key: HIVE-12583 > URL: https://issues.apache.org/jira/browse/HIVE-12583 > Project: Hive > Issue Type: Bug > Components: Locking >Affects Versions: 1.3.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-12583.1.patch > > > HIVE-12266 add a shutdown hook for every Driver instance to release the lock > th session holds in case Driver does not exist elegantly. However, that holds > all Driver instances and HS2 may run out of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7989) Optimize Windowing function performance for row frames
[ https://issues.apache.org/jira/browse/HIVE-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038417#comment-15038417 ] Dongwook Kwon commented on HIVE-7989: - Hi, also from my tests as Ankit did, the performance improvement of this patch is significant, definitely this is worth to implement. Could someone take a look at and merge into trunk that next release can pick it up? > Optimize Windowing function performance for row frames > -- > > Key: HIVE-7989 > URL: https://issues.apache.org/jira/browse/HIVE-7989 > Project: Hive > Issue Type: Improvement > Components: PTF-Windowing >Affects Versions: 0.13.0 >Reporter: Ankit Kamboj > Attachments: HIVE-7989.patch > > > To find aggregate value for each row, current windowing function > implementation creates a new aggregation buffer for each row, iterates over > all the rows in respective window frame, puts them in buffer and then finds > the aggregated value. This causes bottleneck for partitions with huge number > of rows because this process runs in n-square complexity (n being rows in a > partition) for each partition. So, if there are multiple partitions in a > dataset, each with millions of rows, aggregation for all rows will take days > to finish. > There is scope of optimization for row frames, for following cases: > a) For UNBOUNDED PRECEDING start and bounded end: Instead of iterating on > window frame again for each row, we can slide the end one row at a time and > aggregate, since we know the start is fixed for each row. This will have > running time linear to the size of partition. > b) For bounded start and UNBOUNDED FOLLOWING end: Instead of iterating on > window frame again for each row, we can slide the start one row at a time and > aggregate in reverse, since we know the end is fixed for each row. This will > have running time linear to the size of partition. > Also, In general for both row and value frames, we don't need to iterate over > the range and re-create aggregation buffer if the start as well as end remain > same. Instead, can re-use the previously created aggregation buffer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12583) HS2 ShutdownHookManager holds extra of Driver instance
[ https://issues.apache.org/jira/browse/HIVE-12583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12583: Target Version/s: 1.3.0, 2.0.0, 2.1.0 (was: 1.3.0) > HS2 ShutdownHookManager holds extra of Driver instance > --- > > Key: HIVE-12583 > URL: https://issues.apache.org/jira/browse/HIVE-12583 > Project: Hive > Issue Type: Bug > Components: Locking >Affects Versions: 1.3.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-12583.1.patch > > > HIVE-12266 add a shutdown hook for every Driver instance to release the lock > th session holds in case Driver does not exist elegantly. However, that holds > all Driver instances and HS2 may run out of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12583) HS2 ShutdownHookManager holds extra of Driver instance
[ https://issues.apache.org/jira/browse/HIVE-12583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-12583: -- Attachment: HIVE-12583.1.patch The patch removes shutdown hook if the Driver exist normally. > HS2 ShutdownHookManager holds extra of Driver instance > --- > > Key: HIVE-12583 > URL: https://issues.apache.org/jira/browse/HIVE-12583 > Project: Hive > Issue Type: Bug > Components: Locking >Affects Versions: 1.3.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-12583.1.patch > > > HIVE-12266 add a shutdown hook for every Driver instance to release the lock > th session holds in case Driver does not exist elegantly. However, that holds > all Driver instances and HS2 may run out of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11312) ORC format: where clause with CHAR data type not returning any rows
[
https://issues.apache.org/jira/browse/HIVE-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-11312:
Fix Version/s: 2.1.0
> ORC format: where clause with CHAR data type not returning any rows
> ---
>
> Key: HIVE-11312
> URL: https://issues.apache.org/jira/browse/HIVE-11312
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.0, 1.3.0, 1.2.1, 2.0.0
>Reporter: Thomas Friedrich
>Assignee: Prasanth Jayachandran
>Priority: Blocker
> Labels: orc
> Fix For: 1.3.0, 2.0.0, 2.1.0
>
> Attachments: HIVE-11312-branch-1.patch, HIVE-11312.1.patch,
> HIVE-11312.2.patch, HIVE-11312.3.patch, HIVE-11312.4.patch, HIVE-11312.5.patch
>
>
> Test case:
> Setup:
> create table orc_test( col1 string, col2 char(10)) stored as orc
> tblproperties ("orc.compress"="NONE");
> insert into orc_test values ('val1', '1');
> Query:
> select * from orc_test where col2='1';
> Query returns no row.
> Problem is introduced with HIVE-10286, class RecordReaderImpl.java, method
> evaluatePredicateRange.
> Old code:
> - Object baseObj = predicate.getLiteral(PredicateLeaf.FileFormat.ORC);
> - Object minValue = getConvertedStatsObj(min, baseObj);
> - Object maxValue = getConvertedStatsObj(max, baseObj);
> - Object predObj = getBaseObjectForComparison(baseObj, minValue);
> New code:
> + Object baseObj = predicate.getLiteral();
> + Object minValue = getBaseObjectForComparison(predicate.getType(), min);
> + Object maxValue = getBaseObjectForComparison(predicate.getType(), max);
> + Object predObj = getBaseObjectForComparison(predicate.getType(), baseObj);
> The values for min and max are of type String which contain as many
> characters as the CHAR column indicated. For example if the type is CHAR(10),
> and the row has value 1, the value of String min is "1 ";
> Before Hive 1.2, the method getConvertedStatsObj would call
> StringUtils.stripEnd(statsObj.toString(), null); which would remove the
> trailing spaces from min and max. Later in the compareToRange method, it was
> able to compare "1" with "1".
> In Hive 1.2 with the use getBaseObjectForComparison method, it simply returns
> obj.String if the data type is String, which means minValue and maxValue are
> still "1 ".
> As a result, the compareToRange method will return a wrong value
> ("1".compareTo("1 ") -9 instead of 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11312) ORC format: where clause with CHAR data type not returning any rows
[
https://issues.apache.org/jira/browse/HIVE-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038384#comment-15038384
]
Sergey Shelukhin commented on HIVE-11312:
-
Isn't it implied that something fixed in 2.0 is also fixed in 2.1?
> ORC format: where clause with CHAR data type not returning any rows
> ---
>
> Key: HIVE-11312
> URL: https://issues.apache.org/jira/browse/HIVE-11312
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.0, 1.3.0, 1.2.1, 2.0.0
>Reporter: Thomas Friedrich
>Assignee: Prasanth Jayachandran
>Priority: Blocker
> Labels: orc
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-11312-branch-1.patch, HIVE-11312.1.patch,
> HIVE-11312.2.patch, HIVE-11312.3.patch, HIVE-11312.4.patch, HIVE-11312.5.patch
>
>
> Test case:
> Setup:
> create table orc_test( col1 string, col2 char(10)) stored as orc
> tblproperties ("orc.compress"="NONE");
> insert into orc_test values ('val1', '1');
> Query:
> select * from orc_test where col2='1';
> Query returns no row.
> Problem is introduced with HIVE-10286, class RecordReaderImpl.java, method
> evaluatePredicateRange.
> Old code:
> - Object baseObj = predicate.getLiteral(PredicateLeaf.FileFormat.ORC);
> - Object minValue = getConvertedStatsObj(min, baseObj);
> - Object maxValue = getConvertedStatsObj(max, baseObj);
> - Object predObj = getBaseObjectForComparison(baseObj, minValue);
> New code:
> + Object baseObj = predicate.getLiteral();
> + Object minValue = getBaseObjectForComparison(predicate.getType(), min);
> + Object maxValue = getBaseObjectForComparison(predicate.getType(), max);
> + Object predObj = getBaseObjectForComparison(predicate.getType(), baseObj);
> The values for min and max are of type String which contain as many
> characters as the CHAR column indicated. For example if the type is CHAR(10),
> and the row has value 1, the value of String min is "1 ";
> Before Hive 1.2, the method getConvertedStatsObj would call
> StringUtils.stripEnd(statsObj.toString(), null); which would remove the
> trailing spaces from min and max. Later in the compareToRange method, it was
> able to compare "1" with "1".
> In Hive 1.2 with the use getBaseObjectForComparison method, it simply returns
> obj.String if the data type is String, which means minValue and maxValue are
> still "1 ".
> As a result, the compareToRange method will return a wrong value
> ("1".compareTo("1 ") -9 instead of 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11312) ORC format: where clause with CHAR data type not returning any rows
[
https://issues.apache.org/jira/browse/HIVE-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-11312:
Fix Version/s: (was: 2.1.0)
> ORC format: where clause with CHAR data type not returning any rows
> ---
>
> Key: HIVE-11312
> URL: https://issues.apache.org/jira/browse/HIVE-11312
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.0, 1.3.0, 1.2.1, 2.0.0
>Reporter: Thomas Friedrich
>Assignee: Prasanth Jayachandran
>Priority: Blocker
> Labels: orc
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-11312-branch-1.patch, HIVE-11312.1.patch,
> HIVE-11312.2.patch, HIVE-11312.3.patch, HIVE-11312.4.patch, HIVE-11312.5.patch
>
>
> Test case:
> Setup:
> create table orc_test( col1 string, col2 char(10)) stored as orc
> tblproperties ("orc.compress"="NONE");
> insert into orc_test values ('val1', '1');
> Query:
> select * from orc_test where col2='1';
> Query returns no row.
> Problem is introduced with HIVE-10286, class RecordReaderImpl.java, method
> evaluatePredicateRange.
> Old code:
> - Object baseObj = predicate.getLiteral(PredicateLeaf.FileFormat.ORC);
> - Object minValue = getConvertedStatsObj(min, baseObj);
> - Object maxValue = getConvertedStatsObj(max, baseObj);
> - Object predObj = getBaseObjectForComparison(baseObj, minValue);
> New code:
> + Object baseObj = predicate.getLiteral();
> + Object minValue = getBaseObjectForComparison(predicate.getType(), min);
> + Object maxValue = getBaseObjectForComparison(predicate.getType(), max);
> + Object predObj = getBaseObjectForComparison(predicate.getType(), baseObj);
> The values for min and max are of type String which contain as many
> characters as the CHAR column indicated. For example if the type is CHAR(10),
> and the row has value 1, the value of String min is "1 ";
> Before Hive 1.2, the method getConvertedStatsObj would call
> StringUtils.stripEnd(statsObj.toString(), null); which would remove the
> trailing spaces from min and max. Later in the compareToRange method, it was
> able to compare "1" with "1".
> In Hive 1.2 with the use getBaseObjectForComparison method, it simply returns
> obj.String if the data type is String, which means minValue and maxValue are
> still "1 ".
> As a result, the compareToRange method will return a wrong value
> ("1".compareTo("1 ") -9 instead of 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12413) Default mode for hive.mapred.mode should be strict
[ https://issues.apache.org/jira/browse/HIVE-12413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038380#comment-15038380 ] Ashutosh Chauhan commented on HIVE-12413: - yup.. will fix that on commit. From a quick look all these tests are failing on other recent runs. But I will anyway go through the full list to ascertain no new failures are because of this patch. > Default mode for hive.mapred.mode should be strict > -- > > Key: HIVE-12413 > URL: https://issues.apache.org/jira/browse/HIVE-12413 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12413.1.patch, HIVE-12413.2.patch, > HIVE-12413.3.patch, HIVE-12413.4.patch, HIVE-12413.patch > > > Non-strict mode allows some questionable semantics and questionable > operations. Its better that user makes a conscious choice to enable such a > behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12413) Default mode for hive.mapred.mode should be strict
[ https://issues.apache.org/jira/browse/HIVE-12413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15038375#comment-15038375 ] Prasanth Jayachandran commented on HIVE-12413: -- very minor: disable_merge_for_bucketing.q has repeated set command which can be fixed on commit. Otherwise, LGTM +1. Are any of these test failures related? > Default mode for hive.mapred.mode should be strict > -- > > Key: HIVE-12413 > URL: https://issues.apache.org/jira/browse/HIVE-12413 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12413.1.patch, HIVE-12413.2.patch, > HIVE-12413.3.patch, HIVE-12413.4.patch, HIVE-12413.patch > > > Non-strict mode allows some questionable semantics and questionable > operations. Its better that user makes a conscious choice to enable such a > behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
