[jira] [Commented] (HIVE-16100) Dynamic Sorted Partition optimizer loses sibling operators
[
https://issues.apache.org/jira/browse/HIVE-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896853#comment-15896853
]
Prasanth Jayachandran commented on HIVE-16100:
--
I am missing something here. Under what scenario does FS's parent will have
another sibling? If it is multi file sink, then this optimization should be
visited for each FSOp and each path should have a SEL.
I am assuming the case you are describing is something like below. I am not
sure if we will end up with
such plan.
{code}
RS -> SEL -> FS
|
-> FS
{code}
I tried running the query without the patch and it generated the same explain
plan.
> Dynamic Sorted Partition optimizer loses sibling operators
> --
>
> Key: HIVE-16100
> URL: https://issues.apache.org/jira/browse/HIVE-16100
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1, 2.2.0, 2.1.1
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-16100.1.patch, HIVE-16100.2.patch,
> HIVE-16100.2.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java#L173
> {code}
> // unlink connection between FS and its parent
> fsParent = fsOp.getParentOperators().get(0);
> fsParent.getChildOperators().clear();
> {code}
> The optimizer discards any cases where the fsParent has another SEL child
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896838#comment-15896838
]
Hive QA commented on HIVE-16116:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856220/HIVE-16116.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10329 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=229)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
(batchId=147)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=224)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in]
(batchId=119)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3956/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3956/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3956/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856220 - PreCommit-HIVE-Build
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16116.1.patch
>
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16116:
Attachment: HIVE-16116.1.patch
{{BeelineOpts.hiveConfVariables}} is set to null when it tries to load data
from beeline.properties with empty value for {{beeline.hiveconfvariables}}.
This causes NPE at the time of invoking beeline with --hiveconf option. Patch
fixes that case.
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16116.1.patch
>
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896792#comment-15896792
]
Rajesh Balamohan edited comment on HIVE-16116 at 3/6/17 6:26 AM:
-
{{BeelineOpts.hiveConfVariables}} is set to null when it tries to load data
from beeline.properties with empty value for {{beeline.hiveconfvariables}}.
This causes NPE at the time of invoking beeline with --hiveconf option.
Attaching the patch which fixes this case.
was (Author: rajesh.balamohan):
{{BeelineOpts.hiveConfVariables}} is set to null when it tries to load data
from beeline.properties with empty value for {{beeline.hiveconfvariables}}.
This causes NPE at the time of invoking beeline with --hiveconf option. Patch
fixes that case.
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16116.1.patch
>
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16116:
Status: Patch Available (was: Open)
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16116.1.patch
>
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16101) QTest failure BeeLine escape_comments after HIVE-16045
[
https://issues.apache.org/jira/browse/HIVE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896771#comment-15896771
]
Zoltan Haindrich commented on HIVE-16101:
-
looks like the ptest executor have run out of space :)
note: this {{beeline/src/java/org/apache/hive/beeline/util/QFileClient.java}}
file seems a bit out-of place being inside the {{beeline}} module...it would be
great to move test related things like this to under {{itests}} if
possible...I'm not sure if thats possible easily
+1
> QTest failure BeeLine escape_comments after HIVE-16045
> --
>
> Key: HIVE-16101
> URL: https://issues.apache.org/jira/browse/HIVE-16101
> Project: Hive
> Issue Type: Bug
> Components: Test
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-16101.patch
>
>
> HIVE-16045 committed immediately after HIVE-14459, and added two extra lines
> to the output which is written there with another thread. We should remove
> these lines before comparing the out file
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan reassigned HIVE-16116:
---
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-12631) LLAP: support ORC ACID tables
[
https://issues.apache.org/jira/browse/HIVE-12631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896732#comment-15896732
]
Hive QA commented on HIVE-12631:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856208/HIVE-12631.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10327 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=229)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_3]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
(batchId=147)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=224)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in]
(batchId=119)
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
(batchId=218)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3955/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3955/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3955/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856208 - PreCommit-HIVE-Build
> LLAP: support ORC ACID tables
> -
>
> Key: HIVE-12631
> URL: https://issues.apache.org/jira/browse/HIVE-12631
> Project: Hive
> Issue Type: Bug
> Components: llap, Transactions
>Reporter: Sergey Shelukhin
>Assignee: Teddy Choi
> Attachments: HIVE-12631.1.patch, HIVE-12631.2.patch,
> HIVE-12631.3.patch, HIVE-12631.4.patch
>
>
> LLAP uses a completely separate read path in ORC to allow for caching and
> parallelization of reads and processing. This path does not support ACID. As
> far as I remember ACID logic is embedded inside ORC format; we need to
> refactor it to be on top of some interface, if practical; or just port it to
> LLAP read path.
> Another consideration is how the logic will work with cache. The cache is
> currently low-level (CB-level in ORC), so we could just use it to read bases
> and deltas (deltas should be cached with higher priority) and merge as usual.
> We could also cache merged representation in future.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16115) Stop printing progress info from operation logs with beeline progress bar
[ https://issues.apache.org/jira/browse/HIVE-16115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek reassigned HIVE-16115: -- > Stop printing progress info from operation logs with beeline progress bar > - > > Key: HIVE-16115 > URL: https://issues.apache.org/jira/browse/HIVE-16115 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 2.2.0 >Reporter: anishek >Assignee: anishek >Priority: Minor > Fix For: 2.2.0 > > > when in progress bar is enabled, we should not print the progress information > via the operations logs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16114) NullPointerException in TezSessionPoolManager when getting the session
[ https://issues.apache.org/jira/browse/HIVE-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896695#comment-15896695 ] Zhihua Deng commented on HIVE-16114: [~sershe] can you take a look? thanks > NullPointerException in TezSessionPoolManager when getting the session > -- > > Key: HIVE-16114 > URL: https://issues.apache.org/jira/browse/HIVE-16114 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Priority: Minor > Attachments: HIVE-16114.patch > > > hive version: apache-hive-2.1.1 > we use hue(3.11.0) connecting to the HiveServer2. when hue starts up, it > works with no problems, a few hours passed, when we use the same sql, an > exception about unable to execute TezTask will come into being. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16114) NullPointerException in TezSessionPoolManager when getting the session
[ https://issues.apache.org/jira/browse/HIVE-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-16114: --- Attachment: HIVE-16114.patch > NullPointerException in TezSessionPoolManager when getting the session > -- > > Key: HIVE-16114 > URL: https://issues.apache.org/jira/browse/HIVE-16114 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Priority: Minor > Attachments: HIVE-16114.patch > > > hive version: apache-hive-2.1.1 > we use hue(3.11.0) connecting to the HiveServer2. when hue starts up, it > works with no problems, a few hours passed, when we use the same sql, an > exception about unable to execute TezTask will come into being. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16114) NullPointerException in TezSessionPoolManager when getting the session
[ https://issues.apache.org/jira/browse/HIVE-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-16114: --- Description: hive version: apache-hive-2.1.1 we use hue(3.11.0) connecting to the HiveServer2. when hue starts up, it works with no problems, a few hours passed, when we use the same sql, an exception about unable to execute TezTask will come into being. was: hive version: apache-hive-2.1.1 we use hue(3.11.0) connecting to the HiveServer2. when hue starts up, it works with no problems, a few hours passed, when we use the same sql, an exception about unable to initialize TezTask will come into being. > NullPointerException in TezSessionPoolManager when getting the session > -- > > Key: HIVE-16114 > URL: https://issues.apache.org/jira/browse/HIVE-16114 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Priority: Minor > > hive version: apache-hive-2.1.1 > we use hue(3.11.0) connecting to the HiveServer2. when hue starts up, it > works with no problems, a few hours passed, when we use the same sql, an > exception about unable to execute TezTask will come into being. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16114) NullPointerException in TezSessionPoolManager when getting the session
[ https://issues.apache.org/jira/browse/HIVE-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-16114: --- Status: Patch Available (was: Open) > NullPointerException in TezSessionPoolManager when getting the session > -- > > Key: HIVE-16114 > URL: https://issues.apache.org/jira/browse/HIVE-16114 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Priority: Minor > > hive version: apache-hive-2.1.1 > we use hue(3.11.0) connecting to the HiveServer2. when hue starts up, it > works with no problems, a few hours passed, when we use the same sql, an > exception about unable to initialize TezTask will come into being. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16114) NullPointerException in TezSessionPoolManager when getting the session
[ https://issues.apache.org/jira/browse/HIVE-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-16114: --- Status: Open (was: Patch Available) > NullPointerException in TezSessionPoolManager when getting the session > -- > > Key: HIVE-16114 > URL: https://issues.apache.org/jira/browse/HIVE-16114 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Priority: Minor > > hive version: apache-hive-2.1.1 > we use hue(3.11.0) connecting to the HiveServer2. when hue starts up, it > works with no problems, a few hours passed, when we use the same sql, an > exception about unable to initialize TezTask will come into being. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16114) NullPointerException in TezSessionPoolManager when getting the session
[ https://issues.apache.org/jira/browse/HIVE-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896682#comment-15896682 ] Zhihua Deng commented on HIVE-16114: the blame code snippet: if (doAsEnabled != session.getConf().getBoolVar(HiveConf.ConfVars.HIVE_SERVER2_ENABLE_DOAS)) it seems that the session got from threadlocal has not been opened yet or has been closed, So the session.getConf() is probably null. > NullPointerException in TezSessionPoolManager when getting the session > -- > > Key: HIVE-16114 > URL: https://issues.apache.org/jira/browse/HIVE-16114 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Priority: Minor > > hive version: apache-hive-2.1.1 > we use hue(3.11.0) connecting to the HiveServer2. when hue starts up, it > works with no problems, a few hours passed, when we use the same sql, an > exception about unable to initialize TezTask will come into being. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-12631) LLAP: support ORC ACID tables
[ https://issues.apache.org/jira/browse/HIVE-12631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-12631: -- Attachment: HIVE-12631.4.patch This 4th patch makes OrcAcidEncodedDataConsumer to close deleteEventRegistry when it's done. > LLAP: support ORC ACID tables > - > > Key: HIVE-12631 > URL: https://issues.apache.org/jira/browse/HIVE-12631 > Project: Hive > Issue Type: Bug > Components: llap, Transactions >Reporter: Sergey Shelukhin >Assignee: Teddy Choi > Attachments: HIVE-12631.1.patch, HIVE-12631.2.patch, > HIVE-12631.3.patch, HIVE-12631.4.patch > > > LLAP uses a completely separate read path in ORC to allow for caching and > parallelization of reads and processing. This path does not support ACID. As > far as I remember ACID logic is embedded inside ORC format; we need to > refactor it to be on top of some interface, if practical; or just port it to > LLAP read path. > Another consideration is how the logic will work with cache. The cache is > currently low-level (CB-level in ORC), so we could just use it to read bases > and deltas (deltas should be cached with higher priority) and merge as usual. > We could also cache merged representation in future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16114) NullPointerException in TezSessionPoolManager when getting the session
[ https://issues.apache.org/jira/browse/HIVE-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896677#comment-15896677 ] Zhihua Deng commented on HIVE-16114: the corresponding HiveServer2 stack: 2017-03-02T10:50:31,986 ERROR [HiveServer2-Background-Pool: Thread-982] exec.Task: Failed to execute tez graph. java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:430) at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:451) at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:396) at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:134) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1166) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242) at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:334) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1660) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:347) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) > NullPointerException in TezSessionPoolManager when getting the session > -- > > Key: HIVE-16114 > URL: https://issues.apache.org/jira/browse/HIVE-16114 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Priority: Minor > > hive version: apache-hive-2.1.1 > we use hue(3.11.0) connecting to the HiveServer2. when hue starts up, it > works with no problems, a few hours passed, when we use the same sql, an > exception about unable to initialize TezTask will come into being. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16048) Hive UDF doesn't get the right evaluate method
[
https://issues.apache.org/jira/browse/HIVE-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896663#comment-15896663
]
Hive QA commented on HIVE-16048:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856204/HIVE-16048.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 10327 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=229)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autogen_colalias]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_udf_percentile2]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_udf_percentile]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constantfolding]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_udaf] (batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_percentile]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_percentile]
(batchId=59)
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[udaf_example_max]
(batchId=223)
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[udaf_example_min]
(batchId=223)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_windowing_2]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[windowing]
(batchId=149)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[udf_percentile]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[windowing]
(batchId=117)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3954/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3954/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3954/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856204 - PreCommit-HIVE-Build
> Hive UDF doesn't get the right evaluate method
> --
>
> Key: HIVE-16048
> URL: https://issues.apache.org/jira/browse/HIVE-16048
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.1.1
>Reporter: Liao, Xiaoge
> Attachments: HIVE-16048.3.patch
>
>
> Hive UDF doesn't get the right evaluate method if there is variable parameter
> in the method of evaluate.
> For example:
> public class TestUdf extends UDF {
> public String evaluate(String a, String b) throws ParseException {
> return a + ":" + b;
> }
> public String evaluate(String a, String... b) throws ParseException {
> return b[0] + ":" + a;
> }
> }
> the udf may get the wrong result
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16048) Hive UDF doesn't get the right evaluate method
[
https://issues.apache.org/jira/browse/HIVE-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liao, Xiaoge updated HIVE-16048:
Status: Patch Available (was: Open)
> Hive UDF doesn't get the right evaluate method
> --
>
> Key: HIVE-16048
> URL: https://issues.apache.org/jira/browse/HIVE-16048
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.1.1
>Reporter: Liao, Xiaoge
> Attachments: HIVE-16048.3.patch
>
>
> Hive UDF doesn't get the right evaluate method if there is variable parameter
> in the method of evaluate.
> For example:
> public class TestUdf extends UDF {
> public String evaluate(String a, String b) throws ParseException {
> return a + ":" + b;
> }
> public String evaluate(String a, String... b) throws ParseException {
> return b[0] + ":" + a;
> }
> }
> the udf may get the wrong result
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16048) Hive UDF doesn't get the right evaluate method
[
https://issues.apache.org/jira/browse/HIVE-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liao, Xiaoge updated HIVE-16048:
Status: Open (was: Patch Available)
> Hive UDF doesn't get the right evaluate method
> --
>
> Key: HIVE-16048
> URL: https://issues.apache.org/jira/browse/HIVE-16048
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.1.1
>Reporter: Liao, Xiaoge
> Attachments: HIVE-16048.3.patch
>
>
> Hive UDF doesn't get the right evaluate method if there is variable parameter
> in the method of evaluate.
> For example:
> public class TestUdf extends UDF {
> public String evaluate(String a, String b) throws ParseException {
> return a + ":" + b;
> }
> public String evaluate(String a, String... b) throws ParseException {
> return b[0] + ":" + a;
> }
> }
> the udf may get the wrong result
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16048) Hive UDF doesn't get the right evaluate method
[
https://issues.apache.org/jira/browse/HIVE-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liao, Xiaoge updated HIVE-16048:
Component/s: (was: UDF)
Query Planning
> Hive UDF doesn't get the right evaluate method
> --
>
> Key: HIVE-16048
> URL: https://issues.apache.org/jira/browse/HIVE-16048
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.1.1
>Reporter: Liao, Xiaoge
> Attachments: HIVE-16048.3.patch
>
>
> Hive UDF doesn't get the right evaluate method if there is variable parameter
> in the method of evaluate.
> For example:
> public class TestUdf extends UDF {
> public String evaluate(String a, String b) throws ParseException {
> return a + ":" + b;
> }
> public String evaluate(String a, String... b) throws ParseException {
> return b[0] + ":" + a;
> }
> }
> the udf may get the wrong result
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16048) Hive UDF doesn't get the right evaluate method
[
https://issues.apache.org/jira/browse/HIVE-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liao, Xiaoge updated HIVE-16048:
Attachment: HIVE-16048.3.patch
> Hive UDF doesn't get the right evaluate method
> --
>
> Key: HIVE-16048
> URL: https://issues.apache.org/jira/browse/HIVE-16048
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.1.1
>Reporter: Liao, Xiaoge
> Attachments: HIVE-16048.3.patch
>
>
> Hive UDF doesn't get the right evaluate method if there is variable parameter
> in the method of evaluate.
> For example:
> public class TestUdf extends UDF {
> public String evaluate(String a, String b) throws ParseException {
> return a + ":" + b;
> }
> public String evaluate(String a, String... b) throws ParseException {
> return b[0] + ":" + a;
> }
> }
> the udf may get the wrong result
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16048) Hive UDF doesn't get the right evaluate method
[
https://issues.apache.org/jira/browse/HIVE-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liao, Xiaoge updated HIVE-16048:
Attachment: (was: HIVE-16048.2.patch)
> Hive UDF doesn't get the right evaluate method
> --
>
> Key: HIVE-16048
> URL: https://issues.apache.org/jira/browse/HIVE-16048
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.1.1
>Reporter: Liao, Xiaoge
>
> Hive UDF doesn't get the right evaluate method if there is variable parameter
> in the method of evaluate.
> For example:
> public class TestUdf extends UDF {
> public String evaluate(String a, String b) throws ParseException {
> return a + ":" + b;
> }
> public String evaluate(String a, String... b) throws ParseException {
> return b[0] + ":" + a;
> }
> }
> the udf may get the wrong result
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16048) Hive UDF doesn't get the right evaluate method
[
https://issues.apache.org/jira/browse/HIVE-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liao, Xiaoge updated HIVE-16048:
Attachment: (was: HIVE-16048.1.patch)
> Hive UDF doesn't get the right evaluate method
> --
>
> Key: HIVE-16048
> URL: https://issues.apache.org/jira/browse/HIVE-16048
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.1.1
>Reporter: Liao, Xiaoge
>
> Hive UDF doesn't get the right evaluate method if there is variable parameter
> in the method of evaluate.
> For example:
> public class TestUdf extends UDF {
> public String evaluate(String a, String b) throws ParseException {
> return a + ":" + b;
> }
> public String evaluate(String a, String... b) throws ParseException {
> return b[0] + ":" + a;
> }
> }
> the udf may get the wrong result
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16082) Allow user to change number of listener thread in LlapTaskCommunicator
[ https://issues.apache.org/jira/browse/HIVE-16082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896624#comment-15896624 ] Lefty Leverenz commented on HIVE-16082: --- Doc note: This adds *hive.llap.task.communicator.listener.thread-count* to HiveConf.java, so it needs to be documented in the LLAP section of Configuration Properties. * [Configuration Properties -- LLAP | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-LLAP] Added a TODOC2.2 label. > Allow user to change number of listener thread in LlapTaskCommunicator > -- > > Key: HIVE-16082 > URL: https://issues.apache.org/jira/browse/HIVE-16082 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-16082.1.patch, HIVE-16082.2.patch > > > Now LlapTaskCommunicator always has same number of RPC listener thread with > TezTaskCommunicatorImpl. There are scenarios when we want them different: for > example, in Llap only mode, we want less TezTaskCommunicatorImpl's listener > thread to reduce off-heap memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16082) Allow user to change number of listener thread in LlapTaskCommunicator
[ https://issues.apache.org/jira/browse/HIVE-16082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-16082: -- Labels: TODOC2.2 (was: ) > Allow user to change number of listener thread in LlapTaskCommunicator > -- > > Key: HIVE-16082 > URL: https://issues.apache.org/jira/browse/HIVE-16082 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-16082.1.patch, HIVE-16082.2.patch > > > Now LlapTaskCommunicator always has same number of RPC listener thread with > TezTaskCommunicatorImpl. There are scenarios when we want them different: for > example, in Llap only mode, we want less TezTaskCommunicatorImpl's listener > thread to reduce off-heap memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896586#comment-15896586
]
Hive QA commented on HIVE-16113:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856193/HIVE-16113.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10324 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=229)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcr] (batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[push_or] (batchId=15)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_views]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_views]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[special_character_in_tabnames_1]
(batchId=147)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=224)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[pcr] (batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3953/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3953/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3953/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856193 - PreCommit-HIVE-Build
> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.2.0, 2.1.1
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-16113.1.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16057) SchemaTool ignores --passWord argument if hadoop.security.credential.provider.path is configured
[
https://issues.apache.org/jira/browse/HIVE-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896558#comment-15896558
]
Peter Vary commented on HIVE-16057:
---
The failures are not related.
> SchemaTool ignores --passWord argument if
> hadoop.security.credential.provider.path is configured
>
>
> Key: HIVE-16057
> URL: https://issues.apache.org/jira/browse/HIVE-16057
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-16057.02.patch, HIVE-16057.03.patch,
> HIVE-16057.patch
>
>
> It the {{hadoop.security.credential.provider.path}} is defined in command
> line, but the correct {{HADOOP_CREDSTORE_PASSWORD}} is not provided the
> SchemaTool fails, even if the correct metastore password is provided with
> {{--passWord}}
> Could be reproduced if the hive-site.xml contains the following:
> {code}
>
> hadoop.security.credential.provider.path
>
> localjceks://file//Users/petervary/tmp/conf/creds.localjceks
>
> {code}
> {code}
> $ ../schemaTool --dbType=mysql --info --passWord=pwd
> Metastore connection URL:
> jdbc:mysql://localhost:3306/hive?useUnicode=true=UTF-8
> Metastore Connection Driver : com.mysql.jdbc.Driver
> Metastore connection User: hive
> org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema
> version.
> *** schemaTool failed ***
> {code}
> The {{--passWord}} argument should override the errors from the credential
> provider
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-16113:
---
Summary: PartitionPruner::removeNonPartCols needs to handle AND/OR cases
(was: PartitionPruner::removeNonPartCols needs to handle AND/OR specially)
> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.2.0, 2.1.1
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-16113.1.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR specially
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-16113:
---
Summary: PartitionPruner::removeNonPartCols needs to handle AND/OR
specially (was: PartitionPruner::removeNonPartCols breaks NULL based udfs like
NVL)
> PartitionPruner::removeNonPartCols needs to handle AND/OR specially
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.2.0, 2.1.1
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-16113.1.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16113) PartitionPruner::removeNonPartCols breaks NULL based udfs like NVL
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-16113:
---
Attachment: HIVE-16113.1.patch
> PartitionPruner::removeNonPartCols breaks NULL based udfs like NVL
> --
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.2.0, 2.1.1
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-16113.1.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16113) PartitionPruner::removeNonPartCols breaks NULL based udfs like NVL
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-16113:
---
Status: Patch Available (was: Open)
> PartitionPruner::removeNonPartCols breaks NULL based udfs like NVL
> --
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 2.1.1, 1.2.1, 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-16113.1.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16057) SchemaTool ignores --passWord argument if hadoop.security.credential.provider.path is configured
[
https://issues.apache.org/jira/browse/HIVE-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896535#comment-15896535
]
Hive QA commented on HIVE-16057:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856185/HIVE-16057.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10327 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=229)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
(batchId=147)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in]
(batchId=119)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3952/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3952/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3952/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856185 - PreCommit-HIVE-Build
> SchemaTool ignores --passWord argument if
> hadoop.security.credential.provider.path is configured
>
>
> Key: HIVE-16057
> URL: https://issues.apache.org/jira/browse/HIVE-16057
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-16057.02.patch, HIVE-16057.03.patch,
> HIVE-16057.patch
>
>
> It the {{hadoop.security.credential.provider.path}} is defined in command
> line, but the correct {{HADOOP_CREDSTORE_PASSWORD}} is not provided the
> SchemaTool fails, even if the correct metastore password is provided with
> {{--passWord}}
> Could be reproduced if the hive-site.xml contains the following:
> {code}
>
> hadoop.security.credential.provider.path
>
> localjceks://file//Users/petervary/tmp/conf/creds.localjceks
>
> {code}
> {code}
> $ ../schemaTool --dbType=mysql --info --passWord=pwd
> Metastore connection URL:
> jdbc:mysql://localhost:3306/hive?useUnicode=true=UTF-8
> Metastore Connection Driver : com.mysql.jdbc.Driver
> Metastore connection User: hive
> org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema
> version.
> *** schemaTool failed ***
> {code}
> The {{--passWord}} argument should override the errors from the credential
> provider
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16113) PartitionPruner::removeNonPartCols breaks NULL based udfs like NVL
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V reassigned HIVE-16113:
--
Assignee: Gopal V
> PartitionPruner::removeNonPartCols breaks NULL based udfs like NVL
> --
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.2.0, 2.1.1
>Reporter: Gopal V
>Assignee: Gopal V
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16101) QTest failure BeeLine escape_comments after HIVE-16045
[ https://issues.apache.org/jira/browse/HIVE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896501#comment-15896501 ] Peter Vary commented on HIVE-16101: --- [~ngangam]: Could you please review. I do not want the BeeLine tests start as a flaky ones :) Thanks, Peter > QTest failure BeeLine escape_comments after HIVE-16045 > -- > > Key: HIVE-16101 > URL: https://issues.apache.org/jira/browse/HIVE-16101 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-16101.patch > > > HIVE-16045 committed immediately after HIVE-14459, and added two extra lines > to the output which is written there with another thread. We should remove > these lines before comparing the out file -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16057) SchemaTool ignores --passWord argument if hadoop.security.credential.provider.path is configured
[
https://issues.apache.org/jira/browse/HIVE-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16057:
--
Attachment: HIVE-16057.03.patch
Was not aware, that there are itests for SchemaTool too... Always learning :D
Modified the test to set username and password before running the schematool.
[~aihuaxu]: What do you think about this change? Is it ok, or using the
schematool programmatically is an intended use so the doInit method should try
to load the username/password too?
Thanks,
Peter
> SchemaTool ignores --passWord argument if
> hadoop.security.credential.provider.path is configured
>
>
> Key: HIVE-16057
> URL: https://issues.apache.org/jira/browse/HIVE-16057
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-16057.02.patch, HIVE-16057.03.patch,
> HIVE-16057.patch
>
>
> It the {{hadoop.security.credential.provider.path}} is defined in command
> line, but the correct {{HADOOP_CREDSTORE_PASSWORD}} is not provided the
> SchemaTool fails, even if the correct metastore password is provided with
> {{--passWord}}
> Could be reproduced if the hive-site.xml contains the following:
> {code}
>
> hadoop.security.credential.provider.path
>
> localjceks://file//Users/petervary/tmp/conf/creds.localjceks
>
> {code}
> {code}
> $ ../schemaTool --dbType=mysql --info --passWord=pwd
> Metastore connection URL:
> jdbc:mysql://localhost:3306/hive?useUnicode=true=UTF-8
> Metastore Connection Driver : com.mysql.jdbc.Driver
> Metastore connection User: hive
> org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema
> version.
> *** schemaTool failed ***
> {code}
> The {{--passWord}} argument should override the errors from the credential
> provider
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16057) SchemaTool ignores --passWord argument if hadoop.security.credential.provider.path is configured
[
https://issues.apache.org/jira/browse/HIVE-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896472#comment-15896472
]
Hive QA commented on HIVE-16057:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856179/HIVE-16057.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 10327 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=229)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
(batchId=147)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=224)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=224)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in]
(batchId=119)
org.apache.hive.beeline.TestSchemaTool.testNestedScriptsForDerby (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testNestedScriptsForMySQL (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testNestedScriptsForOracle (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testPostgresFilter (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testSchemaInit (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testSchemaInitDryRun (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgradeDryRun (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testScriptMultiRowComment (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testScriptWithDelimiter (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testScripts (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testValidateLocations (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testValidateNullValues (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testValidateSchemaTables (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testValidateSchemaVersions (batchId=212)
org.apache.hive.beeline.TestSchemaTool.testValidateSequences (batchId=212)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3951/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3951/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3951/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856179 - PreCommit-HIVE-Build
> SchemaTool ignores --passWord argument if
> hadoop.security.credential.provider.path is configured
>
>
> Key: HIVE-16057
> URL: https://issues.apache.org/jira/browse/HIVE-16057
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-16057.02.patch, HIVE-16057.patch
>
>
> It the {{hadoop.security.credential.provider.path}} is defined in command
> line, but the correct {{HADOOP_CREDSTORE_PASSWORD}} is not provided the
> SchemaTool fails, even if the correct metastore password is provided with
> {{--passWord}}
> Could be reproduced if the hive-site.xml contains the following:
> {code}
>
> hadoop.security.credential.provider.path
>
> localjceks://file//Users/petervary/tmp/conf/creds.localjceks
>
> {code}
> {code}
> $ ../schemaTool --dbType=mysql --info --passWord=pwd
> Metastore connection URL:
> jdbc:mysql://localhost:3306/hive?useUnicode=true=UTF-8
> Metastore Connection Driver : com.mysql.jdbc.Driver
> Metastore connection User: hive
> org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema
> version.
> *** schemaTool failed ***
> {code}
> The {{--passWord}} argument should override the errors from the credential
> provider
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16113) PartitionPruner::removeNonPartCols breaks NULL based udfs like NVL
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896457#comment-15896457
]
Gopal V commented on HIVE-16113:
{code}
2017-03-05T12:37:46,379 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = dt
2017-03-05T12:37:46,379 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = '2001-01-01'
2017-03-05T12:37:46,379 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = (dt = '2001-01-01')
2017-03-05T12:37:46,380 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = 1
2017-03-05T12:37:46,380 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = (null = 1)
2017-03-05T12:37:46,380 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = ((dt = '2001-01-01') and (null = 1))
2017-03-05T12:37:46,380 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = false
2017-03-05T12:37:46,380 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and (null = 1)),false)
2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = dt
2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = '2001-01-01'
2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = (dt = '2001-01-01')
2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = null
2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = ((dt = '2001-01-01') and null)
2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = false
2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
{code}
> PartitionPruner::removeNonPartCols breaks NULL based udfs like NVL
> --
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.2.0, 2.1.1
>Reporter: Gopal V
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16113) PartitionPruner::removeNonPartCols breaks NULL based udfs like NVL
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-16113:
---
Description:
{code}
create table daysales (customer int) partitioned by (dt string);
insert into daysales partition(dt='2001-01-01') values(1);
select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
0 ROWS
{code}
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
{code}
2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
{code}
Because {{true and null => null}}, this turns into {{NVL(null, false)}}
was:
{code}
create table daysales (customer int) partitioned by (dt string);
insert into daysales partition(dt='2001-01-01') values(1);
select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
0 ROWS
{code}
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
{code}
2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
{code}
Because {{ true and null => null }}, this turns into {{ NVL(null, false) }}
> PartitionPruner::removeNonPartCols breaks NULL based udfs like NVL
> --
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.2.0, 2.1.1
>Reporter: Gopal V
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16113) PartitionPruner::removeNonPartCols breaks NULL based udfs like NVL
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-16113:
---
Description:
{code}
create table daysales (customer int) partitioned by (dt string);
insert into daysales partition(dt='2001-01-01') values(1);
select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
0 ROWS
{code}
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
{code}
2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
{code}
Because {{ true and null => null }}, this turns into {{ NVL(null, false) }}
was:
{code}
create table daysales (customer int) partitioned by (dt string);
insert into daysales partition(dt='2001-01-01') values(1);
select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
0 ROWS
{code}
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> PartitionPruner::removeNonPartCols breaks NULL based udfs like NVL
> --
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.2.0, 2.1.1
>Reporter: Gopal V
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{ true and null => null }}, this turns into {{ NVL(null, false) }}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16057) SchemaTool ignores --passWord argument if hadoop.security.credential.provider.path is configured
[
https://issues.apache.org/jira/browse/HIVE-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16057:
--
Attachment: HIVE-16057.02.patch
Build infrastructure failure:
{code}
+ curl -s -S --location --retry 3
https://issues.apache.org/jira/browse/HIVE-16057
curl: (7) Failed to connect to issues.apache.org port 443: No route to host
{code}
Let's see if it is solved, or not
> SchemaTool ignores --passWord argument if
> hadoop.security.credential.provider.path is configured
>
>
> Key: HIVE-16057
> URL: https://issues.apache.org/jira/browse/HIVE-16057
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-16057.02.patch, HIVE-16057.patch
>
>
> It the {{hadoop.security.credential.provider.path}} is defined in command
> line, but the correct {{HADOOP_CREDSTORE_PASSWORD}} is not provided the
> SchemaTool fails, even if the correct metastore password is provided with
> {{--passWord}}
> Could be reproduced if the hive-site.xml contains the following:
> {code}
>
> hadoop.security.credential.provider.path
>
> localjceks://file//Users/petervary/tmp/conf/creds.localjceks
>
> {code}
> {code}
> $ ../schemaTool --dbType=mysql --info --passWord=pwd
> Metastore connection URL:
> jdbc:mysql://localhost:3306/hive?useUnicode=true=UTF-8
> Metastore Connection Driver : com.mysql.jdbc.Driver
> Metastore connection User: hive
> org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema
> version.
> *** schemaTool failed ***
> {code}
> The {{--passWord}} argument should override the errors from the credential
> provider
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16113) PartitionPruner::removeNonPartCols breaks NULL based udfs like NVL
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-16113:
---
Description:
{code}
create table daysales (customer int) partitioned by (dt string);
insert into daysales partition(dt='2001-01-01') values(1);
select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
0 ROWS
{code}
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
was:
{code}
create table daysales (customer int) partitioned by (dt string);
insert into daysales partition(dt='2001-01-01') values(1);
select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
0 ROWS
{code}
> PartitionPruner::removeNonPartCols breaks NULL based udfs like NVL
> --
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.2.0, 2.1.1
>Reporter: Gopal V
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-14943) Base Implementation
[ https://issues.apache.org/jira/browse/HIVE-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896152#comment-15896152 ] Lefty Leverenz commented on HIVE-14943: --- Doc note: Although this will probably be documented with HIVE-10924, don't forget to add the new keywords MERGE and MATCHED to the list in the wiki for release 2.2.0. MATCHED is a non-reserved keyword and MERGE (after HIVE-16070) is reserved. * [DDL -- Keywords | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Keywords,Non-reservedKeywordsandReservedKeywords] Added a TODOC2.2 label. > Base Implementation > --- > > Key: HIVE-14943 > URL: https://issues.apache.org/jira/browse/HIVE-14943 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14943.10.patch, HIVE-14943.11.patch, > HIVE-14943.12.patch, HIVE-14943.13.patch, HIVE-14943.15.patch, > HIVE-14943.2.patch, HIVE-14943.3.patch, HIVE-14943.4.patch, > HIVE-14943.5.patch, HIVE-14943.6.patch, HIVE-14943.7.patch, > HIVE-14943.8.patch, HIVE-14943.9.patch, HIVE-14943.patch > > > Create the 1st pass functional implementation of MERGE > This should run e2e and produce correct results. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14943) Base Implementation
[ https://issues.apache.org/jira/browse/HIVE-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-14943: -- Labels: TODOC2.2 (was: ) > Base Implementation > --- > > Key: HIVE-14943 > URL: https://issues.apache.org/jira/browse/HIVE-14943 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14943.10.patch, HIVE-14943.11.patch, > HIVE-14943.12.patch, HIVE-14943.13.patch, HIVE-14943.15.patch, > HIVE-14943.2.patch, HIVE-14943.3.patch, HIVE-14943.4.patch, > HIVE-14943.5.patch, HIVE-14943.6.patch, HIVE-14943.7.patch, > HIVE-14943.8.patch, HIVE-14943.9.patch, HIVE-14943.patch > > > Create the 1st pass functional implementation of MERGE > This should run e2e and produce correct results. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16070) fix nonReserved list in IdentifiersParser.q
[ https://issues.apache.org/jira/browse/HIVE-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896143#comment-15896143 ] Lefty Leverenz commented on HIVE-16070: --- Doc note: MERGE should be included in the DDL doc's list of reserved keywords for release 2.2.0. But I won't add a TODOC2.2 label because it's part of the documentation for HIVE-14943. * [DDL -- Reserved Keywords | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ReservedKeywords] Quibble: This jira's summary (title) says IdentifiersParser.q but ".q" should be ".g". > fix nonReserved list in IdentifiersParser.q > --- > > Key: HIVE-16070 > URL: https://issues.apache.org/jira/browse/HIVE-16070 > Project: Hive > Issue Type: Bug > Components: Query Planning, Query Processor, Transactions >Affects Versions: 2.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 2.2.0 > > Attachments: HIVE-16070.01.patch, HIVE-16070.02.patch, > HIVE-16070.03.patch > > > some keywords that should have been reserved > (https://www.postgresql.org/docs/9.5/static/sql-keywords-appendix.html) are > listed in nonReserved list. Should fix this. > KW_MERGE > KW_START > KW_COMMIT > KW_ROLLBACK > KW_ONLY -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16100) Dynamic Sorted Partition optimizer loses sibling operators
[
https://issues.apache.org/jira/browse/HIVE-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896142#comment-15896142
]
Gopal V commented on HIVE-16100:
The query23.q is a flaky result - test failures are unrelated.
{code}
ql/src/test/results/clientpositive/perf/query23.q.out
1,2d0
< Warning: Shuffle Join MERGEJOIN[382][tables = [$hdt$_1, $hdt$_2]] in Stage
'Reducer 51' is a cross product
< Warning: Shuffle Join MERGEJOIN[383][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in
Stage 'Reducer 52' is a cross product
4a3,4
> Warning: Shuffle Join MERGEJOIN[382][tables = [$hdt$_1, $hdt$_2]] in Stage
> 'Reducer 51' is a cross product
> Warning: Shuffle Join MERGEJOIN[383][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in
> Stage 'Reducer 52' is a cross product
{code}
[~prasanth_j]: can you review?
> Dynamic Sorted Partition optimizer loses sibling operators
> --
>
> Key: HIVE-16100
> URL: https://issues.apache.org/jira/browse/HIVE-16100
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1, 2.2.0, 2.1.1
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-16100.1.patch, HIVE-16100.2.patch,
> HIVE-16100.2.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java#L173
> {code}
> // unlink connection between FS and its parent
> fsParent = fsOp.getParentOperators().get(0);
> fsParent.getChildOperators().clear();
> {code}
> The optimizer discards any cases where the fsParent has another SEL child
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-14754) Track the queries execution lifecycle times
[ https://issues.apache.org/jira/browse/HIVE-14754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896138#comment-15896138 ] Lefty Leverenz commented on HIVE-14754: --- Thanks for the fix, [~zsombor.klara]. I updated the wiki: * [Hive Metrics | https://cwiki.apache.org/confluence/display/Hive/Hive+Metrics] > Track the queries execution lifecycle times > --- > > Key: HIVE-14754 > URL: https://issues.apache.org/jira/browse/HIVE-14754 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2 >Affects Versions: 2.2.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Fix For: 2.2.0 > > Attachments: HIVE-14754.1.patch, HIVE-14754.2.patch, HIVE-14754.patch > > > We should be able to track the nr. of queries being compiled/executed at any > given time, as well as the duration of the execution and compilation phase. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15864) Fix typo introduced in HIVE-14754
[ https://issues.apache.org/jira/browse/HIVE-15864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896136#comment-15896136 ] Lefty Leverenz commented on HIVE-15864: --- The wiki has been updated for this fix: * [Hive Metrics | https://cwiki.apache.org/confluence/display/Hive/Hive+Metrics] > Fix typo introduced in HIVE-14754 > - > > Key: HIVE-15864 > URL: https://issues.apache.org/jira/browse/HIVE-15864 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2, Metastore >Affects Versions: 2.2.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Fix For: 2.2.0 > > Attachments: HIVE-15864.patch > > > hs2_suceeded_queries needs another "c": hs2_succeeded_queries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
