[jira] [Commented] (HIVE-16663) String Caching For Rows
[
https://issues.apache.org/jira/browse/HIVE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009172#comment-16009172
]
Hive QA commented on HIVE-16663:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867864/HIVE-16663.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5243/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5243/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5243/
Messages:
{noformat}
This message was trimmed, see log for full details
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/ql/target/hive-exec-3.0.0-SNAPSHOT.jar(org/apache/commons/lang3/ArrayUtils.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/common/target/hive-common-3.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceStability.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-hdfs/2.8.0/hadoop-hdfs-2.8.0.jar(org/apache/hadoop/hdfs/web/AuthFilter.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-3.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/Utils.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.8.0/hadoop-common-2.8.0.jar(org/apache/hadoop/security/UserGroupInformation.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/2.8.0/hadoop-auth-2.8.0.jar(org/apache/hadoop/security/authentication/client/PseudoAuthenticator.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/2.8.0/hadoop-auth-2.8.0.jar(org/apache/hadoop/security/authentication/server/PseudoAuthenticationHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.8.0/hadoop-common-2.8.0.jar(org/apache/hadoop/util/GenericOptionsParser.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-rewrite/9.3.8.v20160314/jetty-rewrite-9.3.8.v20160314.jar(org/eclipse/jetty/rewrite/handler/RedirectPatternRule.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-rewrite/9.3.8.v20160314/jetty-rewrite-9.3.8.v20160314.jar(org/eclipse/jetty/rewrite/handler/RewriteHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/Handler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/Server.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/ServerConnector.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/handler/HandlerList.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/FilterHolder.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/ServletContextHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/ServletHolder.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-xml/9.3.8.v20160314/jetty-xml-9.3.8.v20160314.jar(org/eclipse/jetty/xml/XmlConfiguration.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/slf4j/jul-to-slf4j/1.7.10/jul-to-slf4j-1.7.10.jar(org/slf4j/bridge/SLF4JBridgeHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/DispatcherType.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar(javax/servlet/http/HttpServletRequest.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/common/target/hive-common-3.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceAudience$LimitedPrivate.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/common/target/hive-co
[jira] [Commented] (HIVE-16663) String Caching For Rows
[
https://issues.apache.org/jira/browse/HIVE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009169#comment-16009169
]
Hive QA commented on HIVE-16663:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867864/HIVE-16663.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5242/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5242/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5242/
Messages:
{noformat}
This message was trimmed, see log for full details
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/ql/target/hive-exec-3.0.0-SNAPSHOT.jar(org/apache/commons/lang3/ArrayUtils.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/common/target/hive-common-3.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceStability.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-hdfs/2.8.0/hadoop-hdfs-2.8.0.jar(org/apache/hadoop/hdfs/web/AuthFilter.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-3.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/Utils.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.8.0/hadoop-common-2.8.0.jar(org/apache/hadoop/security/UserGroupInformation.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/2.8.0/hadoop-auth-2.8.0.jar(org/apache/hadoop/security/authentication/client/PseudoAuthenticator.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/2.8.0/hadoop-auth-2.8.0.jar(org/apache/hadoop/security/authentication/server/PseudoAuthenticationHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.8.0/hadoop-common-2.8.0.jar(org/apache/hadoop/util/GenericOptionsParser.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-rewrite/9.3.8.v20160314/jetty-rewrite-9.3.8.v20160314.jar(org/eclipse/jetty/rewrite/handler/RedirectPatternRule.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-rewrite/9.3.8.v20160314/jetty-rewrite-9.3.8.v20160314.jar(org/eclipse/jetty/rewrite/handler/RewriteHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/Handler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/Server.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/ServerConnector.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-server/9.3.8.v20160314/jetty-server-9.3.8.v20160314.jar(org/eclipse/jetty/server/handler/HandlerList.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/FilterHolder.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/ServletContextHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-servlet/9.3.8.v20160314/jetty-servlet-9.3.8.v20160314.jar(org/eclipse/jetty/servlet/ServletHolder.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-xml/9.3.8.v20160314/jetty-xml-9.3.8.v20160314.jar(org/eclipse/jetty/xml/XmlConfiguration.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/slf4j/jul-to-slf4j/1.7.10/jul-to-slf4j-1.7.10.jar(org/slf4j/bridge/SLF4JBridgeHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/DispatcherType.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar(javax/servlet/http/HttpServletRequest.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/common/target/hive-common-3.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceAudience$LimitedPrivate.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/common/target/hive-co
[jira] [Commented] (HIVE-16653) Mergejoin should give itself a correct tag
[
https://issues.apache.org/jira/browse/HIVE-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009167#comment-16009167
]
Hive QA commented on HIVE-16653:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867817/HIVE-16653.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10699 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5241/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5241/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5241/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867817 - PreCommit-HIVE-Build
> Mergejoin should give itself a correct tag
> --
>
> Key: HIVE-16653
> URL: https://issues.apache.org/jira/browse/HIVE-16653
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-16653.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16660) Not able to add partition for views in hive when sentry is enabled
[ https://issues.apache.org/jira/browse/HIVE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009160#comment-16009160 ] Yongzhi Chen commented on HIVE-16660: - [~Yibing], they are all related to partitions to the view. For example, If navigator hook is attached to HS2, add partition operation(after partition added) will trigger second issue. So in hue, user will see NPE even the first issue is solved. > Not able to add partition for views in hive when sentry is enabled > -- > > Key: HIVE-16660 > URL: https://issues.apache.org/jira/browse/HIVE-16660 > Project: Hive > Issue Type: Bug > Components: Parser >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-16660.1.patch > > > Repro: > create table tesnit (a int) partitioned by (p int); > insert into table tesnit partition (p = 1) values (1); > insert into table tesnit partition (p = 2) values (1); > create view test_view partitioned on (p) as select * from tesnit where p =1; > alter view test_view add partition (p = 2); > Error: Error while compiling statement: FAILED: SemanticException [Error > 10056]: The query does not reference any valid partition. To run this query, > set hive.mapred.mode=nonstrict (state=42000,code=10056) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16660) Not able to add partition for views in hive when sentry is enabled
[ https://issues.apache.org/jira/browse/HIVE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009153#comment-16009153 ] Yibing Shi commented on HIVE-16660: --- [~ychena], should we solve these 2 problems in 2 different JIRAs? They are not related. > Not able to add partition for views in hive when sentry is enabled > -- > > Key: HIVE-16660 > URL: https://issues.apache.org/jira/browse/HIVE-16660 > Project: Hive > Issue Type: Bug > Components: Parser >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-16660.1.patch > > > Repro: > create table tesnit (a int) partitioned by (p int); > insert into table tesnit partition (p = 1) values (1); > insert into table tesnit partition (p = 2) values (1); > create view test_view partitioned on (p) as select * from tesnit where p =1; > alter view test_view add partition (p = 2); > Error: Error while compiling statement: FAILED: SemanticException [Error > 10056]: The query does not reference any valid partition. To run this query, > set hive.mapred.mode=nonstrict (state=42000,code=10056) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16600) Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel order by in multi_insert cases
[
https://issues.apache.org/jira/browse/HIVE-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009144#comment-16009144
]
Hive QA commented on HIVE-16600:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867802/mr.explain
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5240/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5240/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5240/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-13 05:32:36.913
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-5240/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-13 05:32:36.916
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at f2fa83c HIVE-16658: TestTimestampTZ.java has missed the ASF
header (Saijin Huang via Rui)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at f2fa83c HIVE-16658: TestTimestampTZ.java has missed the ASF
header (Saijin Huang via Rui)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-13 05:32:37.843
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867802 - PreCommit-HIVE-Build
> Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel
> order by in multi_insert cases
>
>
> Key: HIVE-16600
> URL: https://issues.apache.org/jira/browse/HIVE-16600
> Project: Hive
> Issue Type: Sub-task
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16600.1.patch, HIVE-16600.2.patch,
> HIVE-16600.3.patch, mr.explain, mr.explain.log.HIVE-16600
>
>
> multi_insert_gby.case.q
> {code}
> set hive.exec.reducers.bytes.per.reducer=256;
> set hive.optimize.sampling.orderby=true;
> drop table if exists e1;
> drop table if exists e2;
> create table e1 (key string, value string);
> create table e2 (key string);
> FROM (select key, cast(key as double) as keyD, value from src order by key) a
> INSERT OVERWRITE TABLE e1
> SELECT key, value
> INSERT OVERWRITE TABLE e2
> SELECT key;
> select * from e1;
> select * from e2;
> {code}
> the parallelism of Sort is 1 even we enable parallel order
> by("hive.optimize.sampling.orderby" is set as "true"). This is not
> reasonable because the parallelism should be calcuated by
> [Utilities.estimateReducers|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L170]
> this is because SetSparkReducerParallelism#needSetParallelism returns false
> when [children size of
> RS|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L207]
> is greater than 1.
> in this case, the children size of {{RS[2]}} is two.
> the logical plan of the case
> {code}
>TS[0]-SEL[1]-RS[2]-SEL[3]-SEL[4]-FS[5]
> -SEL[6]-FS[7]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15725) Make it possible to run checkstyle for a specific module
[
https://issues.apache.org/jira/browse/HIVE-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009141#comment-16009141
]
Hive QA commented on HIVE-15725:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12849308/HIVE-15725.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10698 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5239/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5239/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5239/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12849308 - PreCommit-HIVE-Build
> Make it possible to run checkstyle for a specific module
>
>
> Key: HIVE-15725
> URL: https://issues.apache.org/jira/browse/HIVE-15725
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-15725.patch
>
>
> Remove relative paths from the checkstyle.xml, so it could be run for every
> module, not just from the root.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[
https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009127#comment-16009127
]
Hive QA commented on HIVE-14731:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867798/HIVE-14731.18.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 36 failed/errored test(s), 10702 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort]
(batchId=78)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_partition_pruning_2]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join0]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join29]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join30]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join_filters]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join_nulls]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_12]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_join]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_prod_1]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_prod_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_prod_4]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_product_check_2]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_partition_pruning]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[empty_join]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_1]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[jdbc_handler]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[leftsemijoin]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_exists]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_null_agg]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_select]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_between_columns]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_mapjoin]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_include_no_sel]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join_filters]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join_nulls]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_leftsemi_mapjoin]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_partition_pruning]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[hybridgrace_hashjoin_1]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5238/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5238/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5238/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 36 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867798 - PreCommit-HIVE-Build
> Use Tez cartesian product edge in Hive (unpartitioned case only)
>
>
> Key: HIVE-14731
[jira] [Updated] (HIVE-16658) TestTimestampTZ.java has missed the ASF header
[ https://issues.apache.org/jira/browse/HIVE-16658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-16658: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks Saijin for the contribution. > TestTimestampTZ.java has missed the ASF header > -- > > Key: HIVE-16658 > URL: https://issues.apache.org/jira/browse/HIVE-16658 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-16658.1.patch > > > when i test the issue HIVE-14412, TestTimestampTZ has missed the ASF header -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16607) ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID
[
https://issues.apache.org/jira/browse/HIVE-16607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009100#comment-16009100
]
Hive QA commented on HIVE-16607:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867786/HIVE-16607.06.patch
{color:green}SUCCESS:{color} +1 due to 13 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10699 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=236)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_25]
(batchId=83)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5237/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5237/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5237/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867786 - PreCommit-HIVE-Build
> ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID
> -
>
> Key: HIVE-16607
> URL: https://issues.apache.org/jira/browse/HIVE-16607
> Project: Hive
> Issue Type: Bug
> Components: Beeline, Logging
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16607.02.patch, HIVE-16607.04.patch,
> HIVE-16607.05.patch, HIVE-16607.06.patch, HIVE-16607.patch
>
>
> Creating a new {{QueryState}} object regenerates the HIVEQUERYID stored in
> the {{HiveConf}}.
> In HiveServer logs it makes hard to follow the life of the query since a new
> queryid is assigned to the query during the execution.
> Since BeeLine is showing the operation logs based on the queryid, only the
> first several line of the logs is showed in BeeLine.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16643) BeeLine tests output should keep the PREHOOK/POSTHOOK Input/Output orderdering
[
https://issues.apache.org/jira/browse/HIVE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009066#comment-16009066
]
Hive QA commented on HIVE-16643:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867785/HIVE-16643.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10699 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testSelectThriftSerializeInTasks
(batchId=225)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5236/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5236/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5236/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867785 - PreCommit-HIVE-Build
> BeeLine tests output should keep the PREHOOK/POSTHOOK Input/Output orderdering
> --
>
> Key: HIVE-16643
> URL: https://issues.apache.org/jira/browse/HIVE-16643
> Project: Hive
> Issue Type: New Feature
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16643.01.patch, HIVE-16643.patch
>
>
> The {{PreExecutePrinter}} and the {{PostExecutePrinter}} prints the query
> input and the output list in alphabetical order in {{printEntities}} method.
> Our goal is to have the same output from the BeeLine query tests, and the Cli
> query tests. Since the BeeLine tests are using test specific databases to run
> the tests, and only converting the results in the end to remove this specific
> database names from the output, we have to reorder the lists after this
> conversion.
> Raw BeeLine output:
> {code}
> [..]
> INFO : PREHOOK: Output: create_merge_compressed@src_rc_merge_test
> INFO : PREHOOK: Output: database:create_merge_compressed
> [..]
> {code}
> Before patch BeeLine output:
> {code}
> [..]
> PREHOOK: Output: default@src_rc_merge_test
> PREHOOK: Output: database:default
> [..]
> {code}
> Expected output:
> {code}
> [..]
> PREHOOK: Output: database:default
> PREHOOK: Output: default@src_rc_merge_test
> [..]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16647) Improve the validation output to make the output to stderr and stdout more consistent
[
https://issues.apache.org/jira/browse/HIVE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009037#comment-16009037
]
Hive QA commented on HIVE-16647:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867784/HIVE-16647.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10698 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5235/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5235/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5235/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867784 - PreCommit-HIVE-Build
> Improve the validation output to make the output to stderr and stdout more
> consistent
> -
>
> Key: HIVE-16647
> URL: https://issues.apache.org/jira/browse/HIVE-16647
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Minor
> Attachments: HIVE-16647.1.patch
>
>
> Some output are printed to stderr or stdout inconsistently. Here are some of
> them. Update to make them more consistent.
> * Version table validation
> When the version table is missing, the err msg goes to stderr
> When the version table is not valid, the err msg goes to stdout with a
> message like "Failed in schema version validation:
> * Metastore/schema table validation
> ** When the version table contains the wrong version or there are no rows in
> the version table, err msg goes to stderr
> ** When there diffs between the schema and metastore tables, the err msg goes
> to stdout
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15104) Hive on Spark generate more shuffle data than hive on mr
[ https://issues.apache.org/jira/browse/HIVE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009026#comment-16009026 ] Rui Li commented on HIVE-15104: --- [~xuefuz], kryo was relocated in HIVE-5915. So it's not intended for Spark. Actually, we're on the same version as Spark-2.0.0: kryo-shaded-3.0.3. bq. I'm concerned that class conflicts might come back if we stop relocating Kryo You're right. I'm not sure whether it's a conflict or loading issue, but when I tried to run some TPC-H benchmark, I got a ClassNotFoundException, although the class is there in hive-exec.jar. I'll see how to workaround this. BTW, the test in my last comment shuffles very little data. That's why optimizing the overhead can have a significant improvement. I guess this won't be the case in real world query. That's why I want to run some more serious benchmark. > Hive on Spark generate more shuffle data than hive on mr > > > Key: HIVE-15104 > URL: https://issues.apache.org/jira/browse/HIVE-15104 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.1 >Reporter: wangwenli >Assignee: Rui Li > Attachments: HIVE-15104.1.patch > > > the same sql, running on spark and mr engine, will generate different size > of shuffle data. > i think it is because of hive on mr just serialize part of HiveKey, but hive > on spark which using kryo will serialize full of Hivekey object. > what is your opionion? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16660) Not able to add partition for views in hive when sentry is enabled
[ https://issues.apache.org/jira/browse/HIVE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009024#comment-16009024 ] Yongzhi Chen commented on HIVE-16660: - The two failures are not related. They are known random failure tests. > Not able to add partition for views in hive when sentry is enabled > -- > > Key: HIVE-16660 > URL: https://issues.apache.org/jira/browse/HIVE-16660 > Project: Hive > Issue Type: Bug > Components: Parser >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-16660.1.patch > > > Repro: > create table tesnit (a int) partitioned by (p int); > insert into table tesnit partition (p = 1) values (1); > insert into table tesnit partition (p = 2) values (1); > create view test_view partitioned on (p) as select * from tesnit where p =1; > alter view test_view add partition (p = 2); > Error: Error while compiling statement: FAILED: SemanticException [Error > 10056]: The query does not reference any valid partition. To run this query, > set hive.mapred.mode=nonstrict (state=42000,code=10056) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16660) Not able to add partition for views in hive when sentry is enabled
[
https://issues.apache.org/jira/browse/HIVE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009003#comment-16009003
]
Hive QA commented on HIVE-16660:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867782/HIVE-16660.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10698 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5234/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5234/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5234/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867782 - PreCommit-HIVE-Build
> Not able to add partition for views in hive when sentry is enabled
> --
>
> Key: HIVE-16660
> URL: https://issues.apache.org/jira/browse/HIVE-16660
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-16660.1.patch
>
>
> Repro:
> create table tesnit (a int) partitioned by (p int);
> insert into table tesnit partition (p = 1) values (1);
> insert into table tesnit partition (p = 2) values (1);
> create view test_view partitioned on (p) as select * from tesnit where p =1;
> alter view test_view add partition (p = 2);
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10056]: The query does not reference any valid partition. To run this query,
> set hive.mapred.mode=nonstrict (state=42000,code=10056)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-1010) Implement INFORMATION_SCHEMA in Hive
[ https://issues.apache.org/jira/browse/HIVE-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-1010: - Attachment: HIVE-1010.16.patch .16 fixes test case and adds the comment to conf variable [~thejas] was asking for. > Implement INFORMATION_SCHEMA in Hive > > > Key: HIVE-1010 > URL: https://issues.apache.org/jira/browse/HIVE-1010 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Jeff Hammerbacher >Assignee: Gunther Hagleitner > Attachments: HIVE-1010.10.patch, HIVE-1010.11.patch, > HIVE-1010.12.patch, HIVE-1010.13.patch, HIVE-1010.14.patch, > HIVE-1010.15.patch, HIVE-1010.16.patch, HIVE-1010.7.patch, HIVE-1010.8.patch, > HIVE-1010.9.patch > > > INFORMATION_SCHEMA is part of the SQL92 standard and would be useful to > implement using our metastore. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16655) LLAP: Avoid preempting fragments before they enter the running state
[ https://issues.apache.org/jira/browse/HIVE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-16655: -- Status: Patch Available (was: Open) > LLAP: Avoid preempting fragments before they enter the running state > > > Key: HIVE-16655 > URL: https://issues.apache.org/jira/browse/HIVE-16655 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-16655.01.patch, HIVE-16655.02.patch > > > Currently in the AM, fragments may be preempted as soon as they are > allocated, without knowing whether they will move into the RUNNING state or > not. Leads to a lot of unnecessary kills. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16655) LLAP: Avoid preempting fragments before they enter the running state
[ https://issues.apache.org/jira/browse/HIVE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-16655: -- Attachment: HIVE-16655.02.patch Updated patch with the tez dependency changed to run ptests. Have already tested this on a cluster. > LLAP: Avoid preempting fragments before they enter the running state > > > Key: HIVE-16655 > URL: https://issues.apache.org/jira/browse/HIVE-16655 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-16655.01.patch, HIVE-16655.02.patch > > > Currently in the AM, fragments may be preempted as soon as they are > allocated, without knowing whether they will move into the RUNNING state or > not. Leads to a lot of unnecessary kills. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16652) LlapInputFormat: Seeing "output error" WARN message
[
https://issues.apache.org/jira/browse/HIVE-16652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-16652:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Committed to master
> LlapInputFormat: Seeing "output error" WARN message
> ---
>
> Key: HIVE-16652
> URL: https://issues.apache.org/jira/browse/HIVE-16652
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 3.0.0
>
> Attachments: HIVE-16652.1.patch
>
>
> Another warning message I'm seeing in the logs for TestJdbcWithMiniLlap after
> adding the line to close the RecordReader in the test:
> {noformat}
> 2017-05-11T11:08:34,511 WARN [IPC Server handler 0 on 54847] ipc.Server: IPC
> Server handler 0 on 54847, call Call#341 Retry#0 heartbeat({
> containerId=container_6830411502416918223_0003_00_00, requestId=2,
> startIndex=0, preRoutedStartIndex=0, maxEventsToGet=500,
> taskAttemptId=attempt_6830411502416918223_0003_0_00_00_0, eventCount=2
> }), rpc version=2, client version=1, methodsFingerPrint=996603002 from
> 10.22.8.180:54849: output error
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16602) Implement shared scans with Tez
[
https://issues.apache.org/jira/browse/HIVE-16602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008931#comment-16008931
]
Hive QA commented on HIVE-16602:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867783/HIVE-16602.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 10698 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5233/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5233/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5233/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867783 - PreCommit-HIVE-Build
> Implement shared scans with Tez
> ---
>
> Key: HIVE-16602
> URL: https://issues.apache.org/jira/browse/HIVE-16602
> Project: Hive
> Issue Type: New Feature
> Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-16602.01.patch, HIVE-16602.02.patch,
> HIVE-16602.03.patch, HIVE-16602.04.patch, HIVE-16602.patch
>
>
> Given a query plan, the goal is to identify scans on input tables that can be
> merged so the data is read only once. Optimization will be carried out at the
> physical level.
> In the longer term, identification of equivalent expressions and
> reutilization of intermediary results should be done at the logical layer via
> Spool operator.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16651) LlapProtocolClientProxy stack trace when using llap input format
[
https://issues.apache.org/jira/browse/HIVE-16651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-16651:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Committed to master
> LlapProtocolClientProxy stack trace when using llap input format
>
>
> Key: HIVE-16651
> URL: https://issues.apache.org/jira/browse/HIVE-16651
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 3.0.0
>
> Attachments: HIVE-16651.1.patch
>
>
> Seeing this after LlapBaseRecordReader.close():
> {noformat}
> 16/06/28 22:05:32 WARN LlapProtocolClientProxy: RequestManager shutdown with
> error
> java.util.concurrent.CancellationException
> at java.util.concurrent.FutureTask.report(FutureTask.java:121)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)
> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1170)
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:91)
> at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:384)
> at java.util.concurrent.FutureTask.cancel(FutureTask.java:180)
> at
> org.apache.hadoop.hive.llap.tez.LlapProtocolClientProxy.serviceStop(LlapProtocolClientProxy.java:131)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.hive.llap.ext.LlapTaskUmbilicalExternalClient.serviceStop(LlapTaskUmbilicalExternalClient.java:135)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.AbstractService.close(AbstractService.java:250)
> at
> org.apache.hadoop.hive.llap.LlapBaseRecordReader.close(LlapBaseRecordReader.java:84)
> at
> org.apache.hadoop.hive.llap.LlapRowRecordReader.close(LlapRowRecordReader.java:80)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-1010) Implement INFORMATION_SCHEMA in Hive
[ https://issues.apache.org/jira/browse/HIVE-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008864#comment-16008864 ] Thejas M Nair commented on HIVE-1010: - One thing to note is that the priv information made available through this patch is the sql standard authorization privileges. Sentry/Ranger stores it in separate dbs. We need some 'RangerStorageHandler' or 'JDBCStorageHandler' that accesses Ranger rdbms for that info. > Implement INFORMATION_SCHEMA in Hive > > > Key: HIVE-1010 > URL: https://issues.apache.org/jira/browse/HIVE-1010 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Jeff Hammerbacher >Assignee: Gunther Hagleitner > Attachments: HIVE-1010.10.patch, HIVE-1010.11.patch, > HIVE-1010.12.patch, HIVE-1010.13.patch, HIVE-1010.14.patch, > HIVE-1010.15.patch, HIVE-1010.7.patch, HIVE-1010.8.patch, HIVE-1010.9.patch > > > INFORMATION_SCHEMA is part of the SQL92 standard and would be useful to > implement using our metastore. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16113) PartitionPruner::removeNonPartCols needs to handle AND/OR cases
[
https://issues.apache.org/jira/browse/HIVE-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008863#comment-16008863
]
Hive QA commented on HIVE-16113:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867771/HIVE-16113.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10700 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_windowing2]
(batchId=10)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5232/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5232/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5232/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867771 - PreCommit-HIVE-Build
> PartitionPruner::removeNonPartCols needs to handle AND/OR cases
> ---
>
> Key: HIVE-16113
> URL: https://issues.apache.org/jira/browse/HIVE-16113
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Remus Rusanu
> Attachments: HIVE-16113.1.patch, HIVE-16113.2.patch
>
>
> {code}
> create table daysales (customer int) partitioned by (dt string);
> insert into daysales partition(dt='2001-01-01') values(1);
> select * from daysales where nvl(dt='2001-01-01' and customer=1, false);
> 0 ROWS
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L384
> {code}
> 2017-03-05T12:37:47,153 WARN [6f053d71-6ad6-4ad0-833d-337f2d499c82 main]
> ppr.PartitionPruner: The expr = NVL(((dt = '2001-01-01') and null),false)
> {code}
> Because {{true and null => null}}, this turns into {{NVL(null, false)}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16639) LLAP: Derive shuffle thread counts and keep-alive connections from instance count
[ https://issues.apache.org/jira/browse/HIVE-16639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-16639: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > LLAP: Derive shuffle thread counts and keep-alive connections from instance > count > - > > Key: HIVE-16639 > URL: https://issues.apache.org/jira/browse/HIVE-16639 > Project: Hive > Issue Type: Improvement >Reporter: Gopal V >Assignee: Siddharth Seth > Fix For: 3.0.0 > > Attachments: HIVE-16639.01.patch, HIVE-16639.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16639) LLAP: Derive shuffle thread counts and keep-alive connections from instance count
[ https://issues.apache.org/jira/browse/HIVE-16639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008855#comment-16008855 ] Siddharth Seth commented on HIVE-16639: --- Test failures are not related. Verified locally. Committing Thanks for the review [~gopalv] > LLAP: Derive shuffle thread counts and keep-alive connections from instance > count > - > > Key: HIVE-16639 > URL: https://issues.apache.org/jira/browse/HIVE-16639 > Project: Hive > Issue Type: Improvement >Reporter: Gopal V >Assignee: Siddharth Seth > Attachments: HIVE-16639.01.patch, HIVE-16639.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16639) LLAP: Derive shuffle thread counts and keep-alive connections from instance count
[ https://issues.apache.org/jira/browse/HIVE-16639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-16639: -- Attachment: HIVE-16639.02.patch Updated patch. > LLAP: Derive shuffle thread counts and keep-alive connections from instance > count > - > > Key: HIVE-16639 > URL: https://issues.apache.org/jira/browse/HIVE-16639 > Project: Hive > Issue Type: Improvement >Reporter: Gopal V >Assignee: Siddharth Seth > Attachments: HIVE-16639.01.patch, HIVE-16639.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16452) Database UUID for metastore DB
[ https://issues.apache.org/jira/browse/HIVE-16452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008800#comment-16008800 ] Lefty Leverenz commented on HIVE-16452: --- Ummm, I was hoping you'd figure out where to put the docs. ;) Perhaps we need a new wiki page for APIs. In the meantime, the APIs Overview might be the best place -- either the Metastore (Java) section or a new section. Another possibility is the Metastore Administration page. A new section could go after the list of supported databases, or it could be a subsection: * [Metastore Administration -- Supported Backend Databases | https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-SupportedBackendDatabasesforMetastore] Wherever it goes, let's have a crossreference from the other page(s). > Database UUID for metastore DB > -- > > Key: HIVE-16452 > URL: https://issues.apache.org/jira/browse/HIVE-16452 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > > In cloud environments it is possible that a same database instance is used as > the long running metadata persistence layer and multiple HMS access this > database. These HMS instances could be running the same time or in case of > transient workloads come up on an on-demand basis. HMS is used by multiple > projects in the Hadoop eco-system as the de-facto metadata keeper for various > SQL engines on the cluster. Currently, there is no way to uniquely identify > the database instance which is backing the HMS. For example, if there are two > instances of HMS running on top of same metastore DB, there is no way to > identify that data received from both the metastore clients is coming from > the same database. Similarly, if there in case of transient workloads > multiple HMS services come up and go, a external application which is > fetching data from a HMS has no way to identify that these multiple instances > of HMS are in fact returning the same data. > We can potentially use the combination of javax.jdo.option.ConnectionURL, > javax.jdo.option.ConnectionDriverName configuration of each HMS instance but > this is approach may not be very robust. If the database is migrated to > another server for some reason the ConnectionURL can change. Having a UUID in > the metastore DB which can be queried using a Thrift API can help solve this > problem. This way any application talking to multiple HMS instances can > recognize if the data is coming the same backing database. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16498) [Tez] ReduceRecordProcessor has no check to see if all the operators are done or not and is reading complete data
[
https://issues.apache.org/jira/browse/HIVE-16498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008784#comment-16008784
]
Hive QA commented on HIVE-16498:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867770/HIVE-16498-branch-1.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5231/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5231/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5231/
Messages:
{noformat}
This message was trimmed, see log for full details
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-7-openjdk-amd64/lib/ct.sym(META-INF/sym/rt.jar/java/util/Collection.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-7-openjdk-amd64/lib/ct.sym(META-INF/sym/rt.jar/java/util/Collections.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-7-openjdk-amd64/lib/ct.sym(META-INF/sym/rt.jar/java/util/Comparator.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-7-openjdk-amd64/lib/ct.sym(META-INF/sym/rt.jar/java/util/Iterator.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-7-openjdk-amd64/lib/ct.sym(META-INF/sym/rt.jar/java/util/List.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-7-openjdk-amd64/lib/ct.sym(META-INF/sym/rt.jar/java/util/Map.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-7-openjdk-amd64/lib/ct.sym(META-INF/sym/rt.jar/java/util/StringTokenizer.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar(org/apache/hadoop/conf/Configuration.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar(org/apache/hadoop/fs/Path.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar(org/apache/hadoop/util/StringUtils.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar(org/apache/hadoop/util/VersionInfo.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-7-openjdk-amd64/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Iterable.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar(org/apache/hadoop/io/Writable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-7-openjdk-amd64/lib/ct.sym(META-INF/sym/rt.jar/java/lang/String.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/aggregate/jetty-all-server/7.6.0.v20120127/jetty-all-server-7.6.0.v20120127.jar(org/eclipse/jetty/http/HttpStatus.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-7-openjdk-amd64/lib/ct.sym(META-INF/sym/rt.jar/java/util/HashMap.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-core/1.14/jersey-core-1.14.jar(javax/ws/rs/core/MediaType.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-core/1.14/jersey-core-1.14.jar(javax/ws/rs/core/Response.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-branch-1.2-source/ql/target/hive-exec-1.2.3-SNAPSHOT.jar(org/codehaus/jackson/map/ObjectMapper.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-7-openjdk-amd64/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Exception.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-7-openjdk-amd64/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Throwable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-7-openjdk-amd64/lib/ct.sym(META-INF/sym/rt.jar/java/io/Serializable.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-server/1.14/jersey-server-1.14.jar(com/sun/jersey/api/core/PackagesResourceConfig.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-servlet/1.14/jersey-servlet-1.14.jar(com/sun/jersey/spi/container/servlet/ServletContainer.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-branch-1.2-source/common/target/hive-common-1.2.3-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceStability.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-hdfs/2.6.0/hadoop-hdfs-2.6.0.jar(org/apache/hadoop/hdfs/web/AuthFilter.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar(org/apache/hadoop/security/UserGroupInformation.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/2.6.0/hadoop-auth-2.6.0.jar(org/apache/hadoop/security/authentication
[jira] [Commented] (HIVE-16625) Extra '\0' characters in the output, when SeparatedValuesOutputFormat is used and the quoting is disabled
[
https://issues.apache.org/jira/browse/HIVE-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008777#comment-16008777
]
Hive QA commented on HIVE-16625:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867764/HIVE-16625.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10699 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5230/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5230/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5230/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867764 - PreCommit-HIVE-Build
> Extra '\0' characters in the output, when SeparatedValuesOutputFormat is used
> and the quoting is disabled
> -
>
> Key: HIVE-16625
> URL: https://issues.apache.org/jira/browse/HIVE-16625
> Project: Hive
> Issue Type: Bug
> Components: Beeline, Testing Infrastructure
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16625.02.patch, HIVE-16625.03.patch,
> HIVE-16625.patch
>
>
> If the output format is using {{SeparatedValuesOutputFormat}}, and the
> quoting is disabled (by default is disabled), and the value of the cell
> contains the separator character, then the the output is "quoted" with '\0'
> characters.
> To reproduce:
> {code}
> create table quotes(s string);
> insert into quotes values('a\ta');
> !set outputFormat tsv2
> select * from quotes;
> {code}
> The result is:
> {code}
> quotes.s
> ^@a a^@
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16452) Database UUID for metastore DB
[ https://issues.apache.org/jira/browse/HIVE-16452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008765#comment-16008765 ] Vihang Karajgaonkar commented on HIVE-16452: Hi [~leftylev] I looked at https://cwiki.apache.org/confluence/display/Hive/Hive+APIs+Overview but it doesn't look like the right page since it doesn't have information of the other APIs we have already. Do you know which page should I update to document the new API? > Database UUID for metastore DB > -- > > Key: HIVE-16452 > URL: https://issues.apache.org/jira/browse/HIVE-16452 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > > In cloud environments it is possible that a same database instance is used as > the long running metadata persistence layer and multiple HMS access this > database. These HMS instances could be running the same time or in case of > transient workloads come up on an on-demand basis. HMS is used by multiple > projects in the Hadoop eco-system as the de-facto metadata keeper for various > SQL engines on the cluster. Currently, there is no way to uniquely identify > the database instance which is backing the HMS. For example, if there are two > instances of HMS running on top of same metastore DB, there is no way to > identify that data received from both the metastore clients is coming from > the same database. Similarly, if there in case of transient workloads > multiple HMS services come up and go, a external application which is > fetching data from a HMS has no way to identify that these multiple instances > of HMS are in fact returning the same data. > We can potentially use the combination of javax.jdo.option.ConnectionURL, > javax.jdo.option.ConnectionDriverName configuration of each HMS instance but > this is approach may not be very robust. If the database is migrated to > another server for some reason the ConnectionURL can change. Having a UUID in > the metastore DB which can be queried using a Thrift API can help solve this > problem. This way any application talking to multiple HMS instances can > recognize if the data is coming the same backing database. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16634) LLAP Use a pool of connections to a single AM from a daemon
[ https://issues.apache.org/jira/browse/HIVE-16634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-16634: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > LLAP Use a pool of connections to a single AM from a daemon > --- > > Key: HIVE-16634 > URL: https://issues.apache.org/jira/browse/HIVE-16634 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Siddharth Seth > Fix For: 3.0.0 > > Attachments: HIVE-16634.01.patch, HIVE-16634.02.patch, > locked-threads-ipc.png > > > !locked-threads-ipc.png! -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-13583) E061-14: Search Conditions
[ https://issues.apache.org/jira/browse/HIVE-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008743#comment-16008743 ] Lefty Leverenz commented on HIVE-13583: --- Thanks [~cartershanklin]. Should these predicates be documented in the same sections as IS NULL and IS NOT NULL? * [Operators and UDFs -- Operators Precedences | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-OperatorsprecedencesOperatorsPrecedencesOperatorsPrecedences] * [Operators and UDFs -- Relational Operators | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-RelationalOperators] The conditional functions belong here: * [Operators and UDFs -- Conditional Functions | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-ConditionalFunctions] Any new keywords belong here, in the 3.0.0 row: * [DDL -- Keywords | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Keywords,Non-reservedKeywordsandReservedKeywords] I added a TODOC3.0 label. > E061-14: Search Conditions > -- > > Key: HIVE-13583 > URL: https://issues.apache.org/jira/browse/HIVE-13583 > Project: Hive > Issue Type: Sub-task >Reporter: Carter Shanklin >Assignee: Zoltan Haindrich > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-13583.1.patch, HIVE-13583.2.patch > > > This is a part of the SQL:2011 Analytics Complete Umbrella JIRA HIVE-13554. > Support for various forms of search conditions are mandatory in the SQL > standard. For example, " is not true;" Hive should support those > forms mandated by the standard. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16663) String Caching For Rows
[ https://issues.apache.org/jira/browse/HIVE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-16663: --- Attachment: HIVE-16663.4.patch > String Caching For Rows > --- > > Key: HIVE-16663 > URL: https://issues.apache.org/jira/browse/HIVE-16663 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.0.1 >Reporter: BELUGA BEHR >Priority: Minor > Attachments: HIVE-16663.1.patch, HIVE-16663.2.patch, > HIVE-16663.3.patch, HIVE-16663.4.patch > > > It is very common that there are many repeated values in the result set of a > query. As it currently stands, beeline does not attempt to cache any of > these values and therefore it consumes a lot of memory. > Adding a string cache may save a lot of memory. There are organizations that > use beeline to perform ETL processing of result sets into CSV. This will > better support those organizations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16143) Improve msck repair batching
[
https://issues.apache.org/jira/browse/HIVE-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-16143:
---
Attachment: HIVE-16143.09.patch
Addressed review comments
> Improve msck repair batching
>
>
> Key: HIVE-16143
> URL: https://issues.apache.org/jira/browse/HIVE-16143
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16143.01.patch, HIVE-16143.02.patch,
> HIVE-16143.03.patch, HIVE-16143.04.patch, HIVE-16143.05.patch,
> HIVE-16143.06.patch, HIVE-16143.07.patch, HIVE-16143.08.patch,
> HIVE-16143.09.patch
>
>
> Currently, the {{msck repair table}} command batches the number of partitions
> created in the metastore using the config {{HIVE_MSCK_REPAIR_BATCH_SIZE}}.
> Following snippet shows the batching logic. There can be couple of
> improvements to this batching logic:
> {noformat}
> int batch_size = conf.getIntVar(ConfVars.HIVE_MSCK_REPAIR_BATCH_SIZE);
> if (batch_size > 0 && partsNotInMs.size() > batch_size) {
> int counter = 0;
> for (CheckResult.PartitionResult part : partsNotInMs) {
> counter++;
>
> apd.addPartition(Warehouse.makeSpecFromName(part.getPartitionName()), null);
> repairOutput.add("Repair: Added partition to metastore " +
> msckDesc.getTableName()
> + ':' + part.getPartitionName());
> if (counter % batch_size == 0 || counter ==
> partsNotInMs.size()) {
> db.createPartitions(apd);
> apd = new AddPartitionDesc(table.getDbName(),
> table.getTableName(), false);
> }
> }
> } else {
> for (CheckResult.PartitionResult part : partsNotInMs) {
>
> apd.addPartition(Warehouse.makeSpecFromName(part.getPartitionName()), null);
> repairOutput.add("Repair: Added partition to metastore " +
> msckDesc.getTableName()
> + ':' + part.getPartitionName());
> }
> db.createPartitions(apd);
> }
> } catch (Exception e) {
> LOG.info("Could not bulk-add partitions to metastore; trying one by
> one", e);
> repairOutput.clear();
> msckAddPartitionsOneByOne(db, table, partsNotInMs, repairOutput);
> }
> {noformat}
> 1. If the batch size is too aggressive the code falls back to adding
> partitions one by one which is almost always very slow. It is easily possible
> that users increase the batch size to higher value to make the command run
> faster but end up with a worse performance because code falls back to adding
> one by one. Users are then expected to determine the tuned value of batch
> size which works well for their environment. I think the code could handle
> this situation better by exponentially decaying the batch size instead of
> falling back to one by one.
> 2. The other issue with this implementation is if lets say first batch
> succeeds and the second one fails, the code tries to add all the partitions
> one by one irrespective of whether some of the were successfully added or
> not. If we need to fall back to one by one we should atleast remove the ones
> which we know for sure are already added successfully.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16634) LLAP Use a pool of connections to a single AM from a daemon
[ https://issues.apache.org/jira/browse/HIVE-16634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008715#comment-16008715 ] Siddharth Seth commented on HIVE-16634: --- Committing the .1 patch. I don't think the .2 patch test failures are related, but I don't like the patch too much. Will create a new jira to fix the pool size properly. Thanks for the review [~sershe] > LLAP Use a pool of connections to a single AM from a daemon > --- > > Key: HIVE-16634 > URL: https://issues.apache.org/jira/browse/HIVE-16634 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Rajesh Balamohan >Assignee: Siddharth Seth > Attachments: HIVE-16634.01.patch, HIVE-16634.02.patch, > locked-threads-ipc.png > > > !locked-threads-ipc.png! -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HIVE-16639) LLAP: Derive shuffle thread counts and keep-alive connections from instance count
[ https://issues.apache.org/jira/browse/HIVE-16639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008596#comment-16008596 ] Gopal V edited comment on HIVE-16639 at 5/12/17 9:27 PM: - [~sseth]: this is very pessimistic when running with 1 instance (i.e lowers below even the default), I think this needs to be Max(instances, executors)+1 was (Author: gopalv): [~sseth]: this is very pessimistic when running with 1 instance (i.e lowers below even the default), I think this needs to be min(instances, executors)+1 > LLAP: Derive shuffle thread counts and keep-alive connections from instance > count > - > > Key: HIVE-16639 > URL: https://issues.apache.org/jira/browse/HIVE-16639 > Project: Hive > Issue Type: Improvement >Reporter: Gopal V >Assignee: Siddharth Seth > Attachments: HIVE-16639.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13583) E061-14: Search Conditions
[ https://issues.apache.org/jira/browse/HIVE-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-13583: -- Labels: TODOC3.0 (was: ) > E061-14: Search Conditions > -- > > Key: HIVE-13583 > URL: https://issues.apache.org/jira/browse/HIVE-13583 > Project: Hive > Issue Type: Sub-task >Reporter: Carter Shanklin >Assignee: Zoltan Haindrich > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-13583.1.patch, HIVE-13583.2.patch > > > This is a part of the SQL:2011 Analytics Complete Umbrella JIRA HIVE-13554. > Support for various forms of search conditions are mandatory in the SQL > standard. For example, " is not true;" Hive should support those > forms mandated by the standard. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16652) LlapInputFormat: Seeing "output error" WARN message
[
https://issues.apache.org/jira/browse/HIVE-16652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008711#comment-16008711
]
Siddharth Seth commented on HIVE-16652:
---
+1. Looks good.
> LlapInputFormat: Seeing "output error" WARN message
> ---
>
> Key: HIVE-16652
> URL: https://issues.apache.org/jira/browse/HIVE-16652
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-16652.1.patch
>
>
> Another warning message I'm seeing in the logs for TestJdbcWithMiniLlap after
> adding the line to close the RecordReader in the test:
> {noformat}
> 2017-05-11T11:08:34,511 WARN [IPC Server handler 0 on 54847] ipc.Server: IPC
> Server handler 0 on 54847, call Call#341 Retry#0 heartbeat({
> containerId=container_6830411502416918223_0003_00_00, requestId=2,
> startIndex=0, preRoutedStartIndex=0, maxEventsToGet=500,
> taskAttemptId=attempt_6830411502416918223_0003_0_00_00_0, eventCount=2
> }), rpc version=2, client version=1, methodsFingerPrint=996603002 from
> 10.22.8.180:54849: output error
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16452) Database UUID for metastore DB
[ https://issues.apache.org/jira/browse/HIVE-16452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008706#comment-16008706 ] Vihang Karajgaonkar commented on HIVE-16452: Yes, I will update the wiki. It adds a new Thrift API for the metastore. > Database UUID for metastore DB > -- > > Key: HIVE-16452 > URL: https://issues.apache.org/jira/browse/HIVE-16452 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > > In cloud environments it is possible that a same database instance is used as > the long running metadata persistence layer and multiple HMS access this > database. These HMS instances could be running the same time or in case of > transient workloads come up on an on-demand basis. HMS is used by multiple > projects in the Hadoop eco-system as the de-facto metadata keeper for various > SQL engines on the cluster. Currently, there is no way to uniquely identify > the database instance which is backing the HMS. For example, if there are two > instances of HMS running on top of same metastore DB, there is no way to > identify that data received from both the metastore clients is coming from > the same database. Similarly, if there in case of transient workloads > multiple HMS services come up and go, a external application which is > fetching data from a HMS has no way to identify that these multiple instances > of HMS are in fact returning the same data. > We can potentially use the combination of javax.jdo.option.ConnectionURL, > javax.jdo.option.ConnectionDriverName configuration of each HMS instance but > this is approach may not be very robust. If the database is migrated to > another server for some reason the ConnectionURL can change. Having a UUID in > the metastore DB which can be queried using a Thrift API can help solve this > problem. This way any application talking to multiple HMS instances can > recognize if the data is coming the same backing database. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16452) Database UUID for metastore DB
[ https://issues.apache.org/jira/browse/HIVE-16452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008703#comment-16008703 ] Lefty Leverenz commented on HIVE-16452: --- Does this need to be documented in the wiki? > Database UUID for metastore DB > -- > > Key: HIVE-16452 > URL: https://issues.apache.org/jira/browse/HIVE-16452 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > > In cloud environments it is possible that a same database instance is used as > the long running metadata persistence layer and multiple HMS access this > database. These HMS instances could be running the same time or in case of > transient workloads come up on an on-demand basis. HMS is used by multiple > projects in the Hadoop eco-system as the de-facto metadata keeper for various > SQL engines on the cluster. Currently, there is no way to uniquely identify > the database instance which is backing the HMS. For example, if there are two > instances of HMS running on top of same metastore DB, there is no way to > identify that data received from both the metastore clients is coming from > the same database. Similarly, if there in case of transient workloads > multiple HMS services come up and go, a external application which is > fetching data from a HMS has no way to identify that these multiple instances > of HMS are in fact returning the same data. > We can potentially use the combination of javax.jdo.option.ConnectionURL, > javax.jdo.option.ConnectionDriverName configuration of each HMS instance but > this is approach may not be very robust. If the database is migrated to > another server for some reason the ConnectionURL can change. Having a UUID in > the metastore DB which can be queried using a Thrift API can help solve this > problem. This way any application talking to multiple HMS instances can > recognize if the data is coming the same backing database. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16651) LlapProtocolClientProxy stack trace when using llap input format
[
https://issues.apache.org/jira/browse/HIVE-16651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008700#comment-16008700
]
Siddharth Seth commented on HIVE-16651:
---
+1
> LlapProtocolClientProxy stack trace when using llap input format
>
>
> Key: HIVE-16651
> URL: https://issues.apache.org/jira/browse/HIVE-16651
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-16651.1.patch
>
>
> Seeing this after LlapBaseRecordReader.close():
> {noformat}
> 16/06/28 22:05:32 WARN LlapProtocolClientProxy: RequestManager shutdown with
> error
> java.util.concurrent.CancellationException
> at java.util.concurrent.FutureTask.report(FutureTask.java:121)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)
> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1170)
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:91)
> at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:384)
> at java.util.concurrent.FutureTask.cancel(FutureTask.java:180)
> at
> org.apache.hadoop.hive.llap.tez.LlapProtocolClientProxy.serviceStop(LlapProtocolClientProxy.java:131)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.hive.llap.ext.LlapTaskUmbilicalExternalClient.serviceStop(LlapTaskUmbilicalExternalClient.java:135)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.AbstractService.close(AbstractService.java:250)
> at
> org.apache.hadoop.hive.llap.LlapBaseRecordReader.close(LlapBaseRecordReader.java:84)
> at
> org.apache.hadoop.hive.llap.LlapRowRecordReader.close(LlapRowRecordReader.java:80)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16663) String Caching For Rows
[ https://issues.apache.org/jira/browse/HIVE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-16663: --- Attachment: HIVE-16663.3.patch Changed to use Java string table instead of explicit cache > String Caching For Rows > --- > > Key: HIVE-16663 > URL: https://issues.apache.org/jira/browse/HIVE-16663 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.0.1 >Reporter: BELUGA BEHR >Priority: Minor > Attachments: HIVE-16663.1.patch, HIVE-16663.2.patch, > HIVE-16663.3.patch > > > It is very common that there are many repeated values in the result set of a > query. As it currently stands, beeline does not attempt to cache any of > these values and therefore it consumes a lot of memory. > Adding a string cache may save a lot of memory. There are organizations that > use beeline to perform ETL processing of result sets into CSV. This will > better support those organizations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16664) Add join related Hive blobstore tests
[ https://issues.apache.org/jira/browse/HIVE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rentao Wu updated HIVE-16664: - Status: Patch Available (was: Open) > Add join related Hive blobstore tests > - > > Key: HIVE-16664 > URL: https://issues.apache.org/jira/browse/HIVE-16664 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Rentao Wu >Assignee: Rentao Wu > Fix For: 2.2.0, 2.3.0, 3.0.0, 2.4.0 > > Attachments: HIVE-16664.patch > > > This patch introduces the following regression tests into the hive-blobstore > qtest module: > * join.q -> Test inner join query > * join2.q-> Test left and right outer join > * map_join.q -> Test mapjoin hint > * map_join_on_filter.q -> Test mapjoin hint with filter > * nested_outer_join.q -> Test nested outer join queries -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16664) Add join related Hive blobstore tests
[ https://issues.apache.org/jira/browse/HIVE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rentao Wu updated HIVE-16664: - Attachment: HIVE-16664.patch > Add join related Hive blobstore tests > - > > Key: HIVE-16664 > URL: https://issues.apache.org/jira/browse/HIVE-16664 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Rentao Wu >Assignee: Rentao Wu > Fix For: 2.2.0, 2.3.0, 3.0.0, 2.4.0 > > Attachments: HIVE-16664.patch > > > This patch introduces the following regression tests into the hive-blobstore > qtest module: > * join.q -> Test inner join query > * join2.q-> Test left and right outer join > * map_join.q -> Test mapjoin hint > * map_join_on_filter.q -> Test mapjoin hint with filter > * nested_outer_join.q -> Test nested outer join queries -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16664) Add join related Hive blobstore tests
[ https://issues.apache.org/jira/browse/HIVE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008695#comment-16008695 ] Rentao Wu commented on HIVE-16664: -- Attached patch and reviewboard CR > Add join related Hive blobstore tests > - > > Key: HIVE-16664 > URL: https://issues.apache.org/jira/browse/HIVE-16664 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Rentao Wu >Assignee: Rentao Wu > Fix For: 2.2.0, 2.3.0, 3.0.0, 2.4.0 > > Attachments: HIVE-16664.patch > > > This patch introduces the following regression tests into the hive-blobstore > qtest module: > * join.q -> Test inner join query > * join2.q-> Test left and right outer join > * map_join.q -> Test mapjoin hint > * map_join_on_filter.q -> Test mapjoin hint with filter > * nested_outer_join.q -> Test nested outer join queries -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16645) Commands.java has missed the catch statement and has some code format errors
[
https://issues.apache.org/jira/browse/HIVE-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008690#comment-16008690
]
Hive QA commented on HIVE-16645:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867743/HIVE-16645.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10698 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5229/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5229/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5229/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867743 - PreCommit-HIVE-Build
> Commands.java has missed the catch statement and has some code format errors
>
>
> Key: HIVE-16645
> URL: https://issues.apache.org/jira/browse/HIVE-16645
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Saijin Huang
>Assignee: Saijin Huang
>Priority: Minor
> Attachments: HIVE-16645.1.patch, HIVE-16645.2.patch
>
>
> In commands.java, the catch statement is missing and the Resultset statement
> is not closed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16665) Race condition in Utilities.GetInputPathsCallable --> createDummyFileForEmptyPartition
[
https://issues.apache.org/jira/browse/HIVE-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar reassigned HIVE-16665:
---
> Race condition in Utilities.GetInputPathsCallable -->
> createDummyFileForEmptyPartition
> --
>
> Key: HIVE-16665
> URL: https://issues.apache.org/jira/browse/HIVE-16665
> Project: Hive
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> Looks like there is a race condition in the {{GetInputPathsCallable}} thread
> when modifying the input {{MapWork}} object.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16628) Fix query25 when it uses a mix of MergeJoin and MapJoin
[ https://issues.apache.org/jira/browse/HIVE-16628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-16628: Resolution: Fixed Status: Resolved (was: Patch Available) > Fix query25 when it uses a mix of MergeJoin and MapJoin > --- > > Key: HIVE-16628 > URL: https://issues.apache.org/jira/browse/HIVE-16628 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.0.0, 2.1.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 3.0.0 > > Attachments: HIVE-16628.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16664) Add join related Hive blobstore tests
[ https://issues.apache.org/jira/browse/HIVE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rentao Wu updated HIVE-16664: - Description: This patch introduces the following regression tests into the hive-blobstore qtest module: * join.q -> Test inner join query * join2.q-> Test left and right outer join * map_join.q -> Test mapjoin hint * map_join_on_filter.q -> Test mapjoin hint with filter * nested_outer_join.q -> Test nested outer join queries was: This patch introduces the following regression tests into the hive-blobstore qtest module: join.q -> Test inner join query · join2.q -> Test left and right outer join · map_join.q-> Test mapjoin hint · map_join_on_filter.q -> Test mapjoin hint with filter · nested_outer_join.q -> Test nested outer join queries > Add join related Hive blobstore tests > - > > Key: HIVE-16664 > URL: https://issues.apache.org/jira/browse/HIVE-16664 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Rentao Wu >Assignee: Rentao Wu > Fix For: 2.2.0, 2.3.0, 3.0.0, 2.4.0 > > > This patch introduces the following regression tests into the hive-blobstore > qtest module: > * join.q -> Test inner join query > * join2.q-> Test left and right outer join > * map_join.q -> Test mapjoin hint > * map_join_on_filter.q -> Test mapjoin hint with filter > * nested_outer_join.q -> Test nested outer join queries -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16664) Add join related Hive blobstore tests
[ https://issues.apache.org/jira/browse/HIVE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rentao Wu updated HIVE-16664: - Description: This patch introduces the following regression tests into the hive-blobstore qtest module: join.q -> Test inner join query · join2.q -> Test left and right outer join · map_join.q-> Test mapjoin hint · map_join_on_filter.q -> Test mapjoin hint with filter · nested_outer_join.q -> Test nested outer join queries was: This patch introduces the following regression tests into the hive-blobstore qtest module: · join.q -> Test inner join query · join2.q -> Test left and right outer join · map_join.q-> Test mapjoin hint · map_join_on_filter.q -> Test mapjoin hint with filter · nested_outer_join.q -> Test nested outer join queries > Add join related Hive blobstore tests > - > > Key: HIVE-16664 > URL: https://issues.apache.org/jira/browse/HIVE-16664 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Rentao Wu >Assignee: Rentao Wu > Fix For: 2.2.0, 2.3.0, 3.0.0, 2.4.0 > > > This patch introduces the following regression tests into the hive-blobstore > qtest module: > join.q -> Test inner join query > · join2.q -> Test left and right outer > join > · map_join.q-> Test mapjoin hint > · map_join_on_filter.q -> Test mapjoin hint with filter > · nested_outer_join.q -> Test nested outer join queries -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16664) Add join related Hive blobstore tests
[ https://issues.apache.org/jira/browse/HIVE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rentao Wu reassigned HIVE-16664: Assignee: Rentao Wu > Add join related Hive blobstore tests > - > > Key: HIVE-16664 > URL: https://issues.apache.org/jira/browse/HIVE-16664 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Rentao Wu >Assignee: Rentao Wu > Fix For: 2.2.0, 2.3.0, 3.0.0, 2.4.0 > > > This patch introduces the following regression tests into the hive-blobstore > qtest module: > · join.q -> Test inner join query > · join2.q -> Test left and right outer > join > · map_join.q-> Test mapjoin hint > · map_join_on_filter.q -> Test mapjoin hint with filter > · nested_outer_join.q -> Test nested outer join queries -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16663) String Caching For Rows
[ https://issues.apache.org/jira/browse/HIVE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008655#comment-16008655 ] BELUGA BEHR commented on HIVE-16663: Understanding that one can also set "incremental" for beeline. This patch is to assist in alleviating the following: Setting --incremental=true is recommended if you encounter an OutOfMemory on the client side (due to the fetched result set size being large). > String Caching For Rows > --- > > Key: HIVE-16663 > URL: https://issues.apache.org/jira/browse/HIVE-16663 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.0.1 >Reporter: BELUGA BEHR >Priority: Minor > Attachments: HIVE-16663.1.patch, HIVE-16663.2.patch > > > It is very common that there are many repeated values in the result set of a > query. As it currently stands, beeline does not attempt to cache any of > these values and therefore it consumes a lot of memory. > Adding a string cache may save a lot of memory. There are organizations that > use beeline to perform ETL processing of result sets into CSV. This will > better support those organizations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16663) String Caching For Rows
[ https://issues.apache.org/jira/browse/HIVE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-16663: --- Attachment: HIVE-16663.2.patch Remove superfluous cast > String Caching For Rows > --- > > Key: HIVE-16663 > URL: https://issues.apache.org/jira/browse/HIVE-16663 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.0.1 >Reporter: BELUGA BEHR >Priority: Minor > Attachments: HIVE-16663.1.patch, HIVE-16663.2.patch > > > It is very common that there are many repeated values in the result set of a > query. As it currently stands, beeline does not attempt to cache any of > these values and therefore it consumes a lot of memory. > Adding a string cache may save a lot of memory. There are organizations that > use beeline to perform ETL processing of result sets into CSV. This will > better support those organizations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16663) String Caching For Rows
[ https://issues.apache.org/jira/browse/HIVE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-16663: --- Attachment: HIVE-16663.1.patch > String Caching For Rows > --- > > Key: HIVE-16663 > URL: https://issues.apache.org/jira/browse/HIVE-16663 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.0.1 >Reporter: BELUGA BEHR >Priority: Minor > Attachments: HIVE-16663.1.patch > > > It is very common that there are many repeated values in the result set of a > query. As it currently stands, beeline does not attempt to cache any of > these values and therefore it consumes a lot of memory. > Adding a string cache may save a lot of memory. There are organizations that > use beeline to perform ETL processing of result sets into CSV. This will > better support those organizations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16663) String Caching For Rows
[ https://issues.apache.org/jira/browse/HIVE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-16663: --- Status: Patch Available (was: Open) > String Caching For Rows > --- > > Key: HIVE-16663 > URL: https://issues.apache.org/jira/browse/HIVE-16663 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.0.1 >Reporter: BELUGA BEHR >Priority: Minor > Attachments: HIVE-16663.1.patch > > > It is very common that there are many repeated values in the result set of a > query. As it currently stands, beeline does not attempt to cache any of > these values and therefore it consumes a lot of memory. > Adding a string cache may save a lot of memory. There are organizations that > use beeline to perform ETL processing of result sets into CSV. This will > better support those organizations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16663) String Caching For Rows
[ https://issues.apache.org/jira/browse/HIVE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-16663: --- Flags: Patch > String Caching For Rows > --- > > Key: HIVE-16663 > URL: https://issues.apache.org/jira/browse/HIVE-16663 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.0.1 >Reporter: BELUGA BEHR >Priority: Minor > > It is very common that there are many repeated values in the result set of a > query. As it currently stands, beeline does not attempt to cache any of > these values and therefore it consumes a lot of memory. > Adding a string cache may save a lot of memory. There are organizations that > use beeline to perform ETL processing of result sets into CSV. This will > better support those organizations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HIVE-16452) Database UUID for metastore DB
[ https://issues.apache.org/jira/browse/HIVE-16452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar resolved HIVE-16452. Resolution: Fixed Target Version/s: 3.0.0 Resolving this as both the sub-tasks are merged. > Database UUID for metastore DB > -- > > Key: HIVE-16452 > URL: https://issues.apache.org/jira/browse/HIVE-16452 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > > In cloud environments it is possible that a same database instance is used as > the long running metadata persistence layer and multiple HMS access this > database. These HMS instances could be running the same time or in case of > transient workloads come up on an on-demand basis. HMS is used by multiple > projects in the Hadoop eco-system as the de-facto metadata keeper for various > SQL engines on the cluster. Currently, there is no way to uniquely identify > the database instance which is backing the HMS. For example, if there are two > instances of HMS running on top of same metastore DB, there is no way to > identify that data received from both the metastore clients is coming from > the same database. Similarly, if there in case of transient workloads > multiple HMS services come up and go, a external application which is > fetching data from a HMS has no way to identify that these multiple instances > of HMS are in fact returning the same data. > We can potentially use the combination of javax.jdo.option.ConnectionURL, > javax.jdo.option.ConnectionDriverName configuration of each HMS instance but > this is approach may not be very robust. If the database is migrated to > another server for some reason the ConnectionURL can change. Having a UUID in > the metastore DB which can be queried using a Thrift API can help solve this > problem. This way any application talking to multiple HMS instances can > recognize if the data is coming the same backing database. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16272) support for drop function in incremental replication
[
https://issues.apache.org/jira/browse/HIVE-16272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008638#comment-16008638
]
Hive QA commented on HIVE-16272:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867741/HIVE-16272.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10699 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[named_column_join]
(batchId=72)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5228/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5228/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5228/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867741 - PreCommit-HIVE-Build
> support for drop function in incremental replication
> -
>
> Key: HIVE-16272
> URL: https://issues.apache.org/jira/browse/HIVE-16272
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
> Attachments: HIVE-16272.1.patch, HIVE-16272.2.patch
>
>
> drop function should work in incremental dump and incremental load
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16639) LLAP: Derive shuffle thread counts and keep-alive connections from instance count
[ https://issues.apache.org/jira/browse/HIVE-16639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008596#comment-16008596 ] Gopal V commented on HIVE-16639: [~sseth]: this is very pessimistic when running with 1 instance (i.e lowers below even the default), I think this needs to be min(instances, executors)+1 > LLAP: Derive shuffle thread counts and keep-alive connections from instance > count > - > > Key: HIVE-16639 > URL: https://issues.apache.org/jira/browse/HIVE-16639 > Project: Hive > Issue Type: Improvement >Reporter: Gopal V >Assignee: Siddharth Seth > Attachments: HIVE-16639.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14389) Beeline should not output query and prompt to stdout
[
https://issues.apache.org/jira/browse/HIVE-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-14389:
Resolution: Fixed
Fix Version/s: 2.4.0
3.0.0
Status: Resolved (was: Patch Available)
Push to master and branch-2, thanks [~vihangk1] for your contribution.
> Beeline should not output query and prompt to stdout
>
>
> Key: HIVE-14389
> URL: https://issues.apache.org/jira/browse/HIVE-14389
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-14389.01.patch, HIVE-14389.02.patch,
> HIVE-14389.03.patch, HIVE-14389.04.patch
>
>
> It seems that the Beeline prints the query along with the results in the
> stdout when a script file is passed. The output file in the example below
> needs to only have the results and not the query.
> {noformat}
> .vihang-MBP:bin vihang$ ./beeline --showheader=false --outformat=tsv2 -u
> "jdbc:hive2://localhost:1" -f /tmp/query.sql > /tmp/query.out 2>
> /tmp/query.err
> OK
> $ cat /tmp/query.out
> 1: jdbc:hive2://localhost:1/default> select * from likes limit 4;
> +---+--+--+
> | 1 | chocolate|
> | 1 | car |
> | 1 | games|
> | 1 | chess|
> +---+--+--+
> 1: jdbc:hive2://localhost:1/default>
> 1: jdbc:hive2://localhost:1/default>
> $
> {noformat}
> A lot of people use HiveCLI and in order to transition from HiveCLI scripts
> to Beeline, this needs to be taken care of. The output files generated by
> beeline should contain only the results and nothing else.
> Similarly, when not in silent mode, query are being printed out on stdout,
> which is adding garbage along with results, as just like HIVE CLI does, users
> would like to have only the results on stdout, not errors/debugging info/etc,
> like the full query.
> Query could be printed out, no problem, as long as it is not on stdout (with
> results), instead, it must be printed out along with the debugging info.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16571) HiveServer2: Prefer LIFO over round-robin for Tez session reuse
[
https://issues.apache.org/jira/browse/HIVE-16571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008570#comment-16008570
]
Hive QA commented on HIVE-16571:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867730/HIVE-16571.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10698 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
(batchId=225)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5227/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5227/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5227/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867730 - PreCommit-HIVE-Build
> HiveServer2: Prefer LIFO over round-robin for Tez session reuse
> ---
>
> Key: HIVE-16571
> URL: https://issues.apache.org/jira/browse/HIVE-16571
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Tez
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-16571.2.patch, HIVE-16571.patch
>
>
> Currently Tez session reuse is entirely round-robin, which means a single
> user might have to run upto 32 queries before reusing a warm session on a
> HiveServer2.
> This is not the case when session reuse is disabled, with a user warming up
> their session on the 1st query.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16662) Fix remaining unit test failures when CachedStore is enabled
[ https://issues.apache.org/jira/browse/HIVE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16662: -- Issue Type: Sub-task (was: Bug) Parent: HIVE-16520 > Fix remaining unit test failures when CachedStore is enabled > > > Key: HIVE-16662 > URL: https://issues.apache.org/jira/browse/HIVE-16662 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > > In HIVE-16586, I fixed most of UT failures for CachedStore. This ticket is > for the remainings, and regressions when stats methods in CachedStore are > enabled. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16586) Fix Unit test failures when CachedStore is enabled
[ https://issues.apache.org/jira/browse/HIVE-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16586: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Patch pushed to master. Created HIVE-16662 for the remaining failures. > Fix Unit test failures when CachedStore is enabled > -- > > Key: HIVE-16586 > URL: https://issues.apache.org/jira/browse/HIVE-16586 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 3.0.0 > > Attachments: enable_cachedstore.patch, HIVE-16586.1.patch, > HIVE-16586.2.patch, HIVE-16586.3.patch > > > Though we don't plan to turn on CachedStore by default, we want to make sure > unit tests pass with CachedStore. I turn on CachedStore in the patch in order > to run unit tests with it, but I will turn off CachedStore when committing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16662) Fix remaining unit test failures when CachedStore is enabled
[ https://issues.apache.org/jira/browse/HIVE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai reassigned HIVE-16662: - > Fix remaining unit test failures when CachedStore is enabled > > > Key: HIVE-16662 > URL: https://issues.apache.org/jira/browse/HIVE-16662 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > > In HIVE-16586, I fixed most of UT failures for CachedStore. This ticket is > for the remainings, and regressions when stats methods in CachedStore are > enabled. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16586) Fix Unit test failures when CachedStore is enabled
[ https://issues.apache.org/jira/browse/HIVE-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008534#comment-16008534 ] Thejas M Nair commented on HIVE-16586: -- +1 > Fix Unit test failures when CachedStore is enabled > -- > > Key: HIVE-16586 > URL: https://issues.apache.org/jira/browse/HIVE-16586 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: enable_cachedstore.patch, HIVE-16586.1.patch, > HIVE-16586.2.patch, HIVE-16586.3.patch > > > Though we don't plan to turn on CachedStore by default, we want to make sure > unit tests pass with CachedStore. I turn on CachedStore in the patch in order > to run unit tests with it, but I will turn off CachedStore when committing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16586) Fix Unit test failures when CachedStore is enabled
[ https://issues.apache.org/jira/browse/HIVE-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-16586: -- Attachment: HIVE-16586.3.patch Attach a patch ready for commit (revert the enablement for CachedStore and stats changes). > Fix Unit test failures when CachedStore is enabled > -- > > Key: HIVE-16586 > URL: https://issues.apache.org/jira/browse/HIVE-16586 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: enable_cachedstore.patch, HIVE-16586.1.patch, > HIVE-16586.2.patch, HIVE-16586.3.patch > > > Though we don't plan to turn on CachedStore by default, we want to make sure > unit tests pass with CachedStore. I turn on CachedStore in the patch in order > to run unit tests with it, but I will turn off CachedStore when committing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16586) Fix Unit test failures when CachedStore is enabled
[ https://issues.apache.org/jira/browse/HIVE-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008519#comment-16008519 ] Daniel Dai commented on HIVE-16586: --- Here are the analysis remaining failures (with stats related methods off in CachedStore) annotate_stats_part: need more investigation load_dyn_part14: need more investigation load_dyn_part15: sort order for special character is different in derby and TreeMap repl_1_drop: getPartitionsByFilter is not implemented in CachedStore url_hook: hive.metastore.ds.connection.url.hook is not implemented in CachedStore escape1: sort order for special character is different in derby and TreeMap escape2: sort order for special character is different in derby and TreeMap partition_column_names_with_leading_and_trailing_spaces: the original test rely on derby specific issue not exist in CachedStore > Fix Unit test failures when CachedStore is enabled > -- > > Key: HIVE-16586 > URL: https://issues.apache.org/jira/browse/HIVE-16586 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: enable_cachedstore.patch, HIVE-16586.1.patch, > HIVE-16586.2.patch > > > Though we don't plan to turn on CachedStore by default, we want to make sure > unit tests pass with CachedStore. I turn on CachedStore in the patch in order > to run unit tests with it, but I will turn off CachedStore when committing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15104) Hive on Spark generate more shuffle data than hive on mr
[
https://issues.apache.org/jira/browse/HIVE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008502#comment-16008502
]
Hive QA commented on HIVE-15104:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867727/HIVE-15104.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10688 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=89)
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testSparkQuery (batchId=225)
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable (batchId=225)
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
(batchId=225)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5226/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5226/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5226/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867727 - PreCommit-HIVE-Build
> Hive on Spark generate more shuffle data than hive on mr
>
>
> Key: HIVE-15104
> URL: https://issues.apache.org/jira/browse/HIVE-15104
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.1
>Reporter: wangwenli
>Assignee: Rui Li
> Attachments: HIVE-15104.1.patch
>
>
> the same sql, running on spark and mr engine, will generate different size
> of shuffle data.
> i think it is because of hive on mr just serialize part of HiveKey, but hive
> on spark which using kryo will serialize full of Hivekey object.
> what is your opionion?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16653) Mergejoin should give itself a correct tag
[ https://issues.apache.org/jira/browse/HIVE-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16653: --- Status: Patch Available (was: Open) > Mergejoin should give itself a correct tag > -- > > Key: HIVE-16653 > URL: https://issues.apache.org/jira/browse/HIVE-16653 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16653.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16653) Mergejoin should give itself a correct tag
[ https://issues.apache.org/jira/browse/HIVE-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16653: --- Status: Open (was: Patch Available) > Mergejoin should give itself a correct tag > -- > > Key: HIVE-16653 > URL: https://issues.apache.org/jira/browse/HIVE-16653 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16653.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16653) Mergejoin should give itself a correct tag
[ https://issues.apache.org/jira/browse/HIVE-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16653: --- Attachment: (was: HIVE-16653.01.patch) > Mergejoin should give itself a correct tag > -- > > Key: HIVE-16653 > URL: https://issues.apache.org/jira/browse/HIVE-16653 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16653.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16653) Mergejoin should give itself a correct tag
[ https://issues.apache.org/jira/browse/HIVE-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16653: --- Status: Open (was: Patch Available) > Mergejoin should give itself a correct tag > -- > > Key: HIVE-16653 > URL: https://issues.apache.org/jira/browse/HIVE-16653 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16653.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16653) Mergejoin should give itself a correct tag
[ https://issues.apache.org/jira/browse/HIVE-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16653: --- Attachment: HIVE-16653.01.patch > Mergejoin should give itself a correct tag > -- > > Key: HIVE-16653 > URL: https://issues.apache.org/jira/browse/HIVE-16653 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16653.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16653) Mergejoin should give itself a correct tag
[ https://issues.apache.org/jira/browse/HIVE-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16653: --- Status: Patch Available (was: Open) > Mergejoin should give itself a correct tag > -- > > Key: HIVE-16653 > URL: https://issues.apache.org/jira/browse/HIVE-16653 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16653.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16555) Add a new thrift API call for get_metastore_uuid
[ https://issues.apache.org/jira/browse/HIVE-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-16555: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Fix has been pushed to master for Hive 3.0.0. Thank you for your contribution [~vihangk1] > Add a new thrift API call for get_metastore_uuid > > > Key: HIVE-16555 > URL: https://issues.apache.org/jira/browse/HIVE-16555 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > Fix For: 3.0.0 > > Attachments: HIVE-16555.01.patch, HIVE-16555.02.patch, > HIVE-16555.03.patch, HIVE-16555.04.patch, HIVE-16555.05.patch, > HIVE-16555.06.patch, HIVE-16555.07.patch, HIVE-16555.08.patch > > > Sub-task of the main JIRA to add the new thrift API -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16324) Truncate table should not work when EXTERNAL property of table is true
[
https://issues.apache.org/jira/browse/HIVE-16324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008425#comment-16008425
]
Thejas M Nair commented on HIVE-16324:
--
+1
> Truncate table should not work when EXTERNAL property of table is true
> --
>
> Key: HIVE-16324
> URL: https://issues.apache.org/jira/browse/HIVE-16324
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16324.01.patch
>
>
> Currently if you create an external table using the command {{CREATE EXTERNAL
> TABLE table_name}} the {{TRUNCATE table table_name}} command fails as
> expected because only managed tables should be allowed to be truncated.
> But if you set the external property of a previously managed table using
> {{ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='true')}}, truncate
> table command does not object and deletes all the data from the external
> table.
> Eg: This works but it should not ..
> {noformat}
> 0: jdbc:hive2://localhost:1/default> create table test_ext2 (col1 string);
> No rows affected (0.424 seconds)
> 0: jdbc:hive2://localhost:1/default> alter table test_ext2 set
> tblproperties ('EXTERNAL'='true');
> No rows affected (0.149 seconds)
> 0: jdbc:hive2://localhost:1/default> insert into table test_ext2 values
> ("test");
> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. spark,
> tez) or using Hive 1.X releases.
> No rows affected (3.447 seconds)
> 0: jdbc:hive2://localhost:1/default>
> 0: jdbc:hive2://localhost:1/default>
> 0: jdbc:hive2://localhost:1/default> select * from test_ext2;
> +-+
> | test_ext2.col1 |
> +-+
> | test|
> +-+
> 1 row selected (0.147 seconds)
> 0: jdbc:hive2://localhost:1/default> truncate table test_ext2;
> No rows affected (0.138 seconds)
> 0: jdbc:hive2://localhost:1/default> select * from test_ext2;
> +-+
> | test_ext2.col1 |
> +-+
> +-+
> No rows selected (0.134 seconds)
> 0: jdbc:hive2://localhost:1/default>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-1010) Implement INFORMATION_SCHEMA in Hive
[ https://issues.apache.org/jira/browse/HIVE-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008422#comment-16008422 ] Thejas M Nair commented on HIVE-1010: - Looks like JdbcStorageConfigManagerTest failure might be related. Also added a minor comment to pull request. > Implement INFORMATION_SCHEMA in Hive > > > Key: HIVE-1010 > URL: https://issues.apache.org/jira/browse/HIVE-1010 > Project: Hive > Issue Type: New Feature > Components: Metastore, Query Processor, Server Infrastructure >Reporter: Jeff Hammerbacher >Assignee: Gunther Hagleitner > Attachments: HIVE-1010.10.patch, HIVE-1010.11.patch, > HIVE-1010.12.patch, HIVE-1010.13.patch, HIVE-1010.14.patch, > HIVE-1010.15.patch, HIVE-1010.7.patch, HIVE-1010.8.patch, HIVE-1010.9.patch > > > INFORMATION_SCHEMA is part of the SQL92 standard and would be useful to > implement using our metastore. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16651) LlapProtocolClientProxy stack trace when using llap input format
[
https://issues.apache.org/jira/browse/HIVE-16651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008409#comment-16008409
]
Hive QA commented on HIVE-16651:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867624/HIVE-16651.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10688 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5225/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5225/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5225/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867624 - PreCommit-HIVE-Build
> LlapProtocolClientProxy stack trace when using llap input format
>
>
> Key: HIVE-16651
> URL: https://issues.apache.org/jira/browse/HIVE-16651
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-16651.1.patch
>
>
> Seeing this after LlapBaseRecordReader.close():
> {noformat}
> 16/06/28 22:05:32 WARN LlapProtocolClientProxy: RequestManager shutdown with
> error
> java.util.concurrent.CancellationException
> at java.util.concurrent.FutureTask.report(FutureTask.java:121)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)
> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1170)
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:91)
> at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:384)
> at java.util.concurrent.FutureTask.cancel(FutureTask.java:180)
> at
> org.apache.hadoop.hive.llap.tez.LlapProtocolClientProxy.serviceStop(LlapProtocolClientProxy.java:131)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.hive.llap.ext.LlapTaskUmbilicalExternalClient.serviceStop(LlapTaskUmbilicalExternalClient.java:135)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.AbstractService.close(AbstractService.java:250)
> at
> org.apache.hadoop.hive.llap.LlapBaseRecordReader.close(LlapBaseRecordReader.java:84)
> at
> org.apache.hadoop.hive.llap.LlapRowRecordReader.close(LlapRowRecordReader.java:80)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-14389) Beeline should not output query and prompt to stdout
[
https://issues.apache.org/jira/browse/HIVE-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008408#comment-16008408
]
Vihang Karajgaonkar commented on HIVE-14389:
vector_if_expr is a flaky test reported in HIVE-15776. explainanalyze_3 and
explainuser_3 have been failing since last many builds and unrelated to this
patch. [~ychena] Can this be merged? Thanks!
> Beeline should not output query and prompt to stdout
>
>
> Key: HIVE-14389
> URL: https://issues.apache.org/jira/browse/HIVE-14389
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-14389.01.patch, HIVE-14389.02.patch,
> HIVE-14389.03.patch, HIVE-14389.04.patch
>
>
> It seems that the Beeline prints the query along with the results in the
> stdout when a script file is passed. The output file in the example below
> needs to only have the results and not the query.
> {noformat}
> .vihang-MBP:bin vihang$ ./beeline --showheader=false --outformat=tsv2 -u
> "jdbc:hive2://localhost:1" -f /tmp/query.sql > /tmp/query.out 2>
> /tmp/query.err
> OK
> $ cat /tmp/query.out
> 1: jdbc:hive2://localhost:1/default> select * from likes limit 4;
> +---+--+--+
> | 1 | chocolate|
> | 1 | car |
> | 1 | games|
> | 1 | chess|
> +---+--+--+
> 1: jdbc:hive2://localhost:1/default>
> 1: jdbc:hive2://localhost:1/default>
> $
> {noformat}
> A lot of people use HiveCLI and in order to transition from HiveCLI scripts
> to Beeline, this needs to be taken care of. The output files generated by
> beeline should contain only the results and nothing else.
> Similarly, when not in silent mode, query are being printed out on stdout,
> which is adding garbage along with results, as just like HIVE CLI does, users
> would like to have only the results on stdout, not errors/debugging info/etc,
> like the full query.
> Query could be printed out, no problem, as long as it is not on stdout (with
> results), instead, it must be printed out along with the debugging info.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16324) Truncate table should not work when EXTERNAL property of table is true
[
https://issues.apache.org/jira/browse/HIVE-16324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008403#comment-16008403
]
Vihang Karajgaonkar commented on HIVE-16324:
[~thejas] [~aihuaxu] Can you please review? Thanks!
> Truncate table should not work when EXTERNAL property of table is true
> --
>
> Key: HIVE-16324
> URL: https://issues.apache.org/jira/browse/HIVE-16324
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16324.01.patch
>
>
> Currently if you create an external table using the command {{CREATE EXTERNAL
> TABLE table_name}} the {{TRUNCATE table table_name}} command fails as
> expected because only managed tables should be allowed to be truncated.
> But if you set the external property of a previously managed table using
> {{ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='true')}}, truncate
> table command does not object and deletes all the data from the external
> table.
> Eg: This works but it should not ..
> {noformat}
> 0: jdbc:hive2://localhost:1/default> create table test_ext2 (col1 string);
> No rows affected (0.424 seconds)
> 0: jdbc:hive2://localhost:1/default> alter table test_ext2 set
> tblproperties ('EXTERNAL'='true');
> No rows affected (0.149 seconds)
> 0: jdbc:hive2://localhost:1/default> insert into table test_ext2 values
> ("test");
> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. spark,
> tez) or using Hive 1.X releases.
> No rows affected (3.447 seconds)
> 0: jdbc:hive2://localhost:1/default>
> 0: jdbc:hive2://localhost:1/default>
> 0: jdbc:hive2://localhost:1/default> select * from test_ext2;
> +-+
> | test_ext2.col1 |
> +-+
> | test|
> +-+
> 1 row selected (0.147 seconds)
> 0: jdbc:hive2://localhost:1/default> truncate table test_ext2;
> No rows affected (0.138 seconds)
> 0: jdbc:hive2://localhost:1/default> select * from test_ext2;
> +-+
> | test_ext2.col1 |
> +-+
> +-+
> No rows selected (0.134 seconds)
> 0: jdbc:hive2://localhost:1/default>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HIVE-16324) Truncate table should not work when EXTERNAL property of table is true
[
https://issues.apache.org/jira/browse/HIVE-16324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008403#comment-16008403
]
Vihang Karajgaonkar edited comment on HIVE-16324 at 5/12/17 5:07 PM:
-
[~thejas] [~ctang.ma] Can you please review? Thanks!
was (Author: vihangk1):
[~thejas] [~aihuaxu] Can you please review? Thanks!
> Truncate table should not work when EXTERNAL property of table is true
> --
>
> Key: HIVE-16324
> URL: https://issues.apache.org/jira/browse/HIVE-16324
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16324.01.patch
>
>
> Currently if you create an external table using the command {{CREATE EXTERNAL
> TABLE table_name}} the {{TRUNCATE table table_name}} command fails as
> expected because only managed tables should be allowed to be truncated.
> But if you set the external property of a previously managed table using
> {{ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='true')}}, truncate
> table command does not object and deletes all the data from the external
> table.
> Eg: This works but it should not ..
> {noformat}
> 0: jdbc:hive2://localhost:1/default> create table test_ext2 (col1 string);
> No rows affected (0.424 seconds)
> 0: jdbc:hive2://localhost:1/default> alter table test_ext2 set
> tblproperties ('EXTERNAL'='true');
> No rows affected (0.149 seconds)
> 0: jdbc:hive2://localhost:1/default> insert into table test_ext2 values
> ("test");
> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. spark,
> tez) or using Hive 1.X releases.
> No rows affected (3.447 seconds)
> 0: jdbc:hive2://localhost:1/default>
> 0: jdbc:hive2://localhost:1/default>
> 0: jdbc:hive2://localhost:1/default> select * from test_ext2;
> +-+
> | test_ext2.col1 |
> +-+
> | test|
> +-+
> 1 row selected (0.147 seconds)
> 0: jdbc:hive2://localhost:1/default> truncate table test_ext2;
> No rows affected (0.138 seconds)
> 0: jdbc:hive2://localhost:1/default> select * from test_ext2;
> +-+
> | test_ext2.col1 |
> +-+
> +-+
> No rows selected (0.134 seconds)
> 0: jdbc:hive2://localhost:1/default>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16555) Add a new thrift API call for get_metastore_uuid
[ https://issues.apache.org/jira/browse/HIVE-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008400#comment-16008400 ] Naveen Gangam commented on HIVE-16555: -- I will push it now. Thanks for looking at the results .. > Add a new thrift API call for get_metastore_uuid > > > Key: HIVE-16555 > URL: https://issues.apache.org/jira/browse/HIVE-16555 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-16555.01.patch, HIVE-16555.02.patch, > HIVE-16555.03.patch, HIVE-16555.04.patch, HIVE-16555.05.patch, > HIVE-16555.06.patch, HIVE-16555.07.patch, HIVE-16555.08.patch > > > Sub-task of the main JIRA to add the new thrift API -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16555) Add a new thrift API call for get_metastore_uuid
[ https://issues.apache.org/jira/browse/HIVE-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008387#comment-16008387 ] Vihang Karajgaonkar commented on HIVE-16555: Both the tests above are failing since last 18 builds and unrelated to this patch. [~ngangam] Can this be merged? Thanks! > Add a new thrift API call for get_metastore_uuid > > > Key: HIVE-16555 > URL: https://issues.apache.org/jira/browse/HIVE-16555 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-16555.01.patch, HIVE-16555.02.patch, > HIVE-16555.03.patch, HIVE-16555.04.patch, HIVE-16555.05.patch, > HIVE-16555.06.patch, HIVE-16555.07.patch, HIVE-16555.08.patch > > > Sub-task of the main JIRA to add the new thrift API -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16600) Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel order by in multi_insert cases
[
https://issues.apache.org/jira/browse/HIVE-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang_intel updated HIVE-16600:
Attachment: mr.explain
[~lirui]: attach mr.explain is the total plan of {code}
set hive.mapred.mode=nonstrict;
set hive.exec.reducers.bytes.per.reducer=256;
set hive.optimize.sampling.orderby=true;
drop table if exists e1;
drop table if exists e2;
create table e1 (key string, value string);
create table e2 (key string);
FROM (select key, cast(key as double) as keyD, value from src order by key) a
INSERT OVERWRITE TABLE e1
SELECT key, value
INSERT OVERWRITE TABLE e2
SELECT key limit 10;
{code}
It seems that mr also has extra shuffle stage(Stage-4 in mr.explain) for
multi-insert case. If i am wrong,please tell me.
> Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel
> order by in multi_insert cases
>
>
> Key: HIVE-16600
> URL: https://issues.apache.org/jira/browse/HIVE-16600
> Project: Hive
> Issue Type: Sub-task
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16600.1.patch, HIVE-16600.2.patch,
> HIVE-16600.3.patch, mr.explain, mr.explain.log.HIVE-16600
>
>
> multi_insert_gby.case.q
> {code}
> set hive.exec.reducers.bytes.per.reducer=256;
> set hive.optimize.sampling.orderby=true;
> drop table if exists e1;
> drop table if exists e2;
> create table e1 (key string, value string);
> create table e2 (key string);
> FROM (select key, cast(key as double) as keyD, value from src order by key) a
> INSERT OVERWRITE TABLE e1
> SELECT key, value
> INSERT OVERWRITE TABLE e2
> SELECT key;
> select * from e1;
> select * from e2;
> {code}
> the parallelism of Sort is 1 even we enable parallel order
> by("hive.optimize.sampling.orderby" is set as "true"). This is not
> reasonable because the parallelism should be calcuated by
> [Utilities.estimateReducers|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L170]
> this is because SetSparkReducerParallelism#needSetParallelism returns false
> when [children size of
> RS|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L207]
> is greater than 1.
> in this case, the children size of {{RS[2]}} is two.
> the logical plan of the case
> {code}
>TS[0]-SEL[1]-RS[2]-SEL[3]-SEL[4]-FS[5]
> -SEL[6]-FS[7]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15725) Make it possible to run checkstyle for a specific module
[ https://issues.apache.org/jira/browse/HIVE-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008332#comment-16008332 ] Thejas M Nair commented on HIVE-15725: -- +1 > Make it possible to run checkstyle for a specific module > > > Key: HIVE-15725 > URL: https://issues.apache.org/jira/browse/HIVE-15725 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-15725.patch > > > Remove relative paths from the checkstyle.xml, so it could be run for every > module, not just from the root. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16652) LlapInputFormat: Seeing "output error" WARN message
[
https://issues.apache.org/jira/browse/HIVE-16652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008321#comment-16008321
]
Hive QA commented on HIVE-16652:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867715/HIVE-16652.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10688 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
org.apache.hive.hcatalog.templeton.TestConcurrentJobRequestsThreadsAndTimeout.ConcurrentListJobsVerifyExceptions
(batchId=177)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5224/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5224/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5224/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867715 - PreCommit-HIVE-Build
> LlapInputFormat: Seeing "output error" WARN message
> ---
>
> Key: HIVE-16652
> URL: https://issues.apache.org/jira/browse/HIVE-16652
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-16652.1.patch
>
>
> Another warning message I'm seeing in the logs for TestJdbcWithMiniLlap after
> adding the line to close the RecordReader in the test:
> {noformat}
> 2017-05-11T11:08:34,511 WARN [IPC Server handler 0 on 54847] ipc.Server: IPC
> Server handler 0 on 54847, call Call#341 Retry#0 heartbeat({
> containerId=container_6830411502416918223_0003_00_00, requestId=2,
> startIndex=0, preRoutedStartIndex=0, maxEventsToGet=500,
> taskAttemptId=attempt_6830411502416918223_0003_0_00_00_0, eventCount=2
> }), rpc version=2, client version=1, methodsFingerPrint=996603002 from
> 10.22.8.180:54849: output error
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.18.patch > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.10.patch, HIVE-14731.11.patch, > HIVE-14731.12.patch, HIVE-14731.13.patch, HIVE-14731.14.patch, > HIVE-14731.15.patch, HIVE-14731.16.patch, HIVE-14731.17.patch, > HIVE-14731.18.patch, HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[
https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008283#comment-16008283
]
Hive QA commented on HIVE-14731:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867709/HIVE-14731.17.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5223/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5223/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5223/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-12 15:24:46.966
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-5223/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-12 15:24:46.969
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 603c534 HIVE-16646: Alias in transform ... as clause shouldn't
be case sensitive (Yibing Shi, reviewed by Yongzhi Chen)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 603c534 HIVE-16646: Alias in transform ... as clause shouldn't
be case sensitive (Yibing Shi, reviewed by Yongzhi Chen)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-12 15:24:47.566
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p0
patching file
common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java
patching file
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java
patching file common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
patching file data/conf/llap/hive-site.xml
patching file data/conf/tez/hive-site.xml
patching file itests/src/test/resources/testconfiguration.properties
patching file pom.xml
patching file ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
patching file ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CrossProductCheck.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CrossProductHandler.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkCrossProductCheck.java
patching file ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
patching file ql/src/java/org/apache/hadoop/hive/ql/plan/TezEdgeProperty.java
patching file ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java
patching file ql/src/test/queries/clientpositive/cross_prod_1.q
patching file ql/src/test/queries/clientpositive/cross_prod_2.q
patching file ql/src/test/queries/clientpositive/cross_prod_3.q
patching file ql/src/test/queries/clientpositive/cross_prod_4.q
patching file ql/src/test/results/clientpositive/llap/cross_join.q.out
patching file ql/src/test/results/clientpositive/llap/cross_prod_1.q.out
patching file ql/src/test/results/clientpositive/llap/cross_prod_2.q.out
patching file ql/src/test/results/clientpositive/llap/cross_prod_3.q.out
patching file ql/src/test/results/clientpositive/llap/cross_prod_4.q.out
patching file
ql/src/test/results/clientpositive/llap/cross_product_check_1.q.out
patching file ql/src/test/results/clientpositive/llap/cte_5.q.out
patching file ql/src/test/results/clientpositive/llap/cte_mat_1.q.out
patching file q
[jira] [Commented] (HIVE-16658) TestTimestampTZ.java has missed the ASF header
[
https://issues.apache.org/jira/browse/HIVE-16658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008281#comment-16008281
]
Hive QA commented on HIVE-16658:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867707/HIVE-16658.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10688 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5222/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5222/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5222/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867707 - PreCommit-HIVE-Build
> TestTimestampTZ.java has missed the ASF header
> --
>
> Key: HIVE-16658
> URL: https://issues.apache.org/jira/browse/HIVE-16658
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Saijin Huang
>Assignee: Saijin Huang
>Priority: Minor
> Attachments: HIVE-16658.1.patch
>
>
> when i test the issue HIVE-14412, TestTimestampTZ has missed the ASF header
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16602) Implement shared scans with Tez
[ https://issues.apache.org/jira/browse/HIVE-16602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008279#comment-16008279 ] Ashutosh Chauhan commented on HIVE-16602: - +1 > Implement shared scans with Tez > --- > > Key: HIVE-16602 > URL: https://issues.apache.org/jira/browse/HIVE-16602 > Project: Hive > Issue Type: New Feature > Components: Physical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-16602.01.patch, HIVE-16602.02.patch, > HIVE-16602.03.patch, HIVE-16602.04.patch, HIVE-16602.patch > > > Given a query plan, the goal is to identify scans on input tables that can be > merged so the data is read only once. Optimization will be carried out at the > physical level. > In the longer term, identification of equivalent expressions and > reutilization of intermediary results should be done at the logical layer via > Spool operator. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16607) ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID
[
https://issues.apache.org/jira/browse/HIVE-16607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16607:
--
Attachment: HIVE-16607.06.patch
Patch rebased to the current head
> ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID
> -
>
> Key: HIVE-16607
> URL: https://issues.apache.org/jira/browse/HIVE-16607
> Project: Hive
> Issue Type: Bug
> Components: Beeline, Logging
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16607.02.patch, HIVE-16607.04.patch,
> HIVE-16607.05.patch, HIVE-16607.06.patch, HIVE-16607.patch
>
>
> Creating a new {{QueryState}} object regenerates the HIVEQUERYID stored in
> the {{HiveConf}}.
> In HiveServer logs it makes hard to follow the life of the query since a new
> queryid is assigned to the query during the execution.
> Since BeeLine is showing the operation logs based on the queryid, only the
> first several line of the logs is showed in BeeLine.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HIVE-16643) BeeLine tests output should keep the PREHOOK/POSTHOOK Input/Output orderdering
[
https://issues.apache.org/jira/browse/HIVE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008236#comment-16008236
]
Peter Vary edited comment on HIVE-16643 at 5/12/17 2:46 PM:
Retriggering the tests with rebased patch
was (Author: pvary):
Retriggering the tests
> BeeLine tests output should keep the PREHOOK/POSTHOOK Input/Output orderdering
> --
>
> Key: HIVE-16643
> URL: https://issues.apache.org/jira/browse/HIVE-16643
> Project: Hive
> Issue Type: New Feature
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16643.01.patch, HIVE-16643.patch
>
>
> The {{PreExecutePrinter}} and the {{PostExecutePrinter}} prints the query
> input and the output list in alphabetical order in {{printEntities}} method.
> Our goal is to have the same output from the BeeLine query tests, and the Cli
> query tests. Since the BeeLine tests are using test specific databases to run
> the tests, and only converting the results in the end to remove this specific
> database names from the output, we have to reorder the lists after this
> conversion.
> Raw BeeLine output:
> {code}
> [..]
> INFO : PREHOOK: Output: create_merge_compressed@src_rc_merge_test
> INFO : PREHOOK: Output: database:create_merge_compressed
> [..]
> {code}
> Before patch BeeLine output:
> {code}
> [..]
> PREHOOK: Output: default@src_rc_merge_test
> PREHOOK: Output: database:default
> [..]
> {code}
> Expected output:
> {code}
> [..]
> PREHOOK: Output: database:default
> PREHOOK: Output: default@src_rc_merge_test
> [..]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16643) BeeLine tests output should keep the PREHOOK/POSTHOOK Input/Output orderdering
[
https://issues.apache.org/jira/browse/HIVE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16643:
--
Attachment: HIVE-16643.01.patch
Retriggering the tests
> BeeLine tests output should keep the PREHOOK/POSTHOOK Input/Output orderdering
> --
>
> Key: HIVE-16643
> URL: https://issues.apache.org/jira/browse/HIVE-16643
> Project: Hive
> Issue Type: New Feature
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16643.01.patch, HIVE-16643.patch
>
>
> The {{PreExecutePrinter}} and the {{PostExecutePrinter}} prints the query
> input and the output list in alphabetical order in {{printEntities}} method.
> Our goal is to have the same output from the BeeLine query tests, and the Cli
> query tests. Since the BeeLine tests are using test specific databases to run
> the tests, and only converting the results in the end to remove this specific
> database names from the output, we have to reorder the lists after this
> conversion.
> Raw BeeLine output:
> {code}
> [..]
> INFO : PREHOOK: Output: create_merge_compressed@src_rc_merge_test
> INFO : PREHOOK: Output: database:create_merge_compressed
> [..]
> {code}
> Before patch BeeLine output:
> {code}
> [..]
> PREHOOK: Output: default@src_rc_merge_test
> PREHOOK: Output: database:default
> [..]
> {code}
> Expected output:
> {code}
> [..]
> PREHOOK: Output: database:default
> PREHOOK: Output: default@src_rc_merge_test
> [..]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16233) llap: Query failed with AllocatorOutOfMemoryException
[
https://issues.apache.org/jira/browse/HIVE-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008214#comment-16008214
]
Hive QA commented on HIVE-16233:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12867705/HIVE-16233.05.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10699 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5221/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5221/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5221/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12867705 - PreCommit-HIVE-Build
> llap: Query failed with AllocatorOutOfMemoryException
> -
>
> Key: HIVE-16233
> URL: https://issues.apache.org/jira/browse/HIVE-16233
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16233.01.patch, HIVE-16233.02.patch,
> HIVE-16233.03.patch, HIVE-16233.04.patch, HIVE-16233.05.patch
>
>
> {code}
> TaskAttempt 5 failed, info=[Error: Error while running task ( failure ) :
> attempt_1488231257387_2288_25_05_56_5:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 262144; at 0 out of 1
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 262144; at 0 out of 1
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: java.io.IOException: java.io.IOException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 262144; at 0 out of 1
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveConte
[jira] [Updated] (HIVE-16647) Improve the validation output to make the output to stderr and stdout more consistent
[ https://issues.apache.org/jira/browse/HIVE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-16647: Status: Patch Available (was: In Progress) > Improve the validation output to make the output to stderr and stdout more > consistent > - > > Key: HIVE-16647 > URL: https://issues.apache.org/jira/browse/HIVE-16647 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Minor > Attachments: HIVE-16647.1.patch > > > Some output are printed to stderr or stdout inconsistently. Here are some of > them. Update to make them more consistent. > * Version table validation > When the version table is missing, the err msg goes to stderr > When the version table is not valid, the err msg goes to stdout with a > message like "Failed in schema version validation: > * Metastore/schema table validation > ** When the version table contains the wrong version or there are no rows in > the version table, err msg goes to stderr > ** When there diffs between the schema and metastore tables, the err msg goes > to stdout -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16647) Improve the validation output to make the output to stderr and stdout more consistent
[ https://issues.apache.org/jira/browse/HIVE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-16647: Attachment: (was: HIVE-16647.1.patch) > Improve the validation output to make the output to stderr and stdout more > consistent > - > > Key: HIVE-16647 > URL: https://issues.apache.org/jira/browse/HIVE-16647 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Minor > Attachments: HIVE-16647.1.patch > > > Some output are printed to stderr or stdout inconsistently. Here are some of > them. Update to make them more consistent. > * Version table validation > When the version table is missing, the err msg goes to stderr > When the version table is not valid, the err msg goes to stdout with a > message like "Failed in schema version validation: > * Metastore/schema table validation > ** When the version table contains the wrong version or there are no rows in > the version table, err msg goes to stderr > ** When there diffs between the schema and metastore tables, the err msg goes > to stdout -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16647) Improve the validation output to make the output to stderr and stdout more consistent
[ https://issues.apache.org/jira/browse/HIVE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-16647: Attachment: HIVE-16647.1.patch > Improve the validation output to make the output to stderr and stdout more > consistent > - > > Key: HIVE-16647 > URL: https://issues.apache.org/jira/browse/HIVE-16647 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Minor > Attachments: HIVE-16647.1.patch > > > Some output are printed to stderr or stdout inconsistently. Here are some of > them. Update to make them more consistent. > * Version table validation > When the version table is missing, the err msg goes to stderr > When the version table is not valid, the err msg goes to stdout with a > message like "Failed in schema version validation: > * Metastore/schema table validation > ** When the version table contains the wrong version or there are no rows in > the version table, err msg goes to stderr > ** When there diffs between the schema and metastore tables, the err msg goes > to stdout -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16660) Not able to add partition for views in hive when sentry is enabled
[ https://issues.apache.org/jira/browse/HIVE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-16660: Status: Patch Available (was: Open) Need code review. > Not able to add partition for views in hive when sentry is enabled > -- > > Key: HIVE-16660 > URL: https://issues.apache.org/jira/browse/HIVE-16660 > Project: Hive > Issue Type: Bug > Components: Parser >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-16660.1.patch > > > Repro: > create table tesnit (a int) partitioned by (p int); > insert into table tesnit partition (p = 1) values (1); > insert into table tesnit partition (p = 2) values (1); > create view test_view partitioned on (p) as select * from tesnit where p =1; > alter view test_view add partition (p = 2); > Error: Error while compiling statement: FAILED: SemanticException [Error > 10056]: The query does not reference any valid partition. To run this query, > set hive.mapred.mode=nonstrict (state=42000,code=10056) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16660) Not able to add partition for views in hive when sentry is enabled
[
https://issues.apache.org/jira/browse/HIVE-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-16660:
Attachment: HIVE-16660.1.patch
The error is because hive fail to compile internal query generated for the
alter operation:
SELECT * FROM `default.test_view` WHERE (`p` = '2'); caused NPE when
sentry enabled:
{noformat}
HiveServer2-Handler-Pool: Thread-68]: FAILED: NullPointerException null
java.lang.NullPointerException
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
at com.google.common.cache.LocalCache.get(LocalCache.java:3964)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
at
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:215)
at
org.apache.sentry.provider.common.HadoopGroupMappingService.getGroups(HadoopGroupMappingService.java:60)
at
org.apache.sentry.binding.hive.authz.HiveAuthzBinding.getGroups(HiveAuthzBinding.java:372)
at
org.apache.sentry.binding.hive.HiveAuthzBindingHook.postAnalyze(HiveAuthzBindingHook.java:395)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:449)
at
org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterTableAddParts(DDLSemanticAnalyzer.java:2819)
at
org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:385)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:312)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1201)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1188)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:134)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:206)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:327)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:424)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:401)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:500)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
The NPE is because of null userName for the newly created Driver object for the
query.
The patch fix the issue by adding username information. Also fix the NPE when
some user applicate call get partition path information from a view. The NPE
like:
{noformat}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.metadata.Partition.getDataLocation(Partition.java:241)
at
org.apache.hadoop.hive.ql.metadata.Partition.getPartitionPath(Partition.java:236)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1648)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1318)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1127)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1120)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:178)
... 11 more
{noformat}
> Not able to add partition for views in hive when sentry is enabled
> --
>
> Key: HIVE-16660
> URL: https://issues.apache.org/jira/browse/HIVE-16660
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-16660.1.patch
>
>
> Repro:
> create table tesnit (a int) partitioned by (p int);
> insert into table tesnit partition (p = 1) values (1);
[jira] [Updated] (HIVE-16602) Implement shared scans with Tez
[ https://issues.apache.org/jira/browse/HIVE-16602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-16602: --- Attachment: HIVE-16602.04.patch > Implement shared scans with Tez > --- > > Key: HIVE-16602 > URL: https://issues.apache.org/jira/browse/HIVE-16602 > Project: Hive > Issue Type: New Feature > Components: Physical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-16602.01.patch, HIVE-16602.02.patch, > HIVE-16602.03.patch, HIVE-16602.04.patch, HIVE-16602.patch > > > Given a query plan, the goal is to identify scans on input tables that can be > merged so the data is read only once. Optimization will be carried out at the > physical level. > In the longer term, identification of equivalent expressions and > reutilization of intermediary results should be done at the logical layer via > Spool operator. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
