[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[
https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568617#comment-14568617
]
Hive QA commented on HIVE-4239:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736698/HIVE-4239.05.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 8997 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementParallel
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testGetFunctions
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testOpenSession
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4139/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4139/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4139/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736698 - PreCommit-HIVE-TRUNK-Build
> Remove lock on compilation stage
>
>
> Key: HIVE-4239
> URL: https://issues.apache.org/jira/browse/HIVE-4239
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Processor
>Reporter: Carl Steinbach
>Assignee: Sergey Shelukhin
> Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch,
> HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10674) jars should not be checked in to the source control repo
[ https://issues.apache.org/jira/browse/HIVE-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568618#comment-14568618 ] Ferdinand Xu commented on HIVE-10674: - This failures should only be related to unit test. You can go skipping them via "-DskipTests". > jars should not be checked in to the source control repo > > > Key: HIVE-10674 > URL: https://issues.apache.org/jira/browse/HIVE-10674 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 1.3.0 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan >Priority: Blocker > > The 2 jars added by HIVE-9302 run afoul of the source package generation as > part of the 1.2 release, since a source package is not supposed to contain > any binaries. If we have binaries, they're supposed to be brought in as a > download step during the compile or test-compile phase from a well-known > published location such as a maven repository. The postgres jar we can depend > on as a download, and it is an open source product that is compatible with > the Apache License, but DummyDriver is worse, because there is no source > attached to it either, which makes it not okay to include in the binary > release of hive either. > Thus, for branch-1.2, I am going to do a git rm of those two jars right away. > This, unfortunately, might cause a few tests added here to fail for > branch-1.2, but this should be acceptable for the time being. > I'm opening this jira to track the following: > a) git rm of the postgres and DummyDriver jar from master > b) adding source code for DummyDriver into master, and changing the build so > we depend on it being compiled, rather than included from test-resources. > c) changing the postgres inclusion to a download. > This should also be applied to branch-1.2 after release, preferably before > 1.2.1, so that future updates of 1.2 have this fixed as well. > Note: In addition, there are two more jars added by HIVE-7553 which also need > to have this same fix applied to them : > ql/src/test/resources/SessionStateTest.jar.v1 and > ql/src/test/resources/SessionStateTest.jar.v1 . These are included in a > similar manner, for tests, but are binaries without source code attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10674) jars should not be checked in to the source control repo
[ https://issues.apache.org/jira/browse/HIVE-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-10674: Assignee: Sushanth Sowmyan > jars should not be checked in to the source control repo > > > Key: HIVE-10674 > URL: https://issues.apache.org/jira/browse/HIVE-10674 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 1.3.0 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan >Priority: Blocker > > The 2 jars added by HIVE-9302 run afoul of the source package generation as > part of the 1.2 release, since a source package is not supposed to contain > any binaries. If we have binaries, they're supposed to be brought in as a > download step during the compile or test-compile phase from a well-known > published location such as a maven repository. The postgres jar we can depend > on as a download, and it is an open source product that is compatible with > the Apache License, but DummyDriver is worse, because there is no source > attached to it either, which makes it not okay to include in the binary > release of hive either. > Thus, for branch-1.2, I am going to do a git rm of those two jars right away. > This, unfortunately, might cause a few tests added here to fail for > branch-1.2, but this should be acceptable for the time being. > I'm opening this jira to track the following: > a) git rm of the postgres and DummyDriver jar from master > b) adding source code for DummyDriver into master, and changing the build so > we depend on it being compiled, rather than included from test-resources. > c) changing the postgres inclusion to a download. > This should also be applied to branch-1.2 after release, preferably before > 1.2.1, so that future updates of 1.2 have this fixed as well. > Note: In addition, there are two more jars added by HIVE-7553 which also need > to have this same fix applied to them : > ql/src/test/resources/SessionStateTest.jar.v1 and > ql/src/test/resources/SessionStateTest.jar.v1 . These are included in a > similar manner, for tests, but are binaries without source code attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10674) jars should not be checked in to the source control repo
[ https://issues.apache.org/jira/browse/HIVE-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Wei reassigned HIVE-10674: - Assignee: Yu Wei (was: Sushanth Sowmyan) > jars should not be checked in to the source control repo > > > Key: HIVE-10674 > URL: https://issues.apache.org/jira/browse/HIVE-10674 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 1.3.0 >Reporter: Sushanth Sowmyan >Assignee: Yu Wei >Priority: Blocker > > The 2 jars added by HIVE-9302 run afoul of the source package generation as > part of the 1.2 release, since a source package is not supposed to contain > any binaries. If we have binaries, they're supposed to be brought in as a > download step during the compile or test-compile phase from a well-known > published location such as a maven repository. The postgres jar we can depend > on as a download, and it is an open source product that is compatible with > the Apache License, but DummyDriver is worse, because there is no source > attached to it either, which makes it not okay to include in the binary > release of hive either. > Thus, for branch-1.2, I am going to do a git rm of those two jars right away. > This, unfortunately, might cause a few tests added here to fail for > branch-1.2, but this should be acceptable for the time being. > I'm opening this jira to track the following: > a) git rm of the postgres and DummyDriver jar from master > b) adding source code for DummyDriver into master, and changing the build so > we depend on it being compiled, rather than included from test-resources. > c) changing the postgres inclusion to a download. > This should also be applied to branch-1.2 after release, preferably before > 1.2.1, so that future updates of 1.2 have this fixed as well. > Note: In addition, there are two more jars added by HIVE-7553 which also need > to have this same fix applied to them : > ql/src/test/resources/SessionStateTest.jar.v1 and > ql/src/test/resources/SessionStateTest.jar.v1 . These are included in a > similar manner, for tests, but are binaries without source code attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10674) jars should not be checked in to the source control repo
[ https://issues.apache.org/jira/browse/HIVE-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Wei updated HIVE-10674: -- Assignee: (was: Yu Wei) > jars should not be checked in to the source control repo > > > Key: HIVE-10674 > URL: https://issues.apache.org/jira/browse/HIVE-10674 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 1.3.0 >Reporter: Sushanth Sowmyan >Priority: Blocker > > The 2 jars added by HIVE-9302 run afoul of the source package generation as > part of the 1.2 release, since a source package is not supposed to contain > any binaries. If we have binaries, they're supposed to be brought in as a > download step during the compile or test-compile phase from a well-known > published location such as a maven repository. The postgres jar we can depend > on as a download, and it is an open source product that is compatible with > the Apache License, but DummyDriver is worse, because there is no source > attached to it either, which makes it not okay to include in the binary > release of hive either. > Thus, for branch-1.2, I am going to do a git rm of those two jars right away. > This, unfortunately, might cause a few tests added here to fail for > branch-1.2, but this should be acceptable for the time being. > I'm opening this jira to track the following: > a) git rm of the postgres and DummyDriver jar from master > b) adding source code for DummyDriver into master, and changing the build so > we depend on it being compiled, rather than included from test-resources. > c) changing the postgres inclusion to a download. > This should also be applied to branch-1.2 after release, preferably before > 1.2.1, so that future updates of 1.2 have this fixed as well. > Note: In addition, there are two more jars added by HIVE-7553 which also need > to have this same fix applied to them : > ql/src/test/resources/SessionStateTest.jar.v1 and > ql/src/test/resources/SessionStateTest.jar.v1 . These are included in a > similar manner, for tests, but are binaries without source code attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10667) Upgrade Calcite dependency to newest version
[ https://issues.apache.org/jira/browse/HIVE-10667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568546#comment-14568546 ] Ashutosh Chauhan commented on HIVE-10667: - +1 > Upgrade Calcite dependency to newest version > > > Key: HIVE-10667 > URL: https://issues.apache.org/jira/browse/HIVE-10667 > Project: Hive > Issue Type: Task >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-10667.01.patch, HIVE-10667.02.patch, > HIVE-10667.03.patch, HIVE-10667.04.patch, HIVE-10667.05.patch, > HIVE-10667.patch > > > Upgrade Calcite version to 1.3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10705) Update tests for HIVE-9302 after removing binaries
[ https://issues.apache.org/jira/browse/HIVE-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-10705: Attachment: HIVE-10705.1.patch > Update tests for HIVE-9302 after removing binaries > -- > > Key: HIVE-10705 > URL: https://issues.apache.org/jira/browse/HIVE-10705 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-10705.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10705) Update tests for HIVE-9302 after removing binaries
[ https://issues.apache.org/jira/browse/HIVE-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-10705: Attachment: (was: HIVE-10705.patch) > Update tests for HIVE-9302 after removing binaries > -- > > Key: HIVE-10705 > URL: https://issues.apache.org/jira/browse/HIVE-10705 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-10705.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10749) Implement Insert ACID statement for parquet [Parquet branch]
[ https://issues.apache.org/jira/browse/HIVE-10749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-10749: Attachment: HIVE-10749.3-parquet.patch Reattach the patch to trigger the precommit for parquet branch. > Implement Insert ACID statement for parquet [Parquet branch] > > > Key: HIVE-10749 > URL: https://issues.apache.org/jira/browse/HIVE-10749 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-10749.1.patch, HIVE-10749.1.patch, > HIVE-10749.2-parquet.patch, HIVE-10749.2.patch, HIVE-10749.3-parquet.patch, > HIVE-10749.3-parquet.patch, HIVE-10749.3-parquet.patch, HIVE-10749.patch > > > We need to implement insert statement for parquet format like ORC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10684) Fix the unit test failures for HIVE-7553 after HIVE-10674 removed the binary jar files
[ https://issues.apache.org/jira/browse/HIVE-10684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568522#comment-14568522 ] Ferdinand Xu commented on HIVE-10684: - This patch will not fix the failures you mentioned. Please take a look at HIVE-10705. And this patch has already be committed to branch 1.2. Thank you! > Fix the unit test failures for HIVE-7553 after HIVE-10674 removed the binary > jar files > -- > > Key: HIVE-10684 > URL: https://issues.apache.org/jira/browse/HIVE-10684 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-10684.1.patch, HIVE-10684.2.patch, HIVE-10684.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10885) with vectorization enabled join operation involving interval_day_time fails
[
https://issues.apache.org/jira/browse/HIVE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-10885:
Attachment: HIVE-10885.02.patch
> with vectorization enabled join operation involving interval_day_time fails
> ---
>
> Key: HIVE-10885
> URL: https://issues.apache.org/jira/browse/HIVE-10885
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jagruti Varia
>Assignee: Matt McCline
> Attachments: HIVE-10885.01.patch, HIVE-10885.02.patch
>
>
> When vectorization is on, join operation involving interval_day_time type
> throws following error:
> {noformat}
> Status: Failed
> Vertex failed, vertexName=Map 2, vertexId=vertex_1432858236614_0247_1_01,
> diagnostics=[Task failed, taskId=task_1432858236614_0247_1_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:337)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
> ... 14 more
> Caused by: java.lang.RuntimeException: Cannot allocate vector copy row for
> interval_day_time
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorCopyRow.init(VectorCopyRow.java:213)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.initializeOp(VectorMapJoinCommonOperator.java:581)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:362)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:214)
> ... 15 more
> ], TaskAttempt 1 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:337)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExe
[jira] [Commented] (HIVE-10872) LLAP: make sure tests pass
[
https://issues.apache.org/jira/browse/HIVE-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568510#comment-14568510
]
Hive QA commented on HIVE-10872:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736697/HIVE-10872.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4138/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4138/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4138/
Messages:
{noformat}
This message was trimmed, see log for full details
67/175 KB
71/175 KB
75/175 KB
77/175 KB
81/175 KB
85/175 KB
89/175 KB
93/175 KB
97/175 KB
101/175 KB
105/175 KB
109/175 KB
113/175 KB
117/175 KB
121/175 KB
123/175 KB
127/175 KB
131/175 KB
135/175 KB
137/175 KB
138/175 KB
142/175 KB
146/175 KB
150/175 KB
154/175 KB
158/175 KB
162/175 KB
166/175 KB
170/175 KB
174/175 KB
175/175 KB
Downloaded:
http://repository.apache.org/snapshots/org/apache/tez/tez-runtime-internals/0.8.0-TEZ-2003-SNAPSHOT/tez-runtime-internals-0.8.0-TEZ-2003-20150601.210803-1.jar
(175 KB at 217.9 KB/sec)
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-exec ---
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-exec ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (generate-sources) @ hive-exec ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-test-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen
Generating vector expression code
Generating vector expression test code
[INFO] Executed tasks
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-exec ---
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/src/gen/protobuf/gen-java
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/src/gen/thrift/gen-javabean
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java
added.
[INFO]
[INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-exec ---
[INFO] ANTLR: Processing source directory
/data/hive-ptest/working/apache-github-source-source/ql/src/java
ANTLR Parser Generator Version 3.4
org/apache/hadoop/hive/ql/parse/HiveLexer.g
org/apache/hadoop/hive/ql/parse/HiveParser.g
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_ORDER KW_BY" using
multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_GROUP KW_BY" using
multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_CLUSTER KW_BY" using
multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_UNION KW_MAP" using
multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_UNION KW_SELECT"
using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_MAP LPAREN" using
multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_UNION KW_REDUCE"
using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_LATERAL KW_VIEW"
using multiple alternatives: 2, 9
As a result, alternative(s) 9 wer
[jira] [Commented] (HIVE-10884) Enable some beeline tests and turn on HIVE-4239 by default
[
https://issues.apache.org/jira/browse/HIVE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568505#comment-14568505
]
Hive QA commented on HIVE-10884:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736696/HIVE-10884.02.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 8996 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4137/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4137/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4137/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736696 - PreCommit-HIVE-TRUNK-Build
> Enable some beeline tests and turn on HIVE-4239 by default
> --
>
> Key: HIVE-10884
> URL: https://issues.apache.org/jira/browse/HIVE-10884
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-10884.01.patch, HIVE-10884.02.patch,
> HIVE-10884.patch
>
>
> See comments in HIVE-4239.
> Beeline tests with parallelism need to be enabled to turn compilation
> parallelism on by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10667) Upgrade Calcite dependency to newest version
[ https://issues.apache.org/jira/browse/HIVE-10667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-10667: --- Attachment: HIVE-10667.05.patch > Upgrade Calcite dependency to newest version > > > Key: HIVE-10667 > URL: https://issues.apache.org/jira/browse/HIVE-10667 > Project: Hive > Issue Type: Task >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-10667.01.patch, HIVE-10667.02.patch, > HIVE-10667.03.patch, HIVE-10667.04.patch, HIVE-10667.05.patch, > HIVE-10667.patch > > > Upgrade Calcite version to 1.3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10705) Update tests for HIVE-9302 after removing binaries
[ https://issues.apache.org/jira/browse/HIVE-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-10705: Attachment: HIVE-10705.patch Hi [~hsubramaniyan], [~sushanth], [~vgumashta], do you have some time reviewing this patch? Thank you! > Update tests for HIVE-9302 after removing binaries > -- > > Key: HIVE-10705 > URL: https://issues.apache.org/jira/browse/HIVE-10705 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-10705.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568442#comment-14568442 ] Thejas M Nair edited comment on HIVE-4239 at 6/2/15 3:08 AM: - [~sershe] Thanks for all your hard work! This is much needed and long due improvement ! I am just trying to make sure that we are ready to let these guards go. was (Author: thejas): [~sershe] Thanks for all your hard work! This is much needed and long due improvement ! > Remove lock on compilation stage > > > Key: HIVE-4239 > URL: https://issues.apache.org/jira/browse/HIVE-4239 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Query Processor >Reporter: Carl Steinbach >Assignee: Sergey Shelukhin > Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, > HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568442#comment-14568442 ] Thejas M Nair commented on HIVE-4239: - [~sershe] Thanks for all your hard work! This is much needed and long due improvement ! > Remove lock on compilation stage > > > Key: HIVE-4239 > URL: https://issues.apache.org/jira/browse/HIVE-4239 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Query Processor >Reporter: Carl Steinbach >Assignee: Sergey Shelukhin > Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, > HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10887) TestCliDriver tests ordering issues with Mac and CentOS
[
https://issues.apache.org/jira/browse/HIVE-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568441#comment-14568441
]
Hive QA commented on HIVE-10887:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736686/HIVE-10887.1.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 8994 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4136/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4136/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4136/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736686 - PreCommit-HIVE-TRUNK-Build
> TestCliDriver tests ordering issues with Mac and CentOS
> ---
>
> Key: HIVE-10887
> URL: https://issues.apache.org/jira/browse/HIVE-10887
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-10887.1.patch
>
>
> unionDistinct_2 and update_after_multiple_inserts tests give different
> results in different environments due to ordering issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-2496) Allow ALTER TABLE RENAME between schemas
[ https://issues.apache.org/jira/browse/HIVE-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568438#comment-14568438 ] Jay Lee commented on HIVE-2496: --- Since https://issues.apache.org/jira/browse/HIVE-9180 is resolve, should this issue be close too? > Allow ALTER TABLE RENAME between schemas > > > Key: HIVE-2496 > URL: https://issues.apache.org/jira/browse/HIVE-2496 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Patrick Angeles > Attachments: HIVE-2496.1.patch, HIVE-2496.2.patch > > > Currently, this is not allowed which is unfortunate: > ALTER TABLE db1.foo RENAME TO db2.foo ; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568435#comment-14568435 ] Thejas M Nair commented on HIVE-4239: - I am sorry about the delay in adding comments (as the patch has been around for couple of days). I will do some more review of associated code related to the HIVE-6245 issue by tomorrow. [~sershe] Did you do any survey of all static variables used in compilation ? I am just trying to to understand how you determined the changes that are needed. Maybe we should also check if there are potentially any libraries that we use that might be thread unsafe (say the parser libraries etc). For identifying potential race conditions, some testing alone would not suffice, a systematic review of code is very important. Any reports of race conditions are extremely hard to debug. Anything we can do to reduce the chances of that would be very valuable. > Remove lock on compilation stage > > > Key: HIVE-4239 > URL: https://issues.apache.org/jira/browse/HIVE-4239 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Query Processor >Reporter: Carl Steinbach >Assignee: Sergey Shelukhin > Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, > HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-10889) LLAP: HIVE-10778 has NPE
[ https://issues.apache.org/jira/browse/HIVE-10889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-10889. - Resolution: Fixed in branch > LLAP: HIVE-10778 has NPE > > > Key: HIVE-10889 > URL: https://issues.apache.org/jira/browse/HIVE-10889 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10885) with vectorization enabled join operation involving interval_day_time fails
[
https://issues.apache.org/jira/browse/HIVE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568391#comment-14568391
]
Gunther Hagleitner commented on HIVE-10885:
---
Do you still need the isXFamily methods with this? (since you switched to case
statements)
Otherwise +1 assuming tests pass.
> with vectorization enabled join operation involving interval_day_time fails
> ---
>
> Key: HIVE-10885
> URL: https://issues.apache.org/jira/browse/HIVE-10885
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jagruti Varia
>Assignee: Matt McCline
> Attachments: HIVE-10885.01.patch
>
>
> When vectorization is on, join operation involving interval_day_time type
> throws following error:
> {noformat}
> Status: Failed
> Vertex failed, vertexName=Map 2, vertexId=vertex_1432858236614_0247_1_01,
> diagnostics=[Task failed, taskId=task_1432858236614_0247_1_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:337)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
> ... 14 more
> Caused by: java.lang.RuntimeException: Cannot allocate vector copy row for
> interval_day_time
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorCopyRow.init(VectorCopyRow.java:213)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.initializeOp(VectorMapJoinCommonOperator.java:581)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:362)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:214)
> ... 15 more
> ], TaskAttempt 1 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:337)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
>
[jira] [Updated] (HIVE-10889) LLAP: HIVE-10778 has NPE
[ https://issues.apache.org/jira/browse/HIVE-10889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-10889: Summary: LLAP: HIVE-10778 has NPE (was: HIVE-10778 has NPE) > LLAP: HIVE-10778 has NPE > > > Key: HIVE-10889 > URL: https://issues.apache.org/jira/browse/HIVE-10889 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10667) Upgrade Calcite dependency to newest version
[ https://issues.apache.org/jira/browse/HIVE-10667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568387#comment-14568387 ] Ashutosh Chauhan commented on HIVE-10667: - Lets get this in. [~jcamachorodriguez] would you like to rebase the patch and update pom.xml changes. > Upgrade Calcite dependency to newest version > > > Key: HIVE-10667 > URL: https://issues.apache.org/jira/browse/HIVE-10667 > Project: Hive > Issue Type: Task >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-10667.01.patch, HIVE-10667.02.patch, > HIVE-10667.03.patch, HIVE-10667.04.patch, HIVE-10667.patch > > > Upgrade Calcite version to 1.3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[
https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568358#comment-14568358
]
Hive QA commented on HIVE-4239:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736677/HIVE-4239.04.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 8997 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4135/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4135/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4135/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736677 - PreCommit-HIVE-TRUNK-Build
> Remove lock on compilation stage
>
>
> Key: HIVE-4239
> URL: https://issues.apache.org/jira/browse/HIVE-4239
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Processor
>Reporter: Carl Steinbach
>Assignee: Sergey Shelukhin
> Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch,
> HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10728) deprecate unix_timestamp(void) and make it deterministic
[ https://issues.apache.org/jira/browse/HIVE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568344#comment-14568344 ] Ashutosh Chauhan commented on HIVE-10728: - +1 > deprecate unix_timestamp(void) and make it deterministic > > > Key: HIVE-10728 > URL: https://issues.apache.org/jira/browse/HIVE-10728 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-10728.01.patch, HIVE-10728.02.patch, > HIVE-10728.03.patch, HIVE-10728.patch > > > We have a proper current_timestamp function that is not evaluated at runtime. > Behavior of unix_timestamp(void) is both surprising, and is preventing some > optimizations on the other overload since the function becomes > non-deterministic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10885) with vectorization enabled join operation involving interval_day_time fails
[
https://issues.apache.org/jira/browse/HIVE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-10885:
Attachment: HIVE-10885.01.patch
1) Needs some better common routines for mapping primitive category to vector
type.
2) Could add INTERVAL_YEAR_MONTH case in Q file, too?
> with vectorization enabled join operation involving interval_day_time fails
> ---
>
> Key: HIVE-10885
> URL: https://issues.apache.org/jira/browse/HIVE-10885
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jagruti Varia
>Assignee: Matt McCline
> Attachments: HIVE-10885.01.patch
>
>
> When vectorization is on, join operation involving interval_day_time type
> throws following error:
> {noformat}
> Status: Failed
> Vertex failed, vertexName=Map 2, vertexId=vertex_1432858236614_0247_1_01,
> diagnostics=[Task failed, taskId=task_1432858236614_0247_1_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:337)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
> ... 14 more
> Caused by: java.lang.RuntimeException: Cannot allocate vector copy row for
> interval_day_time
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorCopyRow.init(VectorCopyRow.java:213)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.initializeOp(VectorMapJoinCommonOperator.java:581)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:362)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:214)
> ... 15 more
> ], TaskAttempt 1 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:337)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concu
[jira] [Commented] (HIVE-10705) Update tests for HIVE-9302 after removing binaries
[ https://issues.apache.org/jira/browse/HIVE-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568321#comment-14568321 ] Ferdinand Xu commented on HIVE-10705: - Sure, I will work on this issue today. > Update tests for HIVE-9302 after removing binaries > -- > > Key: HIVE-10705 > URL: https://issues.apache.org/jira/browse/HIVE-10705 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-4239: --- Attachment: HIVE-4239.05.patch New patch addressing Thejas' feedback > Remove lock on compilation stage > > > Key: HIVE-4239 > URL: https://issues.apache.org/jira/browse/HIVE-4239 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Query Processor >Reporter: Carl Steinbach >Assignee: Sergey Shelukhin > Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, > HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.05.patch, HIVE-4239.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10872) LLAP: make sure tests pass
[ https://issues.apache.org/jira/browse/HIVE-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-10872: Attachment: (was: HIVE-10872.patch) > LLAP: make sure tests pass > -- > > Key: HIVE-10872 > URL: https://issues.apache.org/jira/browse/HIVE-10872 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-10872.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10872) LLAP: make sure tests pass
[ https://issues.apache.org/jira/browse/HIVE-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-10872: Attachment: HIVE-10872.patch Let's see what HiveQA says... > LLAP: make sure tests pass > -- > > Key: HIVE-10872 > URL: https://issues.apache.org/jira/browse/HIVE-10872 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-10872.patch, HIVE-10872.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10884) Enable some beeline tests and turn on HIVE-4239 by default
[ https://issues.apache.org/jira/browse/HIVE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-10884: Attachment: HIVE-10884.02.patch build was broken again... rince, repeat > Enable some beeline tests and turn on HIVE-4239 by default > -- > > Key: HIVE-10884 > URL: https://issues.apache.org/jira/browse/HIVE-10884 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-10884.01.patch, HIVE-10884.02.patch, > HIVE-10884.patch > > > See comments in HIVE-4239. > Beeline tests with parallelism need to be enabled to turn compilation > parallelism on by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10704) Errors in Tez HashTableLoader when estimated table size is 0
[ https://issues.apache.org/jira/browse/HIVE-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568289#comment-14568289 ] Mostafa Mokhtar commented on HIVE-10704: [~apivovarov] Ping. > Errors in Tez HashTableLoader when estimated table size is 0 > > > Key: HIVE-10704 > URL: https://issues.apache.org/jira/browse/HIVE-10704 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: Jason Dere >Assignee: Mostafa Mokhtar > Fix For: 1.2.1 > > Attachments: HIVE-10704.1.patch, HIVE-10704.2.patch, > HIVE-10704.3.patch > > > Couple of issues: > - If the table sizes in MapJoinOperator.getParentDataSizes() are 0 for all > tables, the largest small table selection is wrong and could select the large > table (which results in NPE) > - The memory estimates can either divide-by-zero, or allocate 0 memory if the > table size is 0. Try to come up with a sensible default for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10748) Replace StringBuffer with StringBuilder where possible
[
https://issues.apache.org/jira/browse/HIVE-10748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568290#comment-14568290
]
Alexander Pivovarov commented on HIVE-10748:
Build 4130 has 3 failed tests. I think errors are unrelated to the patch #2.
I checked prev build 4129 - it also has the same 3 failed tests.
So, everything looks good!
> Replace StringBuffer with StringBuilder where possible
> --
>
> Key: HIVE-10748
> URL: https://issues.apache.org/jira/browse/HIVE-10748
> Project: Hive
> Issue Type: Improvement
>Reporter: Alexander Pivovarov
>Assignee: Alexander Pivovarov
>Priority: Minor
> Attachments: HIVE-10748.1.patch, HIVE-10748.1.patch,
> HIVE-10748.2.patch
>
>
> I found 40 places in Hive where "new StringBuffer(" is used.
> "Where possible, it is recommended that StringBuilder be used in preference
> to StringBuffer as it will be faster under most implementations"
> https://docs.oracle.com/javase/7/docs/api/java/lang/StringBuilder.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10479) CBO: Calcite Operator To Hive Operator (Calcite Return Path) Empty tabAlias in columnInfo which triggers PPD
[ https://issues.apache.org/jira/browse/HIVE-10479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568285#comment-14568285 ] Pengcheng Xiong commented on HIVE-10479: The test failures are unrelated. [~ashutoshc], could you please take a look? Thanks. > CBO: Calcite Operator To Hive Operator (Calcite Return Path) Empty tabAlias > in columnInfo which triggers PPD > > > Key: HIVE-10479 > URL: https://issues.apache.org/jira/browse/HIVE-10479 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-10479.01.patch, HIVE-10479.patch > > > in ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java, line 477, > when aliases contains empty string "" and key is an empty string "" too, it > assumes that aliases contains key. This will trigger incorrect PPD. To > reproduce it, apply the HIVE-10455 and run cbo_subq_notin.q. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10728) deprecate unix_timestamp(void) and make it deterministic
[ https://issues.apache.org/jira/browse/HIVE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568268#comment-14568268 ] Sergey Shelukhin commented on HIVE-10728: - unix_timestamp/0 is non-deterministic, so different results are by design. In existing code, results will be different depending on whether task fails or not, for example (retrying will create different timestamp). So it's not a backward incompatible change > deprecate unix_timestamp(void) and make it deterministic > > > Key: HIVE-10728 > URL: https://issues.apache.org/jira/browse/HIVE-10728 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-10728.01.patch, HIVE-10728.02.patch, > HIVE-10728.03.patch, HIVE-10728.patch > > > We have a proper current_timestamp function that is not evaluated at runtime. > Behavior of unix_timestamp(void) is both surprising, and is preventing some > optimizations on the other overload since the function becomes > non-deterministic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10887) TestCliDriver tests ordering issues with Mac and CentOS
[ https://issues.apache.org/jira/browse/HIVE-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568244#comment-14568244 ] Thejas M Nair commented on HIVE-10887: -- +1 > TestCliDriver tests ordering issues with Mac and CentOS > --- > > Key: HIVE-10887 > URL: https://issues.apache.org/jira/browse/HIVE-10887 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-10887.1.patch > > > unionDistinct_2 and update_after_multiple_inserts tests give different > results in different environments due to ordering issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-10886) LLAP: Fixes to TaskReporter after recent Tez changes
[ https://issues.apache.org/jira/browse/HIVE-10886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved HIVE-10886. --- Resolution: Fixed Committed to llap branch. > LLAP: Fixes to TaskReporter after recent Tez changes > > > Key: HIVE-10886 > URL: https://issues.apache.org/jira/browse/HIVE-10886 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-10886.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10887) TestCliDriver tests ordering issues with Mac and CentOS
[ https://issues.apache.org/jira/browse/HIVE-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-10887: - Attachment: HIVE-10887.1.patch cc-ing [~thejas] for review Thanks Hari > TestCliDriver tests ordering issues with Mac and CentOS > --- > > Key: HIVE-10887 > URL: https://issues.apache.org/jira/browse/HIVE-10887 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-10887.1.patch > > > unionDistinct_2 and update_after_multiple_inserts tests give different > results in different environments due to ordering issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10886) LLAP: Fixes to TaskReporter after recent Tez changes
[ https://issues.apache.org/jira/browse/HIVE-10886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-10886: -- Attachment: HIVE-10886.1.txt > LLAP: Fixes to TaskReporter after recent Tez changes > > > Key: HIVE-10886 > URL: https://issues.apache.org/jira/browse/HIVE-10886 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-10886.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10884) Enable some beeline tests and turn on HIVE-4239 by default
[
https://issues.apache.org/jira/browse/HIVE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568235#comment-14568235
]
Hive QA commented on HIVE-10884:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736678/HIVE-10884.01.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4134/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4134/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4134/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/itests/hcatalog-unit/target/hive-hcatalog-it-unit-1.3.0-SNAPSHOT-tests.jar
to
/home/hiveptest/.m2/repository/org/apache/hive/hive-hcatalog-it-unit/1.3.0-SNAPSHOT/hive-hcatalog-it-unit-1.3.0-SNAPSHOT-tests.jar
[INFO]
[INFO]
[INFO] Building Hive Integration - Testing Utilities 1.3.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-it-util ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/itests/util/target
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/itests/util (includes =
[datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-it-util ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (download-spark) @ hive-it-util ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-it-util
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hive-it-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/itests/util/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-it-util ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-it-util
---
[INFO] Compiling 50 source files to
/data/hive-ptest/working/apache-github-source-source/itests/util/target/classes
[WARNING]
/data/hive-ptest/working/apache-github-source-source/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java:
Some input files use or override a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hive-it-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/itests/util/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-it-util ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/itests/util/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/itests/util/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/itests/util/target/tmp/conf
[copy] Copying 11 files to
/data/hive-ptest/working/apache-github-source-source/itests/util/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-it-util ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-it-util ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-it-util ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/itests/util/target/hive-it-util-1.3.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-it-util ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-it-util ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/itests/util/target/hive-it-util-1.3.0-SNAPSHOT.jar
to
/home/hiveptest/.m2/repository/org/apache/hive/hive-it-util/1.3.0-SNAPSHOT/hive-it-util-1.3.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/itests/util/pom.xml to
/home/hiveptest/.m2/repository/org/apache/hive/hive-it-util/
[jira] [Commented] (HIVE-10877) TestUtil class name confuses ptest2
[ https://issues.apache.org/jira/browse/HIVE-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568232#comment-14568232 ] Thejas M Nair commented on HIVE-10877: -- Sorry about that! Yes, I forgot to do a 'git add'. i have now added it! > TestUtil class name confuses ptest2 > --- > > Key: HIVE-10877 > URL: https://issues.apache.org/jira/browse/HIVE-10877 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 1.3.0 > > Attachments: HIVE-10877.1.patch > > > TestUtil.java added in HIVE-10862 seems to confuse ptest2, and it complains > "TestUtil - did not produce a TEST-*.xml file" > That file should be renamed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10728) deprecate unix_timestamp(void) and make it deterministic
[ https://issues.apache.org/jira/browse/HIVE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568230#comment-14568230 ] Alan Gates commented on HIVE-10728: --- bq. At the very least, this seems to fall into the category of something you want to commit on trunk and not on branch-1. +1, as this is a definite backwards incompatible change. > deprecate unix_timestamp(void) and make it deterministic > > > Key: HIVE-10728 > URL: https://issues.apache.org/jira/browse/HIVE-10728 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-10728.01.patch, HIVE-10728.02.patch, > HIVE-10728.03.patch, HIVE-10728.patch > > > We have a proper current_timestamp function that is not evaluated at runtime. > Behavior of unix_timestamp(void) is both surprising, and is preventing some > optimizations on the other overload since the function becomes > non-deterministic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-10728) deprecate unix_timestamp(void) and make it deterministic
[
https://issues.apache.org/jira/browse/HIVE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568227#comment-14568227
]
Ashutosh Chauhan edited comment on HIVE-10728 at 6/1/15 11:39 PM:
--
This is change in behavior of existing functionality. e.g, {{where c1 >
unix_timestamp()}} will give you different resultset depending on values of c1,
before and after this patch on same table.
I want to hear other committer's comment whether this is kosher to do this or
not. [~alangates] What do you think?
At the very least, this seems to fall into the category of something you want
to commit on trunk and not on branch-1.
was (Author: ashutoshc):
This is change in behavior of existing functionality. e.g, {{where c1 >
unix_timestamp() }} will give you different resultset depending on values of
c1, before and after this patch on same table.
I want to hear other committer's comment whether this is kosher to do this or
not. [~alangates] What do you think?
At the very least, this seems to fall into the category of something you want
to commit on trunk and not on branch-1.
> deprecate unix_timestamp(void) and make it deterministic
>
>
> Key: HIVE-10728
> URL: https://issues.apache.org/jira/browse/HIVE-10728
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-10728.01.patch, HIVE-10728.02.patch,
> HIVE-10728.03.patch, HIVE-10728.patch
>
>
> We have a proper current_timestamp function that is not evaluated at runtime.
> Behavior of unix_timestamp(void) is both surprising, and is preventing some
> optimizations on the other overload since the function becomes
> non-deterministic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10728) deprecate unix_timestamp(void) and make it deterministic
[
https://issues.apache.org/jira/browse/HIVE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568227#comment-14568227
]
Ashutosh Chauhan commented on HIVE-10728:
-
This is change in behavior of existing functionality. e.g, {{where c1 >
unix_timestamp() }} will give you different resultset depending on values of
c1, before and after this patch on same table.
I want to hear other committer's comment whether this is kosher to do this or
not. [~alangates] What do you think?
At the very least, this seems to fall into the category of something you want
to commit on trunk and not on branch-1.
> deprecate unix_timestamp(void) and make it deterministic
>
>
> Key: HIVE-10728
> URL: https://issues.apache.org/jira/browse/HIVE-10728
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-10728.01.patch, HIVE-10728.02.patch,
> HIVE-10728.03.patch, HIVE-10728.patch
>
>
> We have a proper current_timestamp function that is not evaluated at runtime.
> Behavior of unix_timestamp(void) is both surprising, and is preventing some
> optimizations on the other overload since the function becomes
> non-deterministic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10874) Fail in TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2.q due to duplicate column name
[
https://issues.apache.org/jira/browse/HIVE-10874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568220#comment-14568220
]
Jesus Camacho Rodriguez commented on HIVE-10874:
[~jpullokkaran], this problem is not only in Hive, the patch should go into
Calcite too, and once the next release is out, we could remove it from here.
In this case, the condition is risen because we have the following plan:
{noformat}
Aggregate (f_1, sum(f_1))
Union
Aggregate (x, sum(x)) ...
Aggregate (x, sum(x)) ...
{noformat}
where f1 is the column with the result of sum(x).
The problem is that Calcite derives the row schema for the aggregation column
sum(f1) automatically. The generated name is f_1 ('f' of function, 1 of the
position in the tuple), which is the same one that the first column has;
however, Calcite was not verifying if the autogenerated name was already in the
tuple or not. This patch checks if the name already exists, and while it does,
it generates a new column name.
> Fail in TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2.q due to
> duplicate column name
> -
>
> Key: HIVE-10874
> URL: https://issues.apache.org/jira/browse/HIVE-10874
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-10874.patch
>
>
> Aggregate operators may derive row types with duplicate column names. The
> reason is that the column names for grouping sets columns and aggregation
> columns might be generated automatically, but we do not check whether the
> column name already exists in the same row.
> This error can be reproduced by
> TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2.q, which fails
> with the following trace:
> {code}
> junit.framework.AssertionFailedError: Unexpected exception
> java.lang.AssertionError: RecordType(BIGINT $f1, BIGINT $f1)
> at org.apache.calcite.rel.core.Project.isValid(Project.java:200)
> at org.apache.calcite.rel.core.Project.(Project.java:85)
> at org.apache.calcite.rel.core.Project.(Project.java:91)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.(HiveProject.java:70)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.create(HiveProject.java:103)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.PlanModifierForASTConv.introduceDerivedTable(PlanModifierForASTConv.java:211)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.PlanModifierForASTConv.convertOpTree(PlanModifierForASTConv.java:67)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.convert(ASTConverter.java:94)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:617)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:248)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10108)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10705) Update tests for HIVE-9302 after removing binaries
[ https://issues.apache.org/jira/browse/HIVE-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568215#comment-14568215 ] Vaibhav Gumashta commented on HIVE-10705: - Hi [~Ferd], was wondering if you plan to work on this. It'll be good to put the fix in 1.2.1 - the absence of the 2 jars (DummyDriver-1.0-SNAPSHOT.jar, postgresql-9.3.jdbc3.jar) raises a false alarm in TestBeelineArgParsing. > Update tests for HIVE-9302 after removing binaries > -- > > Key: HIVE-10705 > URL: https://issues.apache.org/jira/browse/HIVE-10705 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-10425) LLAP: Limit number of threads used to communicate with a single LLAP instance to 1
[ https://issues.apache.org/jira/browse/HIVE-10425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved HIVE-10425. --- Resolution: Fixed Committed to the llap branch. > LLAP: Limit number of threads used to communicate with a single LLAP instance > to 1 > -- > > Key: HIVE-10425 > URL: https://issues.apache.org/jira/browse/HIVE-10425 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-10425.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10877) TestUtil class name confuses ptest2
[
https://issues.apache.org/jira/browse/HIVE-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568208#comment-14568208
]
Jimmy Xiang commented on HIVE-10877:
[~thejas], I got compilation error. Forgot to add UtilsForTest.java?
{noformat}
1:testCompile (default-testCompile) on project hive-it-unit: Compilation
failure: Compilation failure:
[ERROR]
/home/jxiang/git-repos/apache/hive/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/authorization/plugin/TestHiveAuthorizerShowFilters.java:[30,30]
cannot find symbol
[ERROR] symbol: class UtilsForTest
{noformat}
> TestUtil class name confuses ptest2
> ---
>
> Key: HIVE-10877
> URL: https://issues.apache.org/jira/browse/HIVE-10877
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 1.3.0
>
> Attachments: HIVE-10877.1.patch
>
>
> TestUtil.java added in HIVE-10862 seems to confuse ptest2, and it complains
> "TestUtil - did not produce a TEST-*.xml file"
> That file should be renamed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10884) Enable some beeline tests and turn on HIVE-4239 by default
[ https://issues.apache.org/jira/browse/HIVE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-10884: Attachment: HIVE-10884.01.patch same exact patch, HiveQA failed for unrelated reasons > Enable some beeline tests and turn on HIVE-4239 by default > -- > > Key: HIVE-10884 > URL: https://issues.apache.org/jira/browse/HIVE-10884 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-10884.01.patch, HIVE-10884.patch > > > See comments in HIVE-4239. > Beeline tests with parallelism need to be enabled to turn compilation > parallelism on by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-4239: --- Attachment: HIVE-4239.04.patch re-uploading since the build is fixed > Remove lock on compilation stage > > > Key: HIVE-4239 > URL: https://issues.apache.org/jira/browse/HIVE-4239 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Query Processor >Reporter: Carl Steinbach >Assignee: Sergey Shelukhin > Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, > HIVE-4239.03.patch, HIVE-4239.04.patch, HIVE-4239.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10302) Load small tables (for map join) in executor memory only once [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568202#comment-14568202 ] Sergey Shelukhin commented on HIVE-10302: - Actually I wonder why you guys still work on routine jiras on the branch after the main merge. Usually branch is reserved for major feature and abandoned after merge, unless there's some other major feature with epic merge... > Load small tables (for map join) in executor memory only once [Spark Branch] > > > Key: HIVE-10302 > URL: https://issues.apache.org/jira/browse/HIVE-10302 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 1.3.0 > > Attachments: 10302.patch, HIVE-10302.2-spark.patch, > HIVE-10302.3-spark.patch, HIVE-10302.4-spark.patch, HIVE-10302.spark-1.patch > > > Usually there are multiple cores in a Spark executor, and thus it's possible > that multiple map-join tasks can be running in the same executor > (concurrently or sequentially). Currently, each task will load its own copy > of the small tables for map join into memory, ending up with inefficiency. > Ideally, we only load the small tables once and share them among the tasks > running in that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10875) Select query with view in subquery adds underlying table as direct input
[
https://issues.apache.org/jira/browse/HIVE-10875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568199#comment-14568199
]
Prasad Mujumdar commented on HIVE-10875:
[~thejas] Thanks for catching the issue and patch. Looks fine to me.
> Select query with view in subquery adds underlying table as direct input
>
>
> Key: HIVE-10875
> URL: https://issues.apache.org/jira/browse/HIVE-10875
> Project: Hive
> Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 1.2.1
>
> Attachments: HIVE-10875.1.patch, HIVE-10875.2.patch
>
>
> In the following case,
> {code}
> create view V as select * from T;
> select * from (select * from V) A;
> {code}
> The semantic analyzer inputs contain input table T as a direct input instead
> of adding it as an indirect input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10302) Load small tables (for map join) in executor memory only once [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568197#comment-14568197 ] Jimmy Xiang commented on HIVE-10302: The file was lost during rebasing. I pushed it to master. The build is ok for me now. Thanks. > Load small tables (for map join) in executor memory only once [Spark Branch] > > > Key: HIVE-10302 > URL: https://issues.apache.org/jira/browse/HIVE-10302 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 1.3.0 > > Attachments: 10302.patch, HIVE-10302.2-spark.patch, > HIVE-10302.3-spark.patch, HIVE-10302.4-spark.patch, HIVE-10302.spark-1.patch > > > Usually there are multiple cores in a Spark executor, and thus it's possible > that multiple map-join tasks can be running in the same executor > (concurrently or sequentially). Currently, each task will load its own copy > of the small tables for map join into memory, ending up with inefficiency. > Ideally, we only load the small tables once and share them among the tasks > running in that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-1538) FilterOperator is applied twice with ppd on.
[ https://issues.apache.org/jira/browse/HIVE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568180#comment-14568180 ] Lefty Leverenz commented on HIVE-1538: -- Doc note: This added *hive.ppd.remove.duplicatefilters* to HiveConf.java. It needs to be documented in the wiki. * [Configuration Properties | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties] It also needs an appropriate description, as [~apivovarov] pointed out in a message to the dev@hive mailing list: * ["hive.ppd.remove.duplicatefilters description is incorrect. What is the correct one?" | http://mail-archives.apache.org/mod_mbox/hive-dev/201506.mbox/%3ccakkt98q-cordbeqd+9bbsy1+t9b60szwa7xfgsyqpsp1mrs...@mail.gmail.com%3e] > FilterOperator is applied twice with ppd on. > > > Key: HIVE-1538 > URL: https://issues.apache.org/jira/browse/HIVE-1538 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.8.0 > > Attachments: patch-1538-1.txt, patch-1538-2.txt, patch-1538-3.txt, > patch-1538-4.txt, patch-1538.txt > > > With hive.optimize.ppd set to true, FilterOperator is applied twice. And it > seems second operator is always filtering zero rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10728) deprecate unix_timestamp(void) and make it deterministic
[ https://issues.apache.org/jira/browse/HIVE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568173#comment-14568173 ] Sergey Shelukhin commented on HIVE-10728: - [~ashutoshc] ping? > deprecate unix_timestamp(void) and make it deterministic > > > Key: HIVE-10728 > URL: https://issues.apache.org/jira/browse/HIVE-10728 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-10728.01.patch, HIVE-10728.02.patch, > HIVE-10728.03.patch, HIVE-10728.patch > > > We have a proper current_timestamp function that is not evaluated at runtime. > Behavior of unix_timestamp(void) is both surprising, and is preventing some > optimizations on the other overload since the function becomes > non-deterministic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568164#comment-14568164 ] Sergey Shelukhin edited comment on HIVE-4239 at 6/1/15 10:46 PM: - This failure is unrelated, build is broken by HIVE-10302 it seems was (Author: sershe): This failure is unrelated > Remove lock on compilation stage > > > Key: HIVE-4239 > URL: https://issues.apache.org/jira/browse/HIVE-4239 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Query Processor >Reporter: Carl Steinbach >Assignee: Sergey Shelukhin > Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, > HIVE-4239.03.patch, HIVE-4239.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10425) LLAP: Limit number of threads used to communicate with a single LLAP instance to 1
[ https://issues.apache.org/jira/browse/HIVE-10425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-10425: -- Attachment: HIVE-10425.1.txt > LLAP: Limit number of threads used to communicate with a single LLAP instance > to 1 > -- > > Key: HIVE-10425 > URL: https://issues.apache.org/jira/browse/HIVE-10425 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-10425.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10425) LLAP: Limit number of threads used to communicate with a single LLAP instance to 1
[ https://issues.apache.org/jira/browse/HIVE-10425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-10425: -- Summary: LLAP: Limit number of threads used to communicate with a single LLAP instance to 1 (was: LLAP: Control number of threads used to communicate with a single LLAP instance) > LLAP: Limit number of threads used to communicate with a single LLAP instance > to 1 > -- > > Key: HIVE-10425 > URL: https://issues.apache.org/jira/browse/HIVE-10425 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-10425.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10302) Load small tables (for map join) in executor memory only once [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-10302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568170#comment-14568170
]
Sergey Shelukhin commented on HIVE-10302:
-
Hi. This appears to have broken the build:
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hive-exec: Compilation failure: Compilation failure:
[ERROR]
/Users/sergey/git/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java:[51,7]
cannot find symbol
[ERROR] symbol: variable SmallTableCache
[ERROR] location: class
org.apache.hadoop.hive.ql.exec.spark.HivePairFlatMapFunction
[ERROR]
/Users/sergey/git/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java:[129,42]
cannot find symbol
[ERROR] symbol: variable SmallTableCache
[ERROR] location: class org.apache.hadoop.hive.ql.exec.spark.HashTableLoader
[ERROR]
/Users/sergey/git/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java:[132,24]
cannot find symbol
[ERROR] symbol: variable SmallTableCache
[ERROR] location: class org.apache.hadoop.hive.ql.exec.spark.HashTableLoader
[ERROR]
/Users/sergey/git/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java:[135,11]
cannot find symbol
[ERROR] symbol: variable SmallTableCache
[ERROR] location: class org.apache.hadoop.hive.ql.exec.spark.HashTableLoader
[ERROR] -> [Help 1]
{noformat}.
Can you please revert or fix?
> Load small tables (for map join) in executor memory only once [Spark Branch]
>
>
> Key: HIVE-10302
> URL: https://issues.apache.org/jira/browse/HIVE-10302
> Project: Hive
> Issue Type: Improvement
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 1.3.0
>
> Attachments: 10302.patch, HIVE-10302.2-spark.patch,
> HIVE-10302.3-spark.patch, HIVE-10302.4-spark.patch, HIVE-10302.spark-1.patch
>
>
> Usually there are multiple cores in a Spark executor, and thus it's possible
> that multiple map-join tasks can be running in the same executor
> (concurrently or sequentially). Currently, each task will load its own copy
> of the small tables for map join into memory, ending up with inefficiency.
> Ideally, we only load the small tables once and share them among the tasks
> running in that executor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568164#comment-14568164 ] Sergey Shelukhin commented on HIVE-4239: This failure is unrelated > Remove lock on compilation stage > > > Key: HIVE-4239 > URL: https://issues.apache.org/jira/browse/HIVE-4239 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Query Processor >Reporter: Carl Steinbach >Assignee: Sergey Shelukhin > Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, > HIVE-4239.03.patch, HIVE-4239.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10884) Enable some beeline tests and turn on HIVE-4239 by default
[
https://issues.apache.org/jira/browse/HIVE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568123#comment-14568123
]
Hive QA commented on HIVE-10884:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736655/HIVE-10884.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4133/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4133/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4133/
Messages:
{noformat}
This message was trimmed, see log for full details
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf
[copy] Copying 11 files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
spark-client ---
[INFO] Compiling 5 source files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/test-classes
[INFO]
[INFO] --- maven-dependency-plugin:2.8:copy (copy-guava-14) @ spark-client ---
[INFO] Configured Artifact: com.google.guava:guava:14.0.1:jar
[INFO] Copying guava-14.0.1.jar to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/dependency/guava-14.0.1.jar
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ spark-client ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ spark-client ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-1.3.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
spark-client ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ spark-client ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-1.3.0-SNAPSHOT.jar
to
/home/hiveptest/.m2/repository/org/apache/hive/spark-client/1.3.0-SNAPSHOT/spark-client-1.3.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/spark-client/pom.xml to
/home/hiveptest/.m2/repository/org/apache/hive/spark-client/1.3.0-SNAPSHOT/spark-client-1.3.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Query Language 1.3.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-exec ---
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-exec ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (generate-sources) @ hive-exec ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-test-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen
Generating vector expression code
Generating vector expression test code
[INFO] Executed tasks
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-exec ---
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/src/gen/protobuf/gen-java
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/src/gen/thrift/gen-javabean
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java
added.
[INFO]
[INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-exec ---
[INFO] ANTLR: Processing source directory
/data/hive-ptest/working/apache-github-source-source/ql/src/java
ANTLR Parser Generator Version 3.4
org/apache/hadoop/hive/ql/parse/HiveLexer.g
org/apache/hadoop/hive/ql/parse/HiveParser.g
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_ORDER KW_BY" using
multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that
[jira] [Commented] (HIVE-10884) Enable some beeline tests and turn on HIVE-4239 by default
[
https://issues.apache.org/jira/browse/HIVE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568104#comment-14568104
]
Sergey Shelukhin commented on HIVE-10884:
-
Ok, for me tests get stuck after this:
{noformat}
2015-06-01 14:56:19,500 ERROR [HiveServer2-Handler-Pool: Thread-45]:
server.NIOServerCnxnFactory (NIOServerCnxnFactory.java:uncaughtException(44)) -
Thread Thread[HiveServer2-Handler-Pool: Thread-45,5,main] died
java.lang.NoClassDefFoundError:
org/apache/hive/service/cli/thrift/TCLIService$CloseSession_result$CloseSession_resultStandardScheme
at
org.apache.hive.service.cli.thrift.TCLIService$CloseSession_result$CloseSession_resultStandardSchemeFactory.getScheme(TCLIService.java:2988)
at
org.apache.hive.service.cli.thrift.TCLIService$CloseSession_result$CloseSession_resultStandardSchemeFactory.getScheme(TCLIService.java:2986)
at
org.apache.hive.service.cli.thrift.TCLIService$CloseSession_result.write(TCLIService.java:2943)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:53)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
I have tried cleaning maven repo, regenerating thrift (no significant changes).
In debugger, the factory on the stack is executing from correct jar. If I unjar
that, I can also find the offending class:
{noformat}
$ jar -xfv
/Users/sergey/.m2/repository/org/apache/hive/hive-service/1.3.0-SNAPSHOT/hive-service-1.3.0-SNAPSHOT.jar
| grep CloseSession.*resultStandardScheme
Picked up JAVA_TOOL_OPTIONS: -Djava.awt.headless=true
inflated:
org/apache/hive/service/cli/thrift/TCLIService$CloseSession_result$CloseSession_resultStandardScheme.class
inflated:
org/apache/hive/service/cli/thrift/TCLIService$CloseSession_result$CloseSession_resultStandardSchemeFactory.class
{noformat}
However, when I try to load this class from Factory class classloader in
debugger, I also get class not found (factory class classloader can find the
factory class itself).
Not really sure how this can happen.
[~cwsteinbach] do you have any background on this?
[~xuefuz] I am assuming you guys will need to enable beeline tests as part of
beeline-by-default work... have you run into this issue?
> Enable some beeline tests and turn on HIVE-4239 by default
> --
>
> Key: HIVE-10884
> URL: https://issues.apache.org/jira/browse/HIVE-10884
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-10884.patch
>
>
> See comments in HIVE-4239.
> Beeline tests with parallelism need to be enabled to turn compilation
> parallelism on by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9370) SparkJobMonitor timeout as sortByKey would launch extra Spark job before original job get submitted [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568096#comment-14568096 ] Lefty Leverenz commented on HIVE-9370: -- Okay, thanks Chengxiang. > SparkJobMonitor timeout as sortByKey would launch extra Spark job before > original job get submitted [Spark Branch] > -- > > Key: HIVE-9370 > URL: https://issues.apache.org/jira/browse/HIVE-9370 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: yuyun.chen >Assignee: Chengxiang Li > Fix For: 1.1.0 > > Attachments: HIVE-9370.1-spark.patch > > > enable hive on spark and run BigBench Query 8 then got the following > exception: > 2015-01-14 11:43:46,057 INFO [main]: impl.RemoteSparkJobStatus > (RemoteSparkJobStatus.java:getSparkJobInfo(143)) - Job hasn't been submitted > after 30s. Aborting it. > 2015-01-14 11:43:46,061 INFO [main]: impl.RemoteSparkJobStatus > (RemoteSparkJobStatus.java:getSparkJobInfo(143)) - Job hasn't been submitted > after 30s. Aborting it. > 2015-01-14 11:43:46,061 ERROR [main]: status.SparkJobMonitor > (SessionState.java:printError(839)) - Status: Failed > 2015-01-14 11:43:46,062 INFO [main]: log.PerfLogger > (PerfLogger.java:PerfLogEnd(148)) - start=1421206996052 end=1421207026062 duration=30010 > from=org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor> > 2015-01-14 11:43:46,071 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) - 15/01/14 11:43:46 INFO RemoteDriver: Failed > to run job 0a9a7782-0e0b-4561-8468-959a6d8df0a3 > 2015-01-14 11:43:46,071 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) - java.lang.InterruptedException > 2015-01-14 11:43:46,071 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at java.lang.Object.wait(Native > Method) > 2015-01-14 11:43:46,071 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > java.lang.Object.wait(Object.java:503) > 2015-01-14 11:43:46,071 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73) > 2015-01-14 11:43:46,071 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:514) > 2015-01-14 11:43:46,071 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > org.apache.spark.SparkContext.runJob(SparkContext.scala:1282) > 2015-01-14 11:43:46,072 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > org.apache.spark.SparkContext.runJob(SparkContext.scala:1300) > 2015-01-14 11:43:46,072 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > org.apache.spark.SparkContext.runJob(SparkContext.scala:1314) > 2015-01-14 11:43:46,072 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > org.apache.spark.SparkContext.runJob(SparkContext.scala:1328) > 2015-01-14 11:43:46,072 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > org.apache.spark.rdd.RDD.collect(RDD.scala:780) > 2015-01-14 11:43:46,072 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > org.apache.spark.RangePartitioner$.sketch(Partitioner.scala:262) > 2015-01-14 11:43:46,072 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > org.apache.spark.RangePartitioner.(Partitioner.scala:124) > 2015-01-14 11:43:46,072 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > org.apache.spark.rdd.OrderedRDDFunctions.sortByKey(OrderedRDDFunctions.scala:63) > 2015-01-14 11:43:46,073 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > org.apache.spark.api.java.JavaPairRDD.sortByKey(JavaPairRDD.scala:894) > 2015-01-14 11:43:46,073 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > org.apache.spark.api.java.JavaPairRDD.sortByKey(JavaPairRDD.scala:864) > 2015-01-14 11:43:46,073 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > org.apache.hadoop.hive.ql.exec.spark.SortByShuffler.shuffle(SortByShuffler.java:48) > 2015-01-14 11:43:46,073 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(436)) -at > org.apache.hadoop.hive.ql.exec.spark.ShuffleTran.transform(ShuffleTran.java:45) > 2015-01-14 11:43:46,073 INFO
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[
https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568088#comment-14568088
]
Hive QA commented on HIVE-4239:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736649/HIVE-4239.03.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4132/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4132/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4132/
Messages:
{noformat}
This message was trimmed, see log for full details
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf
[copy] Copying 11 files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
spark-client ---
[INFO] Compiling 5 source files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/test-classes
[INFO]
[INFO] --- maven-dependency-plugin:2.8:copy (copy-guava-14) @ spark-client ---
[INFO] Configured Artifact: com.google.guava:guava:14.0.1:jar
[INFO] Copying guava-14.0.1.jar to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/dependency/guava-14.0.1.jar
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ spark-client ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ spark-client ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-1.3.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
spark-client ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ spark-client ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-1.3.0-SNAPSHOT.jar
to
/home/hiveptest/.m2/repository/org/apache/hive/spark-client/1.3.0-SNAPSHOT/spark-client-1.3.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/spark-client/pom.xml to
/home/hiveptest/.m2/repository/org/apache/hive/spark-client/1.3.0-SNAPSHOT/spark-client-1.3.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Query Language 1.3.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-exec ---
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-exec ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (generate-sources) @ hive-exec ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-test-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen
Generating vector expression code
Generating vector expression test code
[INFO] Executed tasks
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-exec ---
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/src/gen/protobuf/gen-java
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/src/gen/thrift/gen-javabean
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java
added.
[INFO]
[INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-exec ---
[INFO] ANTLR: Processing source directory
/data/hive-ptest/working/apache-github-source-source/ql/src/java
ANTLR Parser Generator Version 3.4
org/apache/hadoop/hive/ql/parse/HiveLexer.g
org/apache/hadoop/hive/ql/parse/HiveParser.g
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_ORDER KW_BY" using
multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that
[jira] [Commented] (HIVE-10851) HCatalog NullPointerException when using views in Pig
[
https://issues.apache.org/jira/browse/HIVE-10851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568082#comment-14568082
]
Hive QA commented on HIVE-10851:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736641/HIVE-10851.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4131/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4131/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4131/
Messages:
{noformat}
This message was trimmed, see log for full details
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf
[copy] Copying 11 files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
spark-client ---
[INFO] Compiling 5 source files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/test-classes
[INFO]
[INFO] --- maven-dependency-plugin:2.8:copy (copy-guava-14) @ spark-client ---
[INFO] Configured Artifact: com.google.guava:guava:14.0.1:jar
[INFO] Copying guava-14.0.1.jar to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/dependency/guava-14.0.1.jar
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ spark-client ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ spark-client ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-1.3.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
spark-client ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ spark-client ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-1.3.0-SNAPSHOT.jar
to
/home/hiveptest/.m2/repository/org/apache/hive/spark-client/1.3.0-SNAPSHOT/spark-client-1.3.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/spark-client/pom.xml to
/home/hiveptest/.m2/repository/org/apache/hive/spark-client/1.3.0-SNAPSHOT/spark-client-1.3.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Query Language 1.3.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-exec ---
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-exec ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (generate-sources) @ hive-exec ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-test-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen
Generating vector expression code
Generating vector expression test code
[INFO] Executed tasks
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-exec ---
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/src/gen/protobuf/gen-java
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/src/gen/thrift/gen-javabean
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java
added.
[INFO]
[INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-exec ---
[INFO] ANTLR: Processing source directory
/data/hive-ptest/working/apache-github-source-source/ql/src/java
ANTLR Parser Generator Version 3.4
org/apache/hadoop/hive/ql/parse/HiveLexer.g
org/apache/hadoop/hive/ql/parse/HiveParser.g
warning(200): IdentifiersParser.g:455:5:
Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_ORDER KW_BY" using
multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that
[jira] [Commented] (HIVE-10877) TestUtil class name confuses ptest2
[ https://issues.apache.org/jira/browse/HIVE-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568070#comment-14568070 ] Thejas M Nair commented on HIVE-10877: -- I will commit this soon, this is adding to the noise in test results. > TestUtil class name confuses ptest2 > --- > > Key: HIVE-10877 > URL: https://issues.apache.org/jira/browse/HIVE-10877 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-10877.1.patch > > > TestUtil.java added in HIVE-10862 seems to confuse ptest2, and it complains > "TestUtil - did not produce a TEST-*.xml file" > That file should be renamed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10877) TestUtil class name confuses ptest2
[ https://issues.apache.org/jira/browse/HIVE-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568066#comment-14568066 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-10877: -- +1 > TestUtil class name confuses ptest2 > --- > > Key: HIVE-10877 > URL: https://issues.apache.org/jira/browse/HIVE-10877 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-10877.1.patch > > > TestUtil.java added in HIVE-10862 seems to confuse ptest2, and it complains > "TestUtil - did not produce a TEST-*.xml file" > That file should be renamed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10877) TestUtil class name confuses ptest2
[ https://issues.apache.org/jira/browse/HIVE-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-10877: - Description: TestUtil.java added in HIVE-10862 seems to confuse ptest2, and it complains "TestUtil - did not produce a TEST-*.xml file" That file should be renamed. was: TestUtil.java added in HIVE-10862 seems to confuse ptest2, and it complains that no TEST-*.xml file was produced for it. That file should be renamed. > TestUtil class name confuses ptest2 > --- > > Key: HIVE-10877 > URL: https://issues.apache.org/jira/browse/HIVE-10877 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-10877.1.patch > > > TestUtil.java added in HIVE-10862 seems to confuse ptest2, and it complains > "TestUtil - did not produce a TEST-*.xml file" > That file should be renamed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10748) Replace StringBuffer with StringBuilder where possible
[
https://issues.apache.org/jira/browse/HIVE-10748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568064#comment-14568064
]
Hive QA commented on HIVE-10748:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736630/HIVE-10748.2.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 8994 tests executed
*Failed tests:*
{noformat}
TestUtil - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4130/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4130/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4130/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736630 - PreCommit-HIVE-TRUNK-Build
> Replace StringBuffer with StringBuilder where possible
> --
>
> Key: HIVE-10748
> URL: https://issues.apache.org/jira/browse/HIVE-10748
> Project: Hive
> Issue Type: Improvement
>Reporter: Alexander Pivovarov
>Assignee: Alexander Pivovarov
>Priority: Minor
> Attachments: HIVE-10748.1.patch, HIVE-10748.1.patch,
> HIVE-10748.2.patch
>
>
> I found 40 places in Hive where "new StringBuffer(" is used.
> "Where possible, it is recommended that StringBuilder be used in preference
> to StringBuffer as it will be faster under most implementations"
> https://docs.oracle.com/javase/7/docs/api/java/lang/StringBuilder.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10874) Fail in TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2.q due to duplicate column name
[
https://issues.apache.org/jira/browse/HIVE-10874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568060#comment-14568060
]
Laljo John Pullokkaran commented on HIVE-10874:
---
[~jcamachorodriguez] Could you explain why are we deriving row type differently?
Is it to accommodate the groupings col differences between calcite & hive?
> Fail in TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2.q due to
> duplicate column name
> -
>
> Key: HIVE-10874
> URL: https://issues.apache.org/jira/browse/HIVE-10874
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-10874.patch
>
>
> Aggregate operators may derive row types with duplicate column names. The
> reason is that the column names for grouping sets columns and aggregation
> columns might be generated automatically, but we do not check whether the
> column name already exists in the same row.
> This error can be reproduced by
> TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2.q, which fails
> with the following trace:
> {code}
> junit.framework.AssertionFailedError: Unexpected exception
> java.lang.AssertionError: RecordType(BIGINT $f1, BIGINT $f1)
> at org.apache.calcite.rel.core.Project.isValid(Project.java:200)
> at org.apache.calcite.rel.core.Project.(Project.java:85)
> at org.apache.calcite.rel.core.Project.(Project.java:91)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.(HiveProject.java:70)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.create(HiveProject.java:103)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.PlanModifierForASTConv.introduceDerivedTable(PlanModifierForASTConv.java:211)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.PlanModifierForASTConv.convertOpTree(PlanModifierForASTConv.java:67)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.convert(ASTConverter.java:94)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:617)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:248)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10108)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10841) [WHERE col is not null] does not work sometimes for queries with many JOIN statements
[
https://issues.apache.org/jira/browse/HIVE-10841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568044#comment-14568044
]
Alexander Pivovarov commented on HIVE-10841:
Changing the order of JOIN operators fixes the plan.
Filter Operator predicates are added to Map step for "acct" table.
> [WHERE col is not null] does not work sometimes for queries with many JOIN
> statements
> -
>
> Key: HIVE-10841
> URL: https://issues.apache.org/jira/browse/HIVE-10841
> Project: Hive
> Issue Type: Bug
> Components: Query Planning, Query Processor
>Affects Versions: 0.13.0, 0.14.0, 0.13.1, 1.2.0
>Reporter: Alexander Pivovarov
>Assignee: Alexander Pivovarov
>
> The result from the following SELECT query is 3 rows but it should be 1 row.
> I checked it in MySQL - it returned 1 row.
> To reproduce the issue in Hive
> 1. prepare tables
> {code}
> drop table if exists L;
> drop table if exists LA;
> drop table if exists FR;
> drop table if exists A;
> drop table if exists PI;
> drop table if exists acct;
> create table L as select 4436 id;
> create table LA as select 4436 loan_id, 4748 aid, 4415 pi_id;
> create table FR as select 4436 loan_id;
> create table A as select 4748 id;
> create table PI as select 4415 id;
> create table acct as select 4748 aid, 10 acc_n, 122 brn;
> insert into table acct values(4748, null, null);
> insert into table acct values(4748, null, null);
> {code}
> 2. run SELECT query
> {code}
> select
> acct.ACC_N,
> acct.brn
> FROM L
> JOIN LA ON L.id = LA.loan_id
> JOIN FR ON L.id = FR.loan_id
> JOIN A ON LA.aid = A.id
> JOIN PI ON PI.id = LA.pi_id
> JOIN acct ON A.id = acct.aid
> WHERE
> L.id = 4436
> and acct.brn is not null;
> {code}
> the result is 3 rows
> {code}
> 10122
> NULL NULL
> NULL NULL
> {code}
> but it should be 1 row
> {code}
> 10122
> {code}
> 2.1 "explain select ..." output for hive-1.3.0 MR
> {code}
> STAGE DEPENDENCIES:
> Stage-12 is a root stage
> Stage-9 depends on stages: Stage-12
> Stage-0 depends on stages: Stage-9
> STAGE PLANS:
> Stage: Stage-12
> Map Reduce Local Work
> Alias -> Map Local Tables:
> a
> Fetch Operator
> limit: -1
> acct
> Fetch Operator
> limit: -1
> fr
> Fetch Operator
> limit: -1
> l
> Fetch Operator
> limit: -1
> pi
> Fetch Operator
> limit: -1
> Alias -> Map Local Operator Tree:
> a
> TableScan
> alias: a
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: id is not null (type: boolean)
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 _col5 (type: int)
> 1 id (type: int)
> 2 aid (type: int)
> acct
> TableScan
> alias: acct
> Statistics: Num rows: 3 Data size: 31 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: aid is not null (type: boolean)
> Statistics: Num rows: 2 Data size: 20 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 _col5 (type: int)
> 1 id (type: int)
> 2 aid (type: int)
> fr
> TableScan
> alias: fr
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: (loan_id = 4436) (type: boolean)
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 4436 (type: int)
> 1 4436 (type: int)
> 2 4436 (type: int)
> l
> TableScan
> alias: l
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: (id = 4436) (type: boolean)
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 4436 (type: int)
> 1 4436 (type: int)
> 2 4436 (type: int)
> pi
> TableScan
> alias: pi
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Ope
[jira] [Commented] (HIVE-10841) [WHERE col is not null] does not work sometimes for queries with many JOIN statements
[
https://issues.apache.org/jira/browse/HIVE-10841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568042#comment-14568042
]
Alexander Pivovarov commented on HIVE-10841:
I looked at the query plans for both cases - remove.duplicatefilters false and
true
if remove.duplicatefilters is false then
- Filter Operator predicates are added to Reduce step (including the predicates
for "acct" table)
- But Filter Operator predicates still not added to Map step for "acct" table.
I did some performance tests - looks like duplicating Filter Operator
predicates to Reduce step does not affect the performance noticeably.
So, disabling hive.ppd.remove.duplicatefilters can be used as a good workaround
until we find a way on how to fix query plan to include Filter Operator
predicates on Map step for "acct" table.
> [WHERE col is not null] does not work sometimes for queries with many JOIN
> statements
> -
>
> Key: HIVE-10841
> URL: https://issues.apache.org/jira/browse/HIVE-10841
> Project: Hive
> Issue Type: Bug
> Components: Query Planning, Query Processor
>Affects Versions: 0.13.0, 0.14.0, 0.13.1, 1.2.0
>Reporter: Alexander Pivovarov
>Assignee: Alexander Pivovarov
>
> The result from the following SELECT query is 3 rows but it should be 1 row.
> I checked it in MySQL - it returned 1 row.
> To reproduce the issue in Hive
> 1. prepare tables
> {code}
> drop table if exists L;
> drop table if exists LA;
> drop table if exists FR;
> drop table if exists A;
> drop table if exists PI;
> drop table if exists acct;
> create table L as select 4436 id;
> create table LA as select 4436 loan_id, 4748 aid, 4415 pi_id;
> create table FR as select 4436 loan_id;
> create table A as select 4748 id;
> create table PI as select 4415 id;
> create table acct as select 4748 aid, 10 acc_n, 122 brn;
> insert into table acct values(4748, null, null);
> insert into table acct values(4748, null, null);
> {code}
> 2. run SELECT query
> {code}
> select
> acct.ACC_N,
> acct.brn
> FROM L
> JOIN LA ON L.id = LA.loan_id
> JOIN FR ON L.id = FR.loan_id
> JOIN A ON LA.aid = A.id
> JOIN PI ON PI.id = LA.pi_id
> JOIN acct ON A.id = acct.aid
> WHERE
> L.id = 4436
> and acct.brn is not null;
> {code}
> the result is 3 rows
> {code}
> 10122
> NULL NULL
> NULL NULL
> {code}
> but it should be 1 row
> {code}
> 10122
> {code}
> 2.1 "explain select ..." output for hive-1.3.0 MR
> {code}
> STAGE DEPENDENCIES:
> Stage-12 is a root stage
> Stage-9 depends on stages: Stage-12
> Stage-0 depends on stages: Stage-9
> STAGE PLANS:
> Stage: Stage-12
> Map Reduce Local Work
> Alias -> Map Local Tables:
> a
> Fetch Operator
> limit: -1
> acct
> Fetch Operator
> limit: -1
> fr
> Fetch Operator
> limit: -1
> l
> Fetch Operator
> limit: -1
> pi
> Fetch Operator
> limit: -1
> Alias -> Map Local Operator Tree:
> a
> TableScan
> alias: a
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: id is not null (type: boolean)
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 _col5 (type: int)
> 1 id (type: int)
> 2 aid (type: int)
> acct
> TableScan
> alias: acct
> Statistics: Num rows: 3 Data size: 31 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: aid is not null (type: boolean)
> Statistics: Num rows: 2 Data size: 20 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 _col5 (type: int)
> 1 id (type: int)
> 2 aid (type: int)
> fr
> TableScan
> alias: fr
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: (loan_id = 4436) (type: boolean)
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 4436 (type: int)
> 1 4436 (type: int)
> 2 4436 (type: int)
> l
> TableScan
> alias: l
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filte
[jira] [Updated] (HIVE-10434) Cancel connection when remote Spark driver process has failed [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-10434: --- Fix Version/s: (was: spark-branch) 1.3.0 > Cancel connection when remote Spark driver process has failed [Spark Branch] > - > > Key: HIVE-10434 > URL: https://issues.apache.org/jira/browse/HIVE-10434 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 1.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Fix For: 1.3.0 > > Attachments: HIVE-10434.1-spark.patch, HIVE-10434.3-spark.patch, > HIVE-10434.4-spark.patch > > > Currently in HoS, in SparkClientImpl it first launch a remote Driver process, > and then wait for it to connect back to the HS2. However, in certain > situations (for instance, permission issue), the remote process may fail and > exit with error code. In this situation, the HS2 process will still wait for > the process to connect, and wait for a full timeout period before it throws > the exception. > What makes it worth, user may need to wait for two timeout periods: one for > the SparkSetReducerParallelism, and another for the actual Spark job. This > could be very annoying. > We should cancel the timeout task once we found out that the process has > failed, and set the promise as failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8858) Visualize generated Spark plan [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-8858: -- Fix Version/s: (was: spark-branch) 1.3.0 > Visualize generated Spark plan [Spark Branch] > - > > Key: HIVE-8858 > URL: https://issues.apache.org/jira/browse/HIVE-8858 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Chinna Rao Lalam > Fix For: 1.3.0 > > Attachments: HIVE-8858-spark.patch, HIVE-8858.1-spark.patch, > HIVE-8858.2-spark.patch, HIVE-8858.3-spark.patch, HIVE-8858.4-spark.patch > > > The spark plan generated by SparkPlanGenerator contains info which isn't > available in Hive's explain plan, such as RDD caching. Also, the graph is > slight different from orignal SparkWork. Thus, it would be nice to visualize > the plan as is done for SparkWork. > Preferrably, the visualization can happen as part of Hive explain extended. > If not feasible, we at least can log this at info level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10458) Enable parallel order by for spark [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-10458: --- Fix Version/s: (was: spark-branch) 1.3.0 > Enable parallel order by for spark [Spark Branch] > - > > Key: HIVE-10458 > URL: https://issues.apache.org/jira/browse/HIVE-10458 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Rui Li >Assignee: Rui Li > Fix For: 1.3.0 > > Attachments: HIVE-10458.1-spark.patch, HIVE-10458.2-spark.patch, > HIVE-10458.3-spark.patch > > > We don't have to force reducer# to 1 as spark supports parallel sorting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10626) Spark paln need to be updated [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-10626: --- Fix Version/s: (was: spark-branch) 1.3.0 > Spark paln need to be updated [Spark Branch] > > > Key: HIVE-10626 > URL: https://issues.apache.org/jira/browse/HIVE-10626 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: spark-branch >Reporter: Chinna Rao Lalam >Assignee: Chinna Rao Lalam > Fix For: 1.3.0 > > Attachments: HIVE-10626-spark.patch, HIVE-10626.1-spark.patch, > HIVE-10626.2-spark.patch > > > [HIVE-8858] basic patch was committed, latest patch need to be committed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10302) Load small tables (for map join) in executor memory only once [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-10302: --- Fix Version/s: (was: spark-branch) 1.3.0 > Load small tables (for map join) in executor memory only once [Spark Branch] > > > Key: HIVE-10302 > URL: https://issues.apache.org/jira/browse/HIVE-10302 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 1.3.0 > > Attachments: 10302.patch, HIVE-10302.2-spark.patch, > HIVE-10302.3-spark.patch, HIVE-10302.4-spark.patch, HIVE-10302.spark-1.patch > > > Usually there are multiple cores in a Spark executor, and thus it's possible > that multiple map-join tasks can be running in the same executor > (concurrently or sequentially). Currently, each task will load its own copy > of the small tables for map join into memory, ending up with inefficiency. > Ideally, we only load the small tables once and share them among the tasks > running in that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10527) NPE in SparkUtilities::isDedicatedCluster [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-10527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-10527:
---
Fix Version/s: (was: spark-branch)
1.3.0
> NPE in SparkUtilities::isDedicatedCluster [Spark Branch]
>
>
> Key: HIVE-10527
> URL: https://issues.apache.org/jira/browse/HIVE-10527
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Fix For: 1.3.0
>
> Attachments: HIVE-10527.1-spark.patch
>
>
> We should add {{spark.master}} to HiveConf when it doesn't exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10473) Spark client is recreated even spark configuration is not changed
[ https://issues.apache.org/jira/browse/HIVE-10473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-10473: --- Fix Version/s: (was: spark-branch) 1.3.0 > Spark client is recreated even spark configuration is not changed > - > > Key: HIVE-10473 > URL: https://issues.apache.org/jira/browse/HIVE-10473 > Project: Hive > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 1.3.0 > > Attachments: HIVE-10473.1-spark.patch, HIVE-10473.1.patch > > > Currently, we think a spark setting is changed as long as the set method is > called, even we set it to the same value as before. We should check if the > value is changed too, since it takes time to start a new spark client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10721) SparkSessionManagerImpl leaks SparkSessions [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-10721: --- Fix Version/s: (was: spark-branch) 1.3.0 > SparkSessionManagerImpl leaks SparkSessions [Spark Branch] > -- > > Key: HIVE-10721 > URL: https://issues.apache.org/jira/browse/HIVE-10721 > Project: Hive > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 1.3.0 > > Attachments: HIVE-10721.1-spark.patch > > > In #getSession(), we create a SparkSession and save it in a set. If the > session is failed to open, it will stay in the set till shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10671) yarn-cluster mode offers a degraded performance from yarn-client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-10671: --- Fix Version/s: (was: spark-branch) 1.3.0 > yarn-cluster mode offers a degraded performance from yarn-client [Spark > Branch] > --- > > Key: HIVE-10671 > URL: https://issues.apache.org/jira/browse/HIVE-10671 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Xuefu Zhang >Assignee: Rui Li > Fix For: 1.3.0 > > Attachments: HIVE-10671.1-spark.patch, HIVE-10671.2-spark.patch > > > With Hive on Spark, users noticed that in certain cases > spark.master=yarn-client offers 2x or 3x better performance than if > spark.master=yarn-cluster. However, yarn-cluster is what we recommend and > support. Thus, we should investigate and fix the problem. One of the such > queries is TPC-H 22. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10550) Dynamic RDD caching optimization for HoS.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-10550: --- Fix Version/s: (was: spark-branch) 1.3.0 > Dynamic RDD caching optimization for HoS.[Spark Branch] > --- > > Key: HIVE-10550 > URL: https://issues.apache.org/jira/browse/HIVE-10550 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Chengxiang Li >Assignee: Chengxiang Li > Fix For: 1.3.0 > > Attachments: HIVE-10550.1-spark.patch, HIVE-10550.1.patch, > HIVE-10550.2-spark.patch, HIVE-10550.3-spark.patch, HIVE-10550.4-spark.patch, > HIVE-10550.5-spark.patch, HIVE-10550.6-spark.patch > > > A Hive query may try to scan the same table multi times, like self-join, > self-union, or even share the same subquery, [TPC-DS > Q39|https://github.com/hortonworks/hive-testbench/blob/hive14/sample-queries-tpcds/query39.sql] > is an example. As you may know that, Spark support cache RDD data, which > mean Spark would put the calculated RDD data in memory and get the data from > memory directly for next time, this avoid the calculation cost of this > RDD(and all the cost of its dependencies) at the cost of more memory usage. > Through analyze the query context, we should be able to understand which part > of query could be shared, so that we can reuse the cached RDD in the > generated Spark job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10476) Hive query should fail when it fails to initialize a session in SetSparkReducerParallelism [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-10476: --- Fix Version/s: (was: spark-branch) 1.3.0 > Hive query should fail when it fails to initialize a session in > SetSparkReducerParallelism [Spark Branch] > - > > Key: HIVE-10476 > URL: https://issues.apache.org/jira/browse/HIVE-10476 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Fix For: 1.3.0 > > Attachments: HIVE-10476.1-spark.patch, HIVE-10476.2-spark.patch > > > Currently, for a Hive query HoS need to get a session > a session twice, once in SparkSetReducerParallelism, and another when > submitting the actual job. > The issue is that sometimes there's problem when launching a Yarn application > (e.g., don't have permission), then user will have to wait for two timeouts, > because both session initializations will fail. This turned out to happen > frequently. > This JIRA proposes to fail the query in SparkSetReducerParallelism, when it > cannot initialize the session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive "add jar" command should be able to download and add jars from a repository
[ https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568021#comment-14568021 ] Anthony Hsu commented on HIVE-9664: --- Hey [~nntnag17], Could you please update the [Hive CLI documentation|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli] with information on how to use this feature you added? [~leftylev] suggested some places to update in [this comment|https://issues.apache.org/jira/browse/HIVE-9664?focusedCommentId=1439&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1439]. > Hive "add jar" command should be able to download and add jars from a > repository > > > Key: HIVE-9664 > URL: https://issues.apache.org/jira/browse/HIVE-9664 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.14.0 >Reporter: Anant Nag >Assignee: Anant Nag > Labels: TODOC1.2, hive, patch > Fix For: 1.2.0 > > Attachments: HIVE-9664.4.patch, HIVE-9664.5.patch, HIVE-9664.patch, > HIVE-9664.patch, HIVE-9664.patch > > > Currently Hive's "add jar" command takes a local path to the dependency jar. > This clutters the local file-system as users may forget to remove this jar > later > It would be nice if Hive supported a Gradle like notation to download the jar > from a repository. > Example: add jar org:module:version > > It should also be backward compatible and should take jar from the local > file-system as well. > RB: https://reviews.apache.org/r/31628/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10302) Load small tables (for map join) in executor memory only once [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-10302: --- Attachment: 10302.patch Patch 10302 (without HIVE-) is the result of rebasing with latest master, which is actually committed to master. > Load small tables (for map join) in executor memory only once [Spark Branch] > > > Key: HIVE-10302 > URL: https://issues.apache.org/jira/browse/HIVE-10302 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: spark-branch > > Attachments: 10302.patch, HIVE-10302.2-spark.patch, > HIVE-10302.3-spark.patch, HIVE-10302.4-spark.patch, HIVE-10302.spark-1.patch > > > Usually there are multiple cores in a Spark executor, and thus it's possible > that multiple map-join tasks can be running in the same executor > (concurrently or sequentially). Currently, each task will load its own copy > of the small tables for map join into memory, ending up with inefficiency. > Ideally, we only load the small tables once and share them among the tasks > running in that executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10870) Merge Spark branch to trunk 5/29/2015
[ https://issues.apache.org/jira/browse/HIVE-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568008#comment-14568008 ] Xuefu Zhang commented on HIVE-10870: All patches are cherry-picked to master, with HIVE-10302 rebased with current master. > Merge Spark branch to trunk 5/29/2015 > - > > Key: HIVE-10870 > URL: https://issues.apache.org/jira/browse/HIVE-10870 > Project: Hive > Issue Type: Task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang > Attachments: HIVE-10870.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10884) Enable some beeline tests and turn on HIVE-4239 by default
[ https://issues.apache.org/jira/browse/HIVE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-10884: Attachment: HIVE-10884.patch WIP patch. Doesn't actually work, shutdown of the first test (?) throws some ClassNotFoundException-s and the whole thing gets stuck > Enable some beeline tests and turn on HIVE-4239 by default > -- > > Key: HIVE-10884 > URL: https://issues.apache.org/jira/browse/HIVE-10884 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-10884.patch > > > See comments in HIVE-4239. > Beeline tests with parallelism need to be enabled to turn compilation > parallelism on by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4239) Remove lock on compilation stage
[ https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-4239: --- Attachment: HIVE-4239.03.patch Updated the patch to remove beeline changes, and turn off by default. I filed HIVE-10884 for beeline and turning on by default > Remove lock on compilation stage > > > Key: HIVE-4239 > URL: https://issues.apache.org/jira/browse/HIVE-4239 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Query Processor >Reporter: Carl Steinbach >Assignee: Sergey Shelukhin > Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, > HIVE-4239.03.patch, HIVE-4239.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6727) Table level stats for external tables are set incorrectly
[
https://issues.apache.org/jira/browse/HIVE-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-6727:
---
Component/s: (was: Metastore)
Statistics
> Table level stats for external tables are set incorrectly
> -
>
> Key: HIVE-6727
> URL: https://issues.apache.org/jira/browse/HIVE-6727
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 0.13.0, 0.13.1, 1.2.0
>Reporter: Harish Butani
>Assignee: Bing Li
> Fix For: 1.3.0
>
> Attachments: HIVE-6727.2.patch, HIVE-6727.3.patch
>
>
> if you do the following:
> {code}
> CREATE EXTERNAL TABLE anaylyze_external (a INT) LOCATION
> 'data/files/ext_test';
> describe formatted anaylyze_external;
> {code}
> The table level stats are:
> {noformat}
> Table Parameters:
> COLUMN_STATS_ACCURATE true
> EXTERNALTRUE
> numFiles0
> numRows 6
> rawDataSize 6
> totalSize 0
> {noformat}
> numFiles and totalSize is always 0.
> Issue is:
> MetaStoreUtils:updateUnpartitionedTableStatsFast attempts to set table level
> stats from FileStatus. But it doesn't account for External tables, it always
> calls Warehouse.getFileStatusesForUnpartitionedTable
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-6727) Table level stats for external tables are set incorrectly
[
https://issues.apache.org/jira/browse/HIVE-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-6727:
---
Affects Version/s: 1.1.0
0.14.0
1.0.0
> Table level stats for external tables are set incorrectly
> -
>
> Key: HIVE-6727
> URL: https://issues.apache.org/jira/browse/HIVE-6727
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 0.13.0, 0.14.0, 0.13.1, 1.0.0, 1.2.0, 1.1.0
>Reporter: Harish Butani
>Assignee: Bing Li
> Fix For: 1.3.0
>
> Attachments: HIVE-6727.2.patch, HIVE-6727.3.patch
>
>
> if you do the following:
> {code}
> CREATE EXTERNAL TABLE anaylyze_external (a INT) LOCATION
> 'data/files/ext_test';
> describe formatted anaylyze_external;
> {code}
> The table level stats are:
> {noformat}
> Table Parameters:
> COLUMN_STATS_ACCURATE true
> EXTERNALTRUE
> numFiles0
> numRows 6
> rawDataSize 6
> totalSize 0
> {noformat}
> numFiles and totalSize is always 0.
> Issue is:
> MetaStoreUtils:updateUnpartitionedTableStatsFast attempts to set table level
> stats from FileStatus. But it doesn't account for External tables, it always
> calls Warehouse.getFileStatusesForUnpartitionedTable
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10878) Add tests to cover avg() function for 'x preceding and y preceding' windowing spec.
[ https://issues.apache.org/jira/browse/HIVE-10878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567923#comment-14567923 ] Ashutosh Chauhan commented on HIVE-10878: - +1 > Add tests to cover avg() function for 'x preceding and y preceding' windowing > spec. > --- > > Key: HIVE-10878 > URL: https://issues.apache.org/jira/browse/HIVE-10878 > Project: Hive > Issue Type: Sub-task > Components: PTF-Windowing >Affects Versions: 1.3.0 >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Trivial > Attachments: HIVE-10878.patch > > > avg() function to support 'x preceding and y preceding' windowing spec has > been fixed along with the one for sum(). > Add tests for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10808) Inner join on Null throwing Cast Exception
[
https://issues.apache.org/jira/browse/HIVE-10808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567920#comment-14567920
]
Naveen Gangam commented on HIVE-10808:
--
Thank you [~ashutoshc] .. appreciate it.
> Inner join on Null throwing Cast Exception
> --
>
> Key: HIVE-10808
> URL: https://issues.apache.org/jira/browse/HIVE-10808
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.13.1
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Critical
> Fix For: 1.3.0
>
> Attachments: HIVE-10808.patch
>
>
> select
> > a.col1,
> > a.col2,
> > a.col3,
> > a.col4
> > from
> > tab1 a
> > inner join
> > (
> > select
> > max(x) as x
> > from
> > tab1
> > where
> > x < 20130327
> > ) r
> > on
> > a.x = r.x
> > where
> > a.col1 = 'F'
> > and a.col3 in ('A', 'S', 'G');
> Failed Task log snippet:
> 2015-05-18 19:22:17,372 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ObjectCache: Ignoring retrieval request:
> __MAP_PLAN__
> 2015-05-18 19:22:17,372 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ObjectCache: Ignoring cache key:
> __MAP_PLAN__
> 2015-05-18 19:22:17,457 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Error in configuring
> object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 9 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
> ... 14 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 17 more
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:157)
> ... 22 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.NullStructSerDe$NullStructSerDeObjectInspector
> cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:334)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:352)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
> ... 22 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.NullStructSerDe$NullStructSerDeObjectInspector
> cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettableOI(ObjectInspectorUtils.java:)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1149)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:219)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvert
[jira] [Commented] (HIVE-10752) Revert HIVE-5193
[ https://issues.apache.org/jira/browse/HIVE-10752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567913#comment-14567913 ] Aihua Xu commented on HIVE-10752: - And also, did you verify the issue against upstream? > Revert HIVE-5193 > > > Key: HIVE-10752 > URL: https://issues.apache.org/jira/browse/HIVE-10752 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Affects Versions: 1.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-10752.patch > > > Revert HIVE-5193 since it causes pig+hcatalog not working. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10479) CBO: Calcite Operator To Hive Operator (Calcite Return Path) Empty tabAlias in columnInfo which triggers PPD
[
https://issues.apache.org/jira/browse/HIVE-10479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567912#comment-14567912
]
Hive QA commented on HIVE-10479:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736609/HIVE-10479.01.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 8994 tests executed
*Failed tests:*
{noformat}
TestUtil - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4129/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4129/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4129/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736609 - PreCommit-HIVE-TRUNK-Build
> CBO: Calcite Operator To Hive Operator (Calcite Return Path) Empty tabAlias
> in columnInfo which triggers PPD
>
>
> Key: HIVE-10479
> URL: https://issues.apache.org/jira/browse/HIVE-10479
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-10479.01.patch, HIVE-10479.patch
>
>
> in ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java, line 477,
> when aliases contains empty string "" and key is an empty string "" too, it
> assumes that aliases contains key. This will trigger incorrect PPD. To
> reproduce it, apply the HIVE-10455 and run cbo_subq_notin.q.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10752) Revert HIVE-5193
[ https://issues.apache.org/jira/browse/HIVE-10752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567911#comment-14567911 ] Aihua Xu commented on HIVE-10752: - Thanks [~mithun] Maybe you guys have additional fix in some place internally? > Revert HIVE-5193 > > > Key: HIVE-10752 > URL: https://issues.apache.org/jira/browse/HIVE-10752 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Affects Versions: 1.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-10752.patch > > > Revert HIVE-5193 since it causes pig+hcatalog not working. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10720) Pig using HCatLoader to access RCFile and perform join but get incorrect result.
[
https://issues.apache.org/jira/browse/HIVE-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567905#comment-14567905
]
Aihua Xu commented on HIVE-10720:
-
Thanks [~viraj] I updated the pig script above which you can use to check
against your local repository. I ran against Pig 0.12, but seems like the
version shouldn't matter.
> Pig using HCatLoader to access RCFile and perform join but get incorrect
> result.
>
>
> Key: HIVE-10720
> URL: https://issues.apache.org/jira/browse/HIVE-10720
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 1.3.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-10720.patch
>
>
> {noformat}
> Create table tbl1 (c1 string, c2 string, key string, value string) stored as
> rcfile;
> Create table tbl2 (key string, value string);
> insert into tbl1 values('c1', 'c2', '1', 'value1');
> insert into tbl2 values('1', 'value2');
> {noformat}
> Pig script:
> {noformat}
> tbl1 = LOAD 'tbl1' USING org.apache.hive.hcatalog.pig.HCatLoader();
> tbl2 = LOAD 'tbl2' USING org.apache.hive.hcatalog.pig.HCatLoader();
> src_tbl1 = FILTER tbl1 BY (key == '1');
> prj_tbl1 = FOREACH src_tbl1 GENERATE
>c1 as c1,
>c2 as c2,
>key as tbl1_key;
>
> src_tbl2 = FILTER tbl2 BY (key == '1');
> prj_tbl2 = FOREACH src_tbl2 GENERATE
>key as tbl2_key;
>
> result = JOIN prj_tbl1 BY (tbl1_key), prj_tbl2 BY (tbl2_key);
> dump result;
> {noformat}
> You will see result "(,,1,1)" and we are missing c1 and c2 values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10722) external table creation with msck in Hive can create unusable partition
[ https://issues.apache.org/jira/browse/HIVE-10722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567904#comment-14567904 ] Sergey Shelukhin commented on HIVE-10722: - Added to LanguageManual – DDL – Recover Partitions (MSCK REPAIR TABLE) > external table creation with msck in Hive can create unusable partition > --- > > Key: HIVE-10722 > URL: https://issues.apache.org/jira/browse/HIVE-10722 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.1, 1.0.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Critical > Labels: TODOC1.3 > Fix For: 1.3.0 > > Attachments: HIVE-10722.01.patch, HIVE-10722.patch > > > There can be directories in HDFS containing unprintable characters; when > doing hadoop fs -ls, these characters are not even visible, and can only be > seen for example if output is piped thru od. > When these are loaded via msck, they are stored in e.g. mysql as "?" (literal > question mark, findable via LIKE '%?%' in db) and show accordingly in Hive. > However, datanucleus appears to encode it as %3F; this causes the partition > to be unusable - it cannot be dropped, and other operations like drop table > get stuck (didn't investigate in detail why; drop table got unstuck as soon > as the partition was removed from metastore). > We should probably have a 2-way option for such cases - error out on load > (default), or convert to '?'/drop such characters (and have partition that > actually works, too). > We should also check if partitions with '?' inserted explicitly work at all > with datanucleus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10752) Revert HIVE-5193
[ https://issues.apache.org/jira/browse/HIVE-10752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567892#comment-14567892 ] Mithun Radhakrishnan commented on HIVE-10752: - Yes, of course. +1, as per [HIVE-10720|https://issues.apache.org/jira/browse/HIVE-10720?focusedCommentId=14565768&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14565768]. Let's circle back, after Viraj and I have identified why this isn't a problem with our internal Hive 0.13-0.14 branch. > Revert HIVE-5193 > > > Key: HIVE-10752 > URL: https://issues.apache.org/jira/browse/HIVE-10752 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Affects Versions: 1.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-10752.patch > > > Revert HIVE-5193 since it causes pig+hcatalog not working. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10851) HCatalog NullPointerException when using views in Pig
[ https://issues.apache.org/jira/browse/HIVE-10851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-10851: Attachment: HIVE-10851.patch > HCatalog NullPointerException when using views in Pig > - > > Key: HIVE-10851 > URL: https://issues.apache.org/jira/browse/HIVE-10851 > Project: Hive > Issue Type: Bug > Components: HCatalog, StorageHandler >Reporter: Dustin Cote >Assignee: Aihua Xu > Attachments: HIVE-10851.patch > > > When trying to access a hive table view with HCatalog through Pig, a > NullPointerException is thrown. I realize that it is documented here that > HCatalog and views do not work for Pig: > https://cwiki.apache.org/confluence/display/Hive/HCatalog+CLI#HCatalogCLI-Create/Drop/AlterView > However, surfacing a NullPointerException is not the proper way to handle the > incompatibility. Stack trace below: > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias A > at org.apache.pig.PigServer.openIterator(PigServer.java:892) > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173) > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) > at org.apache.pig.Main.run(Main.java:541) > at org.apache.pig.Main.main(Main.java:156) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A > at org.apache.pig.PigServer.storeEx(PigServer.java:994) > at org.apache.pig.PigServer.store(PigServer.java:954) > at org.apache.pig.PigServer.openIterator(PigServer.java:867) > ... 13 more > Caused by: > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: > ERROR 2017: Internal error creating job configuration. > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:873) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:298) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:190) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1334) > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1319) > at org.apache.pig.PigServer.storeEx(PigServer.java:990) > ... 15 more > Caused by: java.io.IOException: java.lang.NullPointerException > at > org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97) > at > org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:61) > at > org.apache.hive.hcatalog.pig.HCatLoader.setLocation(HCatLoader.java:127) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:498) > ... 20 more > Caused by: java.lang.NullPointerException > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:190) > at > org.apache.hive.hcatalog.mapreduce.FosterStorageHandler.(FosterStorageHandler.java:59) > at > org.apache.hive.hcatalog.common.HCatUtil.getStorageHandler(HCatUtil.java:404) > at > org.apache.hive.hcatalog.common.HCatUtil.getStorageHandler(HCatUtil.java:367) > at > org.apache.hive.hcatalog.mapreduce.InitializeInput.extractPartInfo(InitializeInput.java:158) > at > org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:137) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
