[jira] [Updated] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG
[ https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16589: Attachment: HIVE-16589.092.patch > Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and > COMPLETE for AVG > - > > Key: HIVE-16589 > URL: https://issues.apache.org/jira/browse/HIVE-16589 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch, > HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch, > HIVE-16589.06.patch, HIVE-16589.07.patch, HIVE-16589.08.patch, > HIVE-16589.091.patch, HIVE-16589.092.patch, HIVE-16589.09.patch > > > Allow Complex Types to be vectorized (since HIVE-16207: "Add support for > Complex Types in Fast SerDe" was committed). > Add more classes we vectorize AVG in preparation for fully supporting AVG > GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in > the AVG struct as input. And, add the COMPLETE mode that takes in the > Original data and produces the Full Aggregation for completeness, so to speak. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG
[ https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16589: Status: Patch Available (was: In Progress) > Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and > COMPLETE for AVG > - > > Key: HIVE-16589 > URL: https://issues.apache.org/jira/browse/HIVE-16589 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch, > HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch, > HIVE-16589.06.patch, HIVE-16589.07.patch, HIVE-16589.08.patch, > HIVE-16589.091.patch, HIVE-16589.092.patch, HIVE-16589.09.patch > > > Allow Complex Types to be vectorized (since HIVE-16207: "Add support for > Complex Types in Fast SerDe" was committed). > Add more classes we vectorize AVG in preparation for fully supporting AVG > GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in > the AVG struct as input. And, add the COMPLETE mode that takes in the > Original data and produces the Full Aggregation for completeness, so to speak. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG
[ https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16589: Status: In Progress (was: Patch Available) > Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and > COMPLETE for AVG > - > > Key: HIVE-16589 > URL: https://issues.apache.org/jira/browse/HIVE-16589 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch, > HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch, > HIVE-16589.06.patch, HIVE-16589.07.patch, HIVE-16589.08.patch, > HIVE-16589.091.patch, HIVE-16589.092.patch, HIVE-16589.09.patch > > > Allow Complex Types to be vectorized (since HIVE-16207: "Add support for > Complex Types in Fast SerDe" was committed). > Add more classes we vectorize AVG in preparation for fully supporting AVG > GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in > the AVG struct as input. And, add the COMPLETE mode that takes in the > Original data and produces the Full Aggregation for completeness, so to speak. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16600) Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel order by in multi_insert cases
[
https://issues.apache.org/jira/browse/HIVE-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022458#comment-16022458
]
Hive QA commented on HIVE-16600:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869588/HIVE-16600.8.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10749 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=151)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5413/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5413/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5413/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12869588 - PreCommit-HIVE-Build
> Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel
> order by in multi_insert cases
>
>
> Key: HIVE-16600
> URL: https://issues.apache.org/jira/browse/HIVE-16600
> Project: Hive
> Issue Type: Sub-task
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16600.1.patch, HIVE-16600.2.patch,
> HIVE-16600.3.patch, HIVE-16600.4.patch, HIVE-16600.5.patch,
> HIVE-16600.6.patch, HIVE-16600.7.patch, HIVE-16600.8.patch, mr.explain,
> mr.explain.log.HIVE-16600
>
>
> multi_insert_gby.case.q
> {code}
> set hive.exec.reducers.bytes.per.reducer=256;
> set hive.optimize.sampling.orderby=true;
> drop table if exists e1;
> drop table if exists e2;
> create table e1 (key string, value string);
> create table e2 (key string);
> FROM (select key, cast(key as double) as keyD, value from src order by key) a
> INSERT OVERWRITE TABLE e1
> SELECT key, value
> INSERT OVERWRITE TABLE e2
> SELECT key;
> select * from e1;
> select * from e2;
> {code}
> the parallelism of Sort is 1 even we enable parallel order
> by("hive.optimize.sampling.orderby" is set as "true"). This is not
> reasonable because the parallelism should be calcuated by
> [Utilities.estimateReducers|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L170]
> this is because SetSparkReducerParallelism#needSetParallelism returns false
> when [children size of
> RS|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L207]
> is greater than 1.
> in this case, the children size of {{RS[2]}} is two.
> the logical plan of the case
> {code}
>TS[0]-SEL[1]-RS[2]-SEL[3]-SEL[4]-FS[5]
> -SEL[6]-FS[7]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16747) Remove YETUS*.sh files after a YETUS release
[
https://issues.apache.org/jira/browse/HIVE-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-16747:
-
> Remove YETUS*.sh files after a YETUS release
>
>
> Key: HIVE-16747
> URL: https://issues.apache.org/jira/browse/HIVE-16747
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> For HIVE-15051 we had to add patched YETUS files which contains YETUS fixes
> which are not yet released:
> - dev-support/checkstyle_YETUS-484.sh
> - dev-support/findbugs_YETUS-471.sh
> - dev-support/maven_YETUS-506.sh
> When there is a new YETUS release, then we have to move to the new release,
> and remove the files.
> Also we have to remove the cp commands from the {{yetus-wrapper.sh}} in 3x3
> places:
> {code}
> 75 cp ${BINDIR}/findbugs_YETUS-471.sh
> ${YETUS_HOME}/lib/precommit/test-patch.d/findbugs.sh
> 76 cp ${BINDIR}/checkstyle_YETUS-484.sh
> ${YETUS_HOME}/lib/precommit/test-patch.d/checkstyle.sh
> 77 cp ${BINDIR}/maven_YETUS-506.sh
> ${YETUS_HOME}/lib/precommit/test-patch.d/maven.sh
> {code}
> {code}
> 101 cp ${BINDIR}/findbugs_YETUS-471.sh
> ${HIVE_PATCHPROCESS}/yetus-${HIVE_YETUS_VERSION}/lib/precommit/test-patch.d/findbugs.sh
> 102 cp ${BINDIR}/checkstyle_YETUS-484.sh
> ${HIVE_PATCHPROCESS}/yetus-${HIVE_YETUS_VERSION}/lib/precommit/test-patch.d/checkstyle.sh
> 103 cp ${BINDIR}/maven_YETUS-506.sh
> ${HIVE_PATCHPROCESS}/yetus-${HIVE_YETUS_VERSION}/lib/precommit/test-patch.d/maven.sh
> {code}
> {code}
> 175 cp ${BINDIR}/findbugs_YETUS-471.sh
> ${HIVE_PATCHPROCESS}/yetus-${HIVE_YETUS_VERSION}/lib/precommit/test-patch.d/findbugs.sh
> 176 cp ${BINDIR}/checkstyle_YETUS-484.sh
> ${HIVE_PATCHPROCESS}/yetus-${HIVE_YETUS_VERSION}/lib/precommit/test-patch.d/checkstyle.sh
> 177 cp ${BINDIR}/maven_YETUS-506.sh
> ${HIVE_PATCHPROCESS}/yetus-${HIVE_YETUS_VERSION}/lib/precommit/test-patch.d/maven.sh
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16748) Integreate YETUS to Pre-Commit
[ https://issues.apache.org/jira/browse/HIVE-16748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-16748: - > Integreate YETUS to Pre-Commit > -- > > Key: HIVE-16748 > URL: https://issues.apache.org/jira/browse/HIVE-16748 > Project: Hive > Issue Type: Sub-task >Reporter: Peter Vary >Assignee: Peter Vary > > After HIVE-15051, we should automate the yetus run for the Pre-Commit tests, > so the results are added in comments like > https://issues.apache.org/jira/browse/YARN-6363?focusedCommentId=15937570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15937570 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022489#comment-16022489 ] Peter Vary commented on HIVE-15051: --- Created followup jiras: - HIVE-16747 Remove YETUS*.sh files after a YETUS release - HIVE-16748 Integreate YETUS to Pre-Commit Thanks [~thejas] for the review! > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15834) Add unit tests for org.json usage on master
[ https://issues.apache.org/jira/browse/HIVE-15834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Voros updated HIVE-15834: Attachment: HIVE-15834.3.patch Attaching patch #3. This rebases the previous patch on the current master. > Add unit tests for org.json usage on master > --- > > Key: HIVE-15834 > URL: https://issues.apache.org/jira/browse/HIVE-15834 > Project: Hive > Issue Type: Sub-task >Reporter: Daniel Voros >Assignee: Daniel Voros > Attachments: HIVE-15834.1.patch, HIVE-15834.2.patch, > HIVE-15834.3.patch > > > Before switching implementation, we should add some tests that capture the > current behavior. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022527#comment-16022527 ] Zoltan Haindrich commented on HIVE-15051: - [~pvary] it seems to me that the hadoop guys are using some docker containers to execute yetusand somehow they are already using 0.5.0-SNAPSHOT...I suspect that they have faced similar issues deploying/using yetus - like missing patches. I think it would be great to containerize the yetus runner. That way the current extra "patching" could be described in the Dockerfile. hadoop yetus image: https://github.com/apache/hadoop/blob/trunk/dev-support/bin/create-release#L457 https://github.com/apache/hadoop/blob/trunk/dev-support/docker/Dockerfile what do you think about it? > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16746) Reduce number of index lookups for same table in IndexWhereTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022533#comment-16022533
]
Hive QA commented on HIVE-16746:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869580/HIVE-16746.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10749 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5414/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5414/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5414/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12869580 - PreCommit-HIVE-Build
> Reduce number of index lookups for same table in IndexWhereTaskDispatcher
> -
>
> Key: HIVE-16746
> URL: https://issues.apache.org/jira/browse/HIVE-16746
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16746.1.patch
>
>
> {{IndexWhereTaskDispatcher}} is used when
> {{hive.optimize.index.filter=true}}. It lists all indices for the table and
> depending on the query complexity, this ends up being in the hotpath. For
> e.g, Q14 explain plan takes 180-200 seconds and this index querying multiple
> times for same tables take up 30-40 seconds. This function was invoked around
> 24000 times for same set of tables.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16746) Reduce number of index lookups for same table in IndexWhereTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022540#comment-16022540
]
Rajesh Balamohan commented on HIVE-16746:
-
test failures are not related to this patch.
> Reduce number of index lookups for same table in IndexWhereTaskDispatcher
> -
>
> Key: HIVE-16746
> URL: https://issues.apache.org/jira/browse/HIVE-16746
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16746.1.patch
>
>
> {{IndexWhereTaskDispatcher}} is used when
> {{hive.optimize.index.filter=true}}. It lists all indices for the table and
> depending on the query complexity, this ends up being in the hotpath. For
> e.g, Q14 explain plan takes 180-200 seconds and this index querying multiple
> times for same tables take up 30-40 seconds. This function was invoked around
> 24000 times for same set of tables.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16746) Reduce number of index lookups for same table in IndexWhereTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022552#comment-16022552
]
Rajesh Balamohan commented on HIVE-16746:
-
RB: https://reviews.apache.org/r/59517/
> Reduce number of index lookups for same table in IndexWhereTaskDispatcher
> -
>
> Key: HIVE-16746
> URL: https://issues.apache.org/jira/browse/HIVE-16746
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16746.1.patch
>
>
> {{IndexWhereTaskDispatcher}} is used when
> {{hive.optimize.index.filter=true}}. It lists all indices for the table and
> depending on the query complexity, this ends up being in the hotpath. For
> e.g, Q14 explain plan takes 180-200 seconds and this index querying multiple
> times for same tables take up 30-40 seconds. This function was invoked around
> 24000 times for same set of tables.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16749) Run YETUS in Docker container
[ https://issues.apache.org/jira/browse/HIVE-16749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-16749: - > Run YETUS in Docker container > - > > Key: HIVE-16749 > URL: https://issues.apache.org/jira/browse/HIVE-16749 > Project: Hive > Issue Type: Sub-task >Reporter: Peter Vary >Assignee: Peter Vary > > Think about the pros and cons of running YETUS in a docker container: > - Resources > - Usage complexity > - Yetus version changes > - Findbugs > - etc. > If worthwhile run YETUS in a docker container -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022565#comment-16022565 ] Peter Vary commented on HIVE-15051: --- To be honest I was thinking about using the snapshot version (by downloading the latest Yetus in the wrapper), but decided against it to use a development version of YETUS in "production" :) Where did you find references of 0.5.0-SNAPSHOT in hadoop. I did some digging again, and it seems to me that they are using 0.4.0: https://github.com/apache/hadoop/blob/trunk/dev-support/bin/yetus-wrapper#L76 I might miss something... Docker image would be good. It could help with the stuff which is done now manually hidden in yetus-wrapper.sh and could help with installing findbugs as well. Created a followup jira for it: - HIVE-16749 Run YETUS in Docker container > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[
https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022580#comment-16022580
]
Zoltan Haindrich commented on HIVE-15051:
-
interesting...they might be using that for some reason...the source doesn't
reference it...
I've seen it in the link you posted earlier:
https://issues.apache.org/jira/browse/YARN-6363?focusedCommentId=15937570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15937570
- somewhere at the end there is a note: {{Powered by Apache Yetus
0.5.0-SNAPSHOT http://yetus.apache.org}}
yes...it could simplify the wrapper a bit :) I'll continue commenting on the
new ticket then! :D
> Test framework integration with findbugs, rat checks etc.
> -
>
> Key: HIVE-15051
> URL: https://issues.apache.org/jira/browse/HIVE-15051
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out
>
>
> Find a way to integrate code analysis tools like findbugs, rat checks to
> PreCommit tests, thus removing the burden from reviewers to check the code
> style and other checks which could be done by code.
> Might worth to take a look on Yetus, but keep in mind the Hive has a specific
> parallel test framework.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG
[
https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022600#comment-16022600
]
Hive QA commented on HIVE-16589:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869594/HIVE-16589.092.patch
{color:green}SUCCESS:{color} +1 due to 27 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 52 failed/errored test(s), 10748 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=236)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1]
(batchId=236)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_2]
(batchId=236)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_10]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_11]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_12]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_13]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_14]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_15]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_16]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_17]
(batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_1]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_2]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_3]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_4]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_5]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_6]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_7]
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_8]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_9]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_timestamp]
(batchId=73)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_aggregate]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_udf]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_15]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_short_regress]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_distinct_gby]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_timestamp]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_timestamp_funcs]
(batchId=148)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_aggregate]
(batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_0]
(batchId=134)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_10]
(batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_11]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_12]
(batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_13]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_14]
(batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_15]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_16]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_17]
(batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_1]
(batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_2]
(batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_3]
(batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_4]
(batchId=108)
org.apache.hadoop.hive.cli.TestSpa
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022607#comment-16022607 ] Peter Vary commented on HIVE-15051: --- They might use a different version on the pre-commit jenkins job then. Good to know! Thanks, Peter > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16720) Reorder columns in Hive parquet tables
[ https://issues.apache.org/jira/browse/HIVE-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022651#comment-16022651 ] Barna Zsombor Klara commented on HIVE-16720: [~venkatakarthik], well cascade is guaranteed to work, which is definitely a substantial benefit :). The problem is that we have column information on both the table and the partition level. As long as no columns are dropped or reordered just added to the table it will work fine, but if you replace columns, or just drop from the table but not the partition then you may end up with the exception like you got in the description. I am still looking into a possible solution but it is a bit tricky without significant performance overhead, so I may be forced to leave it like this. FYI there is a HiveConf property introduced in HIVE-4409 that would check column incompatibility during an alter statement. Setting this to true should at least prevent you from ending up with a table you can't select from. > Reorder columns in Hive parquet tables > -- > > Key: HIVE-16720 > URL: https://issues.apache.org/jira/browse/HIVE-16720 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.1.0 >Reporter: Venkata Karthik Penikalapati > > We are using hive 1.1.0 have a requirement to reorder columns for one of huge > table with 100's of columns for user readability. But when we reorder columns > in table its failing with below error. > hive> desc test_parquet; > OK > namestring > age int > deptstring > salary string > citystring > # Partition Information > # col_name data_type comment > citystring > Time taken: 0.053 seconds, Fetched: 10 row(s) > hive> ALTER TABLE test_parquet REPLACE COLUMNS (age int,name string, dept > string, salary string); > OK > Time taken: 0.451 seconds > hive> desc test_parquet; > OK > age int > namestring > deptstring > salary string > citystring > # Partition Information > # col_name data_type comment > citystring > Time taken: 0.051 seconds, Fetched: 10 row(s) > hive> select * from test_parquet; > OK > Failed with exception > java.io.IOException:java.lang.UnsupportedOperationException: Cannot inspect > org.apache.hadoop.io.IntWritable > Time taken: 0.121 seconds -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15834) Add unit tests for org.json usage on master
[
https://issues.apache.org/jira/browse/HIVE-15834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022666#comment-16022666
]
Hive QA commented on HIVE-15834:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869600/HIVE-15834.3.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10774 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
(batchId=149)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5416/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5416/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5416/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12869600 - PreCommit-HIVE-Build
> Add unit tests for org.json usage on master
> ---
>
> Key: HIVE-15834
> URL: https://issues.apache.org/jira/browse/HIVE-15834
> Project: Hive
> Issue Type: Sub-task
>Reporter: Daniel Voros
>Assignee: Daniel Voros
> Attachments: HIVE-15834.1.patch, HIVE-15834.2.patch,
> HIVE-15834.3.patch
>
>
> Before switching implementation, we should add some tests that capture the
> current behavior.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Work started] (HIVE-16750) Support change management for rename table/partition.
[ https://issues.apache.org/jira/browse/HIVE-16750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-16750 started by Sankar Hariappan. --- > Support change management for rename table/partition. > - > > Key: HIVE-16750 > URL: https://issues.apache.org/jira/browse/HIVE-16750 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > > Currently, rename table/partition updates the data location by renaming the > directory which is equivalent to moving files to new path and delete old > path. So, this should trigger move of files into $CMROOT. > Scenario: > 1. Create a table (T1) > 2. Insert a record > 3. Rename the table(T1 -> T2) > 4. Repl Dump till Insert. > 5. Repl Load from the dump. > 6. Target DB should have table T1 with the record. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16600) Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel order by in multi_insert cases
[
https://issues.apache.org/jira/browse/HIVE-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022736#comment-16022736
]
Rui Li commented on HIVE-16600:
---
[~kellyzly], I think there's a bug in latest patch. Suppose we have OP tree
like this: {{RS -> LIM -> MultiInsert}}. The RS shouldn't be a parallel order
by because it's followed by a LIM. But since this is a multi insert, we'll end
up with parallel RS.
How about we use the following algorithm:
{code}
RS rs;
Operator op=rs;
while(op!=null){
if(op instanceof LIM){
return false;
}
if((op instanceof RS && op!=rs) || op instanceof FS){
return true;
}
if(op has branches){
return isMultiInsert;
}
op=op.child;
}
return true;
{code}
> Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel
> order by in multi_insert cases
>
>
> Key: HIVE-16600
> URL: https://issues.apache.org/jira/browse/HIVE-16600
> Project: Hive
> Issue Type: Sub-task
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Attachments: HIVE-16600.1.patch, HIVE-16600.2.patch,
> HIVE-16600.3.patch, HIVE-16600.4.patch, HIVE-16600.5.patch,
> HIVE-16600.6.patch, HIVE-16600.7.patch, HIVE-16600.8.patch, mr.explain,
> mr.explain.log.HIVE-16600
>
>
> multi_insert_gby.case.q
> {code}
> set hive.exec.reducers.bytes.per.reducer=256;
> set hive.optimize.sampling.orderby=true;
> drop table if exists e1;
> drop table if exists e2;
> create table e1 (key string, value string);
> create table e2 (key string);
> FROM (select key, cast(key as double) as keyD, value from src order by key) a
> INSERT OVERWRITE TABLE e1
> SELECT key, value
> INSERT OVERWRITE TABLE e2
> SELECT key;
> select * from e1;
> select * from e2;
> {code}
> the parallelism of Sort is 1 even we enable parallel order
> by("hive.optimize.sampling.orderby" is set as "true"). This is not
> reasonable because the parallelism should be calcuated by
> [Utilities.estimateReducers|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L170]
> this is because SetSparkReducerParallelism#needSetParallelism returns false
> when [children size of
> RS|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L207]
> is greater than 1.
> in this case, the children size of {{RS[2]}} is two.
> the logical plan of the case
> {code}
>TS[0]-SEL[1]-RS[2]-SEL[3]-SEL[4]-FS[5]
> -SEL[6]-FS[7]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16750) Support change management for rename table/partition.
[ https://issues.apache.org/jira/browse/HIVE-16750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan reassigned HIVE-16750: --- > Support change management for rename table/partition. > - > > Key: HIVE-16750 > URL: https://issues.apache.org/jira/browse/HIVE-16750 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > > Currently, rename table/partition updates the data location by renaming the > directory which is equivalent to moving files to new path and delete old > path. So, this should trigger move of files into $CMROOT. > Scenario: > 1. Create a table (T1) > 2. Insert a record > 3. Rename the table(T1 -> T2) > 4. Repl Dump till Insert. > 5. Repl Load from the dump. > 6. Target DB should have table T1 with the record. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15834) Add unit tests for org.json usage on master
[ https://issues.apache.org/jira/browse/HIVE-15834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022775#comment-16022775 ] Daniel Voros commented on HIVE-15834: - Failed tests are unrelated. > Add unit tests for org.json usage on master > --- > > Key: HIVE-15834 > URL: https://issues.apache.org/jira/browse/HIVE-15834 > Project: Hive > Issue Type: Sub-task >Reporter: Daniel Voros >Assignee: Daniel Voros > Attachments: HIVE-15834.1.patch, HIVE-15834.2.patch, > HIVE-15834.3.patch > > > Before switching implementation, we should add some tests that capture the > current behavior. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[
https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022815#comment-16022815
]
Zoltan Haindrich commented on HIVE-15051:
-
[~pvary]: There is some issue with the current working of this patch...if I
don't have hive already installed at the local {{.m2}} it doesn't really work -
because it refers to some hive artifacts which are not yet installed...I don't
entirely understand the concept behind executing {{mvn install}} only in just
the changed modules ( this seems to me an interesting idea to do in a multi
module project ).
Anyway adding {{mvn install -DskipTests -DskipSparkTests}} to the test-patch.sh
have fixed this for me - leaving this out may result in unrelated compile
errors...which might be misleading.
note: the usage of the tool requires a clean workspace; however the cleanup is
left for the user...which is a bit odd; or this is normal behaviour?
> Test framework integration with findbugs, rat checks etc.
> -
>
> Key: HIVE-15051
> URL: https://issues.apache.org/jira/browse/HIVE-15051
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out
>
>
> Find a way to integrate code analysis tools like findbugs, rat checks to
> PreCommit tests, thus removing the burden from reviewers to check the code
> style and other checks which could be done by code.
> Might worth to take a look on Yetus, but keep in mind the Hive has a specific
> parallel test framework.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16750) Support change management for rename table/partition.
[ https://issues.apache.org/jira/browse/HIVE-16750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16750: Description: Currently, rename table/partition updates the data location by renaming the directory which is equivalent to moving files to new path and delete old path. So, this should trigger move of files into $CMROOT. Scenario: 1. Create a table (T1) 2. Insert a record 3. Rename the table(T1 -> T2) 4. Repl Dump till Insert. 5. Repl Load from the dump. 6. Target DB should have table T1 with the record. Similar scenario with rename partition as well. was: Currently, rename table/partition updates the data location by renaming the directory which is equivalent to moving files to new path and delete old path. So, this should trigger move of files into $CMROOT. Scenario: 1. Create a table (T1) 2. Insert a record 3. Rename the table(T1 -> T2) 4. Repl Dump till Insert. 5. Repl Load from the dump. 6. Target DB should have table T1 with the record. > Support change management for rename table/partition. > - > > Key: HIVE-16750 > URL: https://issues.apache.org/jira/browse/HIVE-16750 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > > Currently, rename table/partition updates the data location by renaming the > directory which is equivalent to moving files to new path and delete old > path. So, this should trigger move of files into $CMROOT. > Scenario: > 1. Create a table (T1) > 2. Insert a record > 3. Rename the table(T1 -> T2) > 4. Repl Dump till Insert. > 5. Repl Load from the dump. > 6. Target DB should have table T1 with the record. > Similar scenario with rename partition as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16750) Support change management for rename table/partition.
[ https://issues.apache.org/jira/browse/HIVE-16750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16750: Labels: DR replication (was: ) > Support change management for rename table/partition. > - > > Key: HIVE-16750 > URL: https://issues.apache.org/jira/browse/HIVE-16750 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > > Currently, rename table/partition updates the data location by renaming the > directory which is equivalent to moving files to new path and delete old > path. So, this should trigger move of files into $CMROOT. > Scenario: > 1. Create a table (T1) > 2. Insert a record > 3. Rename the table(T1 -> T2) > 4. Repl Dump till Insert. > 5. Repl Load from the dump. > 6. Target DB should have table T1 with the record. > Similar scenario with rename partition as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[
https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022881#comment-16022881
]
Peter Vary commented on HIVE-15051:
---
[~kgyrtkirk]:
- Preinstalling artifacts / running only changed modules: I will check how the
Hadoop/HBase etc projects solve this problem. Running build only on the changed
modules helps in reducing the runtime. Your suggested solution could be a good
idea too.
- Clean workspace: By design YETUS can run on dirty workspace if needed
{{./dev-support/test-patch.sh --dirty-workspace}}. I do not really know why
would you do it, but at least you can! :) :)
Thanks for the review! Will take a look at the stuff.
Peter
> Test framework integration with findbugs, rat checks etc.
> -
>
> Key: HIVE-15051
> URL: https://issues.apache.org/jira/browse/HIVE-15051
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out
>
>
> Find a way to integrate code analysis tools like findbugs, rat checks to
> PreCommit tests, thus removing the burden from reviewers to check the code
> style and other checks which could be done by code.
> Might worth to take a look on Yetus, but keep in mind the Hive has a specific
> parallel test framework.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16643) BeeLine tests output should keep the PREHOOK/POSTHOOK Input/Output orderdering
[
https://issues.apache.org/jira/browse/HIVE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-16643:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Push the fix into mater. Thanks [~pvary] for your contribution!
> BeeLine tests output should keep the PREHOOK/POSTHOOK Input/Output orderdering
> --
>
> Key: HIVE-16643
> URL: https://issues.apache.org/jira/browse/HIVE-16643
> Project: Hive
> Issue Type: New Feature
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 3.0.0
>
> Attachments: HIVE-16643.01.patch, HIVE-16643.patch
>
>
> The {{PreExecutePrinter}} and the {{PostExecutePrinter}} prints the query
> input and the output list in alphabetical order in {{printEntities}} method.
> Our goal is to have the same output from the BeeLine query tests, and the Cli
> query tests. Since the BeeLine tests are using test specific databases to run
> the tests, and only converting the results in the end to remove this specific
> database names from the output, we have to reorder the lists after this
> conversion.
> Raw BeeLine output:
> {code}
> [..]
> INFO : PREHOOK: Output: create_merge_compressed@src_rc_merge_test
> INFO : PREHOOK: Output: database:create_merge_compressed
> [..]
> {code}
> Before patch BeeLine output:
> {code}
> [..]
> PREHOOK: Output: default@src_rc_merge_test
> PREHOOK: Output: database:default
> [..]
> {code}
> Expected output:
> {code}
> [..]
> PREHOOK: Output: database:default
> PREHOOK: Output: default@src_rc_merge_test
> [..]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16751) Support different types for grouping columns in GroupBy Druid queries
[ https://issues.apache.org/jira/browse/HIVE-16751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-16751: -- > Support different types for grouping columns in GroupBy Druid queries > - > > Key: HIVE-16751 > URL: https://issues.apache.org/jira/browse/HIVE-16751 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Calcite 1.13 pushes EXTRACT and FLOOR function to Druid as an extraction > function (cf CALCITE-1758). Originally, we were assuming that all group by > columns in a druid query were of STRING type; however, this will not true > anymore (result of EXTRACT is an INT and result of FLOOR a TIMESTAMP). > When we upgrade to Calcite 1.13, we will need to extend the DruidSerDe to > handle these functions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16751) Support different types for grouping columns in GroupBy Druid queries
[ https://issues.apache.org/jira/browse/HIVE-16751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-16751: --- Attachment: HIVE-16751.patch > Support different types for grouping columns in GroupBy Druid queries > - > > Key: HIVE-16751 > URL: https://issues.apache.org/jira/browse/HIVE-16751 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-16751.patch > > > Calcite 1.13 pushes EXTRACT and FLOOR function to Druid as an extraction > function (cf CALCITE-1758). Originally, we were assuming that all group by > columns in a druid query were of STRING type; however, this will not true > anymore (result of EXTRACT is an INT and result of FLOOR a TIMESTAMP). > When we upgrade to Calcite 1.13, we will need to extend the DruidSerDe to > handle these functions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16643) BeeLine tests output should keep the PREHOOK/POSTHOOK Input/Output orderdering
[
https://issues.apache.org/jira/browse/HIVE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023089#comment-16023089
]
Peter Vary commented on HIVE-16643:
---
Thanks [~ychena] for the review and the commit!
> BeeLine tests output should keep the PREHOOK/POSTHOOK Input/Output orderdering
> --
>
> Key: HIVE-16643
> URL: https://issues.apache.org/jira/browse/HIVE-16643
> Project: Hive
> Issue Type: New Feature
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 3.0.0
>
> Attachments: HIVE-16643.01.patch, HIVE-16643.patch
>
>
> The {{PreExecutePrinter}} and the {{PostExecutePrinter}} prints the query
> input and the output list in alphabetical order in {{printEntities}} method.
> Our goal is to have the same output from the BeeLine query tests, and the Cli
> query tests. Since the BeeLine tests are using test specific databases to run
> the tests, and only converting the results in the end to remove this specific
> database names from the output, we have to reorder the lists after this
> conversion.
> Raw BeeLine output:
> {code}
> [..]
> INFO : PREHOOK: Output: create_merge_compressed@src_rc_merge_test
> INFO : PREHOOK: Output: database:create_merge_compressed
> [..]
> {code}
> Before patch BeeLine output:
> {code}
> [..]
> PREHOOK: Output: default@src_rc_merge_test
> PREHOOK: Output: database:default
> [..]
> {code}
> Expected output:
> {code}
> [..]
> PREHOOK: Output: database:default
> PREHOOK: Output: default@src_rc_merge_test
> [..]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG, VARIANCE
[ https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16589: Summary: Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG, VARIANCE (was: Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG) > Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and > COMPLETE for AVG, VARIANCE > --- > > Key: HIVE-16589 > URL: https://issues.apache.org/jira/browse/HIVE-16589 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch, > HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch, > HIVE-16589.06.patch, HIVE-16589.07.patch, HIVE-16589.08.patch, > HIVE-16589.091.patch, HIVE-16589.092.patch, HIVE-16589.09.patch > > > Allow Complex Types to be vectorized (since HIVE-16207: "Add support for > Complex Types in Fast SerDe" was committed). > Add more classes we vectorize AVG in preparation for fully supporting AVG > GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in > the AVG struct as input. And, add the COMPLETE mode that takes in the > Original data and produces the Full Aggregation for completeness, so to speak. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG, VARIANCE
[ https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16589: Status: Patch Available (was: In Progress) > Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and > COMPLETE for AVG, VARIANCE > --- > > Key: HIVE-16589 > URL: https://issues.apache.org/jira/browse/HIVE-16589 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch, > HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch, > HIVE-16589.06.patch, HIVE-16589.07.patch, HIVE-16589.08.patch, > HIVE-16589.091.patch, HIVE-16589.092.patch, HIVE-16589.093.patch, > HIVE-16589.09.patch > > > Allow Complex Types to be vectorized (since HIVE-16207: "Add support for > Complex Types in Fast SerDe" was committed). > Add more classes we vectorize AVG in preparation for fully supporting AVG > GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in > the AVG struct as input. And, add the COMPLETE mode that takes in the > Original data and produces the Full Aggregation for completeness, so to speak. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG, VARIANCE
[ https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16589: Status: In Progress (was: Patch Available) > Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and > COMPLETE for AVG, VARIANCE > --- > > Key: HIVE-16589 > URL: https://issues.apache.org/jira/browse/HIVE-16589 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch, > HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch, > HIVE-16589.06.patch, HIVE-16589.07.patch, HIVE-16589.08.patch, > HIVE-16589.091.patch, HIVE-16589.092.patch, HIVE-16589.093.patch, > HIVE-16589.09.patch > > > Allow Complex Types to be vectorized (since HIVE-16207: "Add support for > Complex Types in Fast SerDe" was committed). > Add more classes we vectorize AVG in preparation for fully supporting AVG > GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in > the AVG struct as input. And, add the COMPLETE mode that takes in the > Original data and produces the Full Aggregation for completeness, so to speak. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG, VARIANCE
[ https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16589: Attachment: HIVE-16589.093.patch > Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and > COMPLETE for AVG, VARIANCE > --- > > Key: HIVE-16589 > URL: https://issues.apache.org/jira/browse/HIVE-16589 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch, > HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch, > HIVE-16589.06.patch, HIVE-16589.07.patch, HIVE-16589.08.patch, > HIVE-16589.091.patch, HIVE-16589.092.patch, HIVE-16589.093.patch, > HIVE-16589.09.patch > > > Allow Complex Types to be vectorized (since HIVE-16207: "Add support for > Complex Types in Fast SerDe" was committed). > Add more classes we vectorize AVG in preparation for fully supporting AVG > GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in > the AVG struct as input. And, add the COMPLETE mode that takes in the > Original data and produces the Full Aggregation for completeness, so to speak. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16474) Upgrade Druid version to 0.10
[ https://issues.apache.org/jira/browse/HIVE-16474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-16474: Attachment: HIVE-16474.04.patch > Upgrade Druid version to 0.10 > - > > Key: HIVE-16474 > URL: https://issues.apache.org/jira/browse/HIVE-16474 > Project: Hive > Issue Type: Task > Components: Druid integration >Reporter: Ashutosh Chauhan >Assignee: Nishant Bangarwa > Attachments: HIVE-16474.01.patch, HIVE-16474.02.patch, > HIVE-16474.03.patch, HIVE-16474.04.patch, HIVE-16474.patch > > > Druid 0.10 is out. We shall upgrade to it to take advantage of improvements > it brings. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16686) repl invocations of distcp needs additional handling
[ https://issues.apache.org/jira/browse/HIVE-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-16686: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks, [~thejas] > repl invocations of distcp needs additional handling > > > Key: HIVE-16686 > URL: https://issues.apache.org/jira/browse/HIVE-16686 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-16686.1.patch, HIVE-16686.2.patch, > HIVE-16686.3.patch > > > When REPL LOAD invokes distcp, there needs to be a way for the user invoking > REPL LOAD to pass on arguments to distcp. In addition, there is sometimes a > need for distcp to be invoked from within an impersonated context, such as > running as user "hdfs", asking distcp to preserve ownerships of individual > files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG, VARIANCE
[
https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023193#comment-16023193
]
Hive QA commented on HIVE-16589:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869672/HIVE-16589.093.patch
{color:green}SUCCESS:{color} +1 due to 27 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10749 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=236)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_15]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_short_regress]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_distinct_gby]
(batchId=157)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_15]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress]
(batchId=119)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5417/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5417/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5417/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12869672 - PreCommit-HIVE-Build
> Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and
> COMPLETE for AVG, VARIANCE
> ---
>
> Key: HIVE-16589
> URL: https://issues.apache.org/jira/browse/HIVE-16589
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch,
> HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch,
> HIVE-16589.06.patch, HIVE-16589.07.patch, HIVE-16589.08.patch,
> HIVE-16589.091.patch, HIVE-16589.092.patch, HIVE-16589.093.patch,
> HIVE-16589.09.patch
>
>
> Allow Complex Types to be vectorized (since HIVE-16207: "Add support for
> Complex Types in Fast SerDe" was committed).
> Add more classes we vectorize AVG in preparation for fully supporting AVG
> GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in
> the AVG struct as input. And, add the COMPLETE mode that takes in the
> Original data and produces the Full Aggregation for completeness, so to speak.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16474) Upgrade Druid version to 0.10
[
https://issues.apache.org/jira/browse/HIVE-16474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023205#comment-16023205
]
Hive QA commented on HIVE-16474:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869677/HIVE-16474.04.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5418/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5418/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5418/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-05-24 16:41:59.061
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-5418/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-05-24 16:41:59.063
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
dec96ca..1c891ad master -> origin/master
+ git reset --hard HEAD
HEAD is now at dec96ca HIVE-16643: BeeLine tests output should keep the
PREHOOK/POSTHOOK Input/Output orderdering (Peter Vary, reviewed by Yongzhi Chen)
+ git clean -f -d
Removing ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvgDecimal.txt
Removing ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvgDecimalMerge.txt
Removing ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvgMerge.txt
Removing ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvgTimestamp.txt
Removing ql/src/gen/vectorization/UDAFTemplates/VectorUDAFVarMerge.txt
Removing ql/src/gen/vectorization/UDAFTemplates/VectorUDAFVarTimestamp.txt
Removing
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFSumTimestamp.java
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 1c891ad HIVE-16686 : repl invocations of distcp needs additional
handling (Sushanth Sowmyan, reviewed by Thejas Nair)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-05-24 16:42:00.211
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p1
patching file
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java
patching file
druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidOutputFormat.java
patching file
druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidRecordWriter.java
patching file
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/HiveDruidSerializationModule.java
patching file
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/PeriodGranularitySerializer.java
patching file
druid-handler/src/test/org/apache/hadoop/hive/druid/DerbyConnectorTestUtility.java
patching file
druid-handler/src/test/org/apache/hadoop/hive/druid/TestDruidSerDe.java
patching file
druid-handler/src/test/org/apache/hadoop/hive/druid/TestDruidStorageHandler.java
patching file
druid-handler/src/test/org/apache/hadoop/hive/druid/TestHiveDruidQueryBasedInputFormat.java
patching file
druid-handler/src/test/org/apache/hadoop/hive/ql/io/TestDruidRecordWriter.java
patching file pom.xml
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/metastore/target/generated-sources/antlr3/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must bu
[jira] [Updated] (HIVE-16474) Upgrade Druid version to 0.10
[ https://issues.apache.org/jira/browse/HIVE-16474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-16474: Attachment: HIVE-16474.05.patch > Upgrade Druid version to 0.10 > - > > Key: HIVE-16474 > URL: https://issues.apache.org/jira/browse/HIVE-16474 > Project: Hive > Issue Type: Task > Components: Druid integration >Reporter: Ashutosh Chauhan >Assignee: Nishant Bangarwa > Attachments: HIVE-16474.01.patch, HIVE-16474.02.patch, > HIVE-16474.03.patch, HIVE-16474.04.patch, HIVE-16474.05.patch, > HIVE-16474.patch > > > Druid 0.10 is out. We shall upgrade to it to take advantage of improvements > it brings. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16752) Enable Unit test - TestDruidRecordWriter.testWrite
[ https://issues.apache.org/jira/browse/HIVE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa reassigned HIVE-16752: --- Assignee: Nishant Bangarwa > Enable Unit test - TestDruidRecordWriter.testWrite > -- > > Key: HIVE-16752 > URL: https://issues.apache.org/jira/browse/HIVE-16752 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa > > After the changes done in https://issues.apache.org/jira/browse/HIVE-16474 > the test is failing due to loading of guava classes from hive-exec jar. > this is because the hive-exec jar is a shaded jar which contains all the > dependencies. > For details see - https://github.com/apache/hive/blob/master/ql/pom.xml#L820 > "The way shade was configured since 0.13, is to override the default jar for > ql module with the shaded one but keep the same name." > So when mvn resolves the jar when running the unit test, it sees the shaded > jar which has guava also. > To resolve this, there are two ways i could find - > 1) Tweak the order of dependencies in druid > 2) Somehow add a dependency in druid-handler for non-shaded jar, but since it > has been already overridden, not sure how to do it. > 3) Use a different namespace for guava classes in hive-exec jar. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15144) JSON.org license is now CatX
[
https://issues.apache.org/jira/browse/HIVE-15144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023259#comment-16023259
]
Alan Gates commented on HIVE-15144:
---
It shouldn't. We aren't changing JSON support just changing where we get it
from.
> JSON.org license is now CatX
>
>
> Key: HIVE-15144
> URL: https://issues.apache.org/jira/browse/HIVE-15144
> Project: Hive
> Issue Type: Bug
>Reporter: Robert Kanter
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: HIVE-15144.patch
>
>
> per [update resolved legal|http://www.apache.org/legal/resolved.html#json]:
> {quote}
> CAN APACHE PRODUCTS INCLUDE WORKS LICENSED UNDER THE JSON LICENSE?
> No. As of 2016-11-03 this has been moved to the 'Category X' license list.
> Prior to this, use of the JSON Java library was allowed. See Debian's page
> for a list of alternatives.
> {quote}
> I'm not sure when this dependency was first introduced, but it looks like
> it's currently used in a few places:
> https://github.com/apache/hive/search?p=1&q=%22org.json%22&utf8=%E2%9C%93
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16743) BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()
[
https://issues.apache.org/jira/browse/HIVE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023286#comment-16023286
]
Wei Zheng commented on HIVE-16743:
--
HIVE-16753 created as a follow-up to add tests for those methods
> BitSet set() is not incorrectly used in TxnUtils.createValidCompactTxnList()
>
>
> Key: HIVE-16743
> URL: https://issues.apache.org/jira/browse/HIVE-16743
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-16743.1.patch
>
>
> The second line is problematic
> {code}
> BitSet bitSet = new BitSet(exceptions.length);
> bitSet.set(0, bitSet.length()); // for ValidCompactorTxnList, everything
> in exceptions are aborted
> {code}
> For example, exceptions' length is 2. We declare a BitSet object with initial
> size of 2 via the first line above. But that's not the actual size of the
> BitSet. So bitSet.length() will still return 0.
> The intention of the second line above is to set all the bits to true. This
> was not achieved because bitSet.set(0, bitSet.length()) is equivalent to
> bitSet.set(0, 0).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16753) Add tests that cover createValidReadTxnList and createValidCompactTxnList in TxnUtils.java
[ https://issues.apache.org/jira/browse/HIVE-16753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-16753: > Add tests that cover createValidReadTxnList and createValidCompactTxnList in > TxnUtils.java > -- > > Key: HIVE-16753 > URL: https://issues.apache.org/jira/browse/HIVE-16753 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > > Both are critical methods used in ACID paths. But there is no corresponding > tests for them. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16673) Test for HIVE-16413
[ https://issues.apache.org/jira/browse/HIVE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-16673: Description: unit test for HIVE-16413 > Test for HIVE-16413 > --- > > Key: HIVE-16673 > URL: https://issues.apache.org/jira/browse/HIVE-16673 > Project: Hive > Issue Type: Test >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-16673.1.patch > > > unit test for HIVE-16413 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16474) Upgrade Druid version to 0.10
[
https://issues.apache.org/jira/browse/HIVE-16474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023336#comment-16023336
]
Hive QA commented on HIVE-16474:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869683/HIVE-16474.05.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10747 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=236)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=151)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=238)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5419/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5419/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5419/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12869683 - PreCommit-HIVE-Build
> Upgrade Druid version to 0.10
> -
>
> Key: HIVE-16474
> URL: https://issues.apache.org/jira/browse/HIVE-16474
> Project: Hive
> Issue Type: Task
> Components: Druid integration
>Reporter: Ashutosh Chauhan
>Assignee: Nishant Bangarwa
> Attachments: HIVE-16474.01.patch, HIVE-16474.02.patch,
> HIVE-16474.03.patch, HIVE-16474.04.patch, HIVE-16474.05.patch,
> HIVE-16474.patch
>
>
> Druid 0.10 is out. We shall upgrade to it to take advantage of improvements
> it brings.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16743) BitSet set() is incorrectly used in TxnUtils.createValidCompactTxnList()
[
https://issues.apache.org/jira/browse/HIVE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-16743:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to master. Thanks Eugene for the review.
> BitSet set() is incorrectly used in TxnUtils.createValidCompactTxnList()
>
>
> Key: HIVE-16743
> URL: https://issues.apache.org/jira/browse/HIVE-16743
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-16743.1.patch
>
>
> The second line is problematic
> {code}
> BitSet bitSet = new BitSet(exceptions.length);
> bitSet.set(0, bitSet.length()); // for ValidCompactorTxnList, everything
> in exceptions are aborted
> {code}
> For example, exceptions' length is 2. We declare a BitSet object with initial
> size of 2 via the first line above. But that's not the actual size of the
> BitSet. So bitSet.length() will still return 0.
> The intention of the second line above is to set all the bits to true. This
> was not achieved because bitSet.set(0, bitSet.length()) is equivalent to
> bitSet.set(0, 0).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16743) BitSet set() is incorrectly used in TxnUtils.createValidCompactTxnList()
[
https://issues.apache.org/jira/browse/HIVE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-16743:
-
Summary: BitSet set() is incorrectly used in
TxnUtils.createValidCompactTxnList() (was: BitSet set() is not incorrectly
used in TxnUtils.createValidCompactTxnList())
> BitSet set() is incorrectly used in TxnUtils.createValidCompactTxnList()
>
>
> Key: HIVE-16743
> URL: https://issues.apache.org/jira/browse/HIVE-16743
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-16743.1.patch
>
>
> The second line is problematic
> {code}
> BitSet bitSet = new BitSet(exceptions.length);
> bitSet.set(0, bitSet.length()); // for ValidCompactorTxnList, everything
> in exceptions are aborted
> {code}
> For example, exceptions' length is 2. We declare a BitSet object with initial
> size of 2 via the first line above. But that's not the actual size of the
> BitSet. So bitSet.length() will still return 0.
> The intention of the second line above is to set all the bits to true. This
> was not achieved because bitSet.set(0, bitSet.length()) is equivalent to
> bitSet.set(0, 0).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16743) BitSet set() is incorrectly used in TxnUtils.createValidCompactTxnList()
[
https://issues.apache.org/jira/browse/HIVE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-16743:
-
Fix Version/s: 3.0.0
> BitSet set() is incorrectly used in TxnUtils.createValidCompactTxnList()
>
>
> Key: HIVE-16743
> URL: https://issues.apache.org/jira/browse/HIVE-16743
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Fix For: 3.0.0
>
> Attachments: HIVE-16743.1.patch
>
>
> The second line is problematic
> {code}
> BitSet bitSet = new BitSet(exceptions.length);
> bitSet.set(0, bitSet.length()); // for ValidCompactorTxnList, everything
> in exceptions are aborted
> {code}
> For example, exceptions' length is 2. We declare a BitSet object with initial
> size of 2 via the first line above. But that's not the actual size of the
> BitSet. So bitSet.length() will still return 0.
> The intention of the second line above is to set all the bits to true. This
> was not achieved because bitSet.set(0, bitSet.length()) is equivalent to
> bitSet.set(0, 0).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16754) LLAP: Print hive version info on llap daemon startup
[ https://issues.apache.org/jira/browse/HIVE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-16754: > LLAP: Print hive version info on llap daemon startup > > > Key: HIVE-16754 > URL: https://issues.apache.org/jira/browse/HIVE-16754 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > > For debugging purpose, print out hive version info on llap daemon startup. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14881) integrate MM tables into ACID: merge cleaner into ACID threads
[ https://issues.apache.org/jira/browse/HIVE-14881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14881: - Affects Version/s: hive-14535 > integrate MM tables into ACID: merge cleaner into ACID threads > --- > > Key: HIVE-14881 > URL: https://issues.apache.org/jira/browse/HIVE-14881 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: hive-14535 >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Attachments: HIVE-14881.1.patch, HIVE-14881.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HIVE-14881) integrate MM tables into ACID: merge cleaner into ACID threads
[ https://issues.apache.org/jira/browse/HIVE-14881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng resolved HIVE-14881. -- Resolution: Fixed Fix Version/s: hive-14535 Target Version/s: hive-14535 Committed patch 2 to hive-14535 > integrate MM tables into ACID: merge cleaner into ACID threads > --- > > Key: HIVE-14881 > URL: https://issues.apache.org/jira/browse/HIVE-14881 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: hive-14535 >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-14881.1.patch, HIVE-14881.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16673) Test for HIVE-16413
[ https://issues.apache.org/jira/browse/HIVE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023370#comment-16023370 ] Thejas M Nair commented on HIVE-16673: -- +1 > Test for HIVE-16413 > --- > > Key: HIVE-16673 > URL: https://issues.apache.org/jira/browse/HIVE-16673 > Project: Hive > Issue Type: Test >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-16673.1.patch > > > unit test for HIVE-16413 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14233) Improve vectorization for ACID by eliminating row-by-row stitching
[ https://issues.apache.org/jira/browse/HIVE-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14233: -- Fix Version/s: (was: 2.2.0) 2.3.0 > Improve vectorization for ACID by eliminating row-by-row stitching > -- > > Key: HIVE-14233 > URL: https://issues.apache.org/jira/browse/HIVE-14233 > Project: Hive > Issue Type: New Feature > Components: Transactions, Vectorization >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Labels: TODOC2.2 > Fix For: 2.3.0 > > Attachments: HIVE-14233.01.patch, HIVE-14233.02.patch, > HIVE-14233.03.patch, HIVE-14233.04.patch, HIVE-14233.05.patch, > HIVE-14233.06.patch, HIVE-14233.07.patch, HIVE-14233.08.patch, > HIVE-14233.09.patch, HIVE-14233.10.patch, HIVE-14233.11.patch, > HIVE-14233.12.patch > > > This JIRA proposes to improve vectorization for ACID by eliminating > row-by-row stitching when reading back ACID files. In the current > implementation, a vectorized row batch is created by populating the batch one > row at a time, before the vectorized batch is passed up along the operator > pipeline. This row-by-row stitching limitation was because of the fact that > the ACID insert/update/delete events from various delta files needed to be > merged together before the actual version of a given row was found out. > HIVE-14035 has enabled us to break away from that limitation by splitting > ACID update events into a combination of delete+insert. In fact, it has now > enabled us to create splits on delta files. > Building on top of HIVE-14035, this JIRA proposes to solve this earlier > bottleneck in the vectorized code path for ACID by now directly reading row > batches from the underlying ORC files and avoiding any stitching altogether. > Once a row batch is read from the split (which may be on a base/delta file), > the deleted rows will be found by cross-referencing them against a data > structure that will just keep track of deleted events (found in the > deleted_delta files). This will lead to a large performance gain when reading > ACID files in vectorized fashion, while enabling further optimizations in > future that can be done on top of that. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16754) LLAP: Print hive version info on llap daemon startup
[ https://issues.apache.org/jira/browse/HIVE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16754: - Attachment: HIVE-16754.1.patch > LLAP: Print hive version info on llap daemon startup > > > Key: HIVE-16754 > URL: https://issues.apache.org/jira/browse/HIVE-16754 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > Attachments: HIVE-16754.1.patch > > > For debugging purpose, print out hive version info on llap daemon startup. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16754) LLAP: Print hive version info on llap daemon startup
[ https://issues.apache.org/jira/browse/HIVE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16754: - Status: Patch Available (was: Open) > LLAP: Print hive version info on llap daemon startup > > > Key: HIVE-16754 > URL: https://issues.apache.org/jira/browse/HIVE-16754 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > Attachments: HIVE-16754.1.patch > > > For debugging purpose, print out hive version info on llap daemon startup. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16742) cap the number of reducers for LLAP at the configured value
[ https://issues.apache.org/jira/browse/HIVE-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16742: Fix Version/s: 3.0.0 > cap the number of reducers for LLAP at the configured value > --- > > Key: HIVE-16742 > URL: https://issues.apache.org/jira/browse/HIVE-16742 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Sergey Shelukhin > Fix For: 3.0.0 > > Attachments: HIVE-16742.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16742) cap the number of reducers for LLAP at the configured value
[ https://issues.apache.org/jira/browse/HIVE-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023379#comment-16023379 ] Sergey Shelukhin commented on HIVE-16742: - Thanks, done > cap the number of reducers for LLAP at the configured value > --- > > Key: HIVE-16742 > URL: https://issues.apache.org/jira/browse/HIVE-16742 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Sergey Shelukhin > Fix For: 3.0.0 > > Attachments: HIVE-16742.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG, VARIANCE
[ https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023381#comment-16023381 ] Matt McCline commented on HIVE-16589: - Related test failures: vectorization_15 (overflow in vector modulus division by 0; vectorization_short_regress different query result; vectorized_distinct_gby.q different query result. > Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and > COMPLETE for AVG, VARIANCE > --- > > Key: HIVE-16589 > URL: https://issues.apache.org/jira/browse/HIVE-16589 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch, > HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch, > HIVE-16589.06.patch, HIVE-16589.07.patch, HIVE-16589.08.patch, > HIVE-16589.091.patch, HIVE-16589.092.patch, HIVE-16589.093.patch, > HIVE-16589.09.patch > > > Allow Complex Types to be vectorized (since HIVE-16207: "Add support for > Complex Types in Fast SerDe" was committed). > Add more classes we vectorize AVG in preparation for fully supporting AVG > GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in > the AVG struct as input. And, add the COMPLETE mode that takes in the > Original data and produces the Full Aggregation for completeness, so to speak. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16751) Support different types for grouping columns in GroupBy Druid queries
[ https://issues.apache.org/jira/browse/HIVE-16751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-16751: --- Attachment: (was: HIVE-16751.patch) > Support different types for grouping columns in GroupBy Druid queries > - > > Key: HIVE-16751 > URL: https://issues.apache.org/jira/browse/HIVE-16751 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Calcite 1.13 pushes EXTRACT and FLOOR function to Druid as an extraction > function (cf CALCITE-1758). Originally, we were assuming that all group by > columns in a druid query were of STRING type; however, this will not true > anymore (result of EXTRACT is an INT and result of FLOOR a TIMESTAMP). > When we upgrade to Calcite 1.13, we will need to extend the DruidSerDe to > handle these functions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16751) Support different types for grouping columns in GroupBy Druid queries
[ https://issues.apache.org/jira/browse/HIVE-16751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-16751: --- Attachment: HIVE-16751.patch > Support different types for grouping columns in GroupBy Druid queries > - > > Key: HIVE-16751 > URL: https://issues.apache.org/jira/browse/HIVE-16751 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-16751.patch > > > Calcite 1.13 pushes EXTRACT and FLOOR function to Druid as an extraction > function (cf CALCITE-1758). Originally, we were assuming that all group by > columns in a druid query were of STRING type; however, this will not true > anymore (result of EXTRACT is an INT and result of FLOOR a TIMESTAMP). > When we upgrade to Calcite 1.13, we will need to extend the DruidSerDe to > handle these functions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16474) Upgrade Druid version to 0.10
[ https://issues.apache.org/jira/browse/HIVE-16474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023397#comment-16023397 ] Nishant Bangarwa commented on HIVE-16474: - test failures does not seem related to above changes. > Upgrade Druid version to 0.10 > - > > Key: HIVE-16474 > URL: https://issues.apache.org/jira/browse/HIVE-16474 > Project: Hive > Issue Type: Task > Components: Druid integration >Reporter: Ashutosh Chauhan >Assignee: Nishant Bangarwa > Attachments: HIVE-16474.01.patch, HIVE-16474.02.patch, > HIVE-16474.03.patch, HIVE-16474.04.patch, HIVE-16474.05.patch, > HIVE-16474.patch > > > Druid 0.10 is out. We shall upgrade to it to take advantage of improvements > it brings. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-13690) Shade guava in hive-exec fat jar
[
https://issues.apache.org/jira/browse/HIVE-13690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023399#comment-16023399
]
Hive QA commented on HIVE-13690:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802303/HIVE-13690.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 836 failed/errored test(s), 6160 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.org.apache.hadoop.hive.cli.TestBeeLineDriver
(batchId=236)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_like]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_blobstore_to_blobstore]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_blobstore_to_hdfs]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_hdfs_to_blobstore]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_blobstore]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_local]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_warehouse]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_local_to_blobstore]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore_nonpart]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_local]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse_nonpart]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_local_to_blobstore]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_blobstore_to_blobstore]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_empty_into_blobstore]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[join2]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[join]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[map_join]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[map_join_on_filter]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[nested_outer_join]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_buckets]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_nonpart]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_part]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_nonstd_partitions_loc]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_general_queries]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_matchpath]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_orcfile]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_persistence]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_rcfile]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_seqfile]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_buckets]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_format_nonpart]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_format_part]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_nonstd_partitions_loc]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
(batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[zero_rows_blobstore]
(batchId=2
[jira] [Updated] (HIVE-16673) Test for HIVE-16413
[ https://issues.apache.org/jira/browse/HIVE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-16673: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > Test for HIVE-16413 > --- > > Key: HIVE-16673 > URL: https://issues.apache.org/jira/browse/HIVE-16673 > Project: Hive > Issue Type: Test >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Fix For: 3.0.0 > > Attachments: HIVE-16673.1.patch > > > unit test for HIVE-16413 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16754) LLAP: Print hive version info on llap daemon startup
[
https://issues.apache.org/jira/browse/HIVE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023512#comment-16023512
]
Hive QA commented on HIVE-16754:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869702/HIVE-16754.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10752 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=151)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=288)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5421/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5421/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5421/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12869702 - PreCommit-HIVE-Build
> LLAP: Print hive version info on llap daemon startup
>
>
> Key: HIVE-16754
> URL: https://issues.apache.org/jira/browse/HIVE-16754
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Attachments: HIVE-16754.1.patch
>
>
> For debugging purpose, print out hive version info on llap daemon startup.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16575) Support for 'UNIQUE' and 'NOT NULL' constraints
[
https://issues.apache.org/jira/browse/HIVE-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023542#comment-16023542
]
Ashutosh Chauhan commented on HIVE-16575:
-
+1
> Support for 'UNIQUE' and 'NOT NULL' constraints
> ---
>
> Key: HIVE-16575
> URL: https://issues.apache.org/jira/browse/HIVE-16575
> Project: Hive
> Issue Type: New Feature
> Components: CBO, Logical Optimizer, Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-16575.01.patch, HIVE-16575.02.patch,
> HIVE-16575.03.patch, HIVE-16575.04.patch, HIVE-16575.patch
>
>
> Follow-up on HIVE-13076.
> This issue add support for SQL 'UNIQUE' and 'NOT NULL' constraints when we
> create a table / alter a table
> (https://www.postgresql.org/docs/9.6/static/sql-createtable.html).
> As with PK and FK constraints, currently we do not enforce them; thus, the
> constraints need to use the DISABLE option, but they will be stored and can
> be enabled for rewriting/optimization using RELY.
> This patch also adds support for inlining the constraints next to the column
> type definition, i.e., 'column constraints'.
> Some examples of the extension to the syntax included in the patch:
> {code:sql}
> CREATE TABLE table3 (x string NOT NULL DISABLE, PRIMARY KEY (x) DISABLE,
> CONSTRAINT fk1 FOREIGN KEY (x) REFERENCES table2(a) DISABLE);
> CREATE TABLE table4 (x string CONSTRAINT nn4_1 NOT NULL DISABLE, y string
> CONSTRAINT nn4_2 NOT NULL DISABLE, UNIQUE (x) DISABLE, CONSTRAINT fk2 FOREIGN
> KEY (x) REFERENCES table2(a) DISABLE,
> CONSTRAINT fk3 FOREIGN KEY (y) REFERENCES table2(a) DISABLE);
> CREATE TABLE table12 (a STRING CONSTRAINT nn12_1 NOT NULL DISABLE NORELY, b
> STRING);
> CREATE TABLE table13 (a STRING NOT NULL DISABLE RELY, b STRING);
> CREATE TABLE table14 (a STRING CONSTRAINT nn14_1 NOT NULL DISABLE RELY, b
> STRING);
> CREATE TABLE table15 (a STRING REFERENCES table4(x) DISABLE, b STRING);
> CREATE TABLE table16 (a STRING CONSTRAINT nn16_1 REFERENCES table4(x) DISABLE
> RELY, b STRING);
> ALTER TABLE table16 CHANGE a a STRING REFERENCES table4(x) DISABLE NOVALIDATE;
> ALTER TABLE table12 CHANGE COLUMN b b STRING CONSTRAINT nn12_2 NOT NULL
> DISABLE NOVALIDATE;
> ALTER TABLE table13 CHANGE b b STRING NOT NULL DISABLE NOVALIDATE;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16746) Reduce number of index lookups for same table in IndexWhereTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023561#comment-16023561
]
Ashutosh Chauhan commented on HIVE-16746:
-
+1
> Reduce number of index lookups for same table in IndexWhereTaskDispatcher
> -
>
> Key: HIVE-16746
> URL: https://issues.apache.org/jira/browse/HIVE-16746
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16746.1.patch
>
>
> {{IndexWhereTaskDispatcher}} is used when
> {{hive.optimize.index.filter=true}}. It lists all indices for the table and
> depending on the query complexity, this ends up being in the hotpath. For
> e.g, Q14 explain plan takes 180-200 seconds and this index querying multiple
> times for same tables take up 30-40 seconds. This function was invoked around
> 24000 times for same set of tables.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16755) LLAP IO: incorrect assert may trigger in tests
[ https://issues.apache.org/jira/browse/HIVE-16755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-16755: --- > LLAP IO: incorrect assert may trigger in tests > -- > > Key: HIVE-16755 > URL: https://issues.apache.org/jira/browse/HIVE-16755 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16755) LLAP IO: incorrect assert may trigger in tests
[ https://issues.apache.org/jira/browse/HIVE-16755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16755: Status: Patch Available (was: Open) > LLAP IO: incorrect assert may trigger in tests > -- > > Key: HIVE-16755 > URL: https://issues.apache.org/jira/browse/HIVE-16755 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16755.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16755) LLAP IO: incorrect assert may trigger in tests
[ https://issues.apache.org/jira/browse/HIVE-16755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16755: Attachment: HIVE-16755.patch [~prasanth_j] [~wzheng] can you take a look? > LLAP IO: incorrect assert may trigger in tests > -- > > Key: HIVE-16755 > URL: https://issues.apache.org/jira/browse/HIVE-16755 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16755.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16754) LLAP: Print hive version info on llap daemon startup
[ https://issues.apache.org/jira/browse/HIVE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023608#comment-16023608 ] Prasanth Jayachandran commented on HIVE-16754: -- [~sseth] minor patch. could you please take a look? > LLAP: Print hive version info on llap daemon startup > > > Key: HIVE-16754 > URL: https://issues.apache.org/jira/browse/HIVE-16754 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > Attachments: HIVE-16754.1.patch > > > For debugging purpose, print out hive version info on llap daemon startup. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16413) Create table as select does not check ownership of the location
[
https://issues.apache.org/jira/browse/HIVE-16413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023655#comment-16023655
]
Thejas M Nair commented on HIVE-16413:
--
Also committed to branch-2 and branch-1
> Create table as select does not check ownership of the location
> ---
>
> Key: HIVE-16413
> URL: https://issues.apache.org/jira/browse/HIVE-16413
> Project: Hive
> Issue Type: Bug
> Components: Authorization, SQLStandardAuthorization
>Affects Versions: 1.3.0, 1.2.2, 2.1.1
> Environment: hive-1.2.2, with following conf:
> hive.security.authorization.enabled: true
> hive.security.authorization.manager:
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory
> hive.security.authenticator.manager:
> org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Fix For: 1.3.0, 3.0.0, 2.4.0
>
> Attachments: HIVE-16413.patch
>
>
> 1. following statement failed:
> {code}
> create table foo(id int) location 'hdfs:///tmp/foo';
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: Principal [name=userx, type=USER] does not have following
> privileges for operation CREATETABLE [[OBJECT OWNERSHIP] on Object
> [type=DFS_URI, name=hdfs://hacluster/tmp/foo]] (state=42000,code=4)
> {code}
> 2. but when use create table as select, it successed:
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> create table foo location
> 'hdfs:///tmp/foo' as select * from xxx2;
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : number of splits:1
> INFO : Submitting tokens for job: job_1491449632882_0094
> INFO : Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:hacluster
> INFO : The url to track the job:
> https://189-39-151-44:26001/proxy/application_1491449632882_0094/
> INFO : Starting Job = job_1491449632882_0094, Tracking URL =
> https://189-39-151-44:26001/proxy/application_1491449632882_0094/
> INFO : Kill Command = /opt/hive-1.3.0/bin/..//../hadoop/bin/hadoop job
> -kill job_1491449632882_0094
> INFO : Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> INFO : 2017-04-10 09:44:49,185 Stage-1 map = 0%, reduce = 0%
> INFO : 2017-04-10 09:44:57,202 Stage-1 map = 100%, reduce = 0%, Cumulative
> CPU 1.98 sec
> INFO : MapReduce Total cumulative CPU time: 1 seconds 980 msec
> INFO : Ended Job = job_1491449632882_0094
> INFO : Stage-3 is selected by condition resolver.
> INFO : Stage-2 is filtered out by condition resolver.
> INFO : Stage-4 is filtered out by condition resolver.
> INFO : Moving data to directory
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10001
> from
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10003
> INFO : Moving data to directory hdfs:/tmp/foo from
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10001
> No rows affected (26.969 seconds)
> {code}
> 3. and the table location is hdfs://hacluster/tmp/foo :
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> desc formatted foo;
> +---+---+---+--+
> | col_name| data_type
> |comment|
> +---+---+---+--+
> | # col_name| data_type
> | comment |
> | | NULL
> | NULL |
> | id| int
> | |
> | | NULL
> | NULL |
> | # Detailed Table Information | NULL
> | NULL |
> | Database: | default
> | NULL |
> | Owner:| userx
> | NULL |
> | CreateTime: | Mon Apr 10 09:44:59 CST 2017
> | NULL |
> | LastAccessTime: | UNKNOWN
> | NULL |
> | Protect Mode: | None
[jira] [Updated] (HIVE-16413) Create table as select does not check ownership of the location
[
https://issues.apache.org/jira/browse/HIVE-16413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-16413:
-
Fix Version/s: 2.4.0
1.3.0
> Create table as select does not check ownership of the location
> ---
>
> Key: HIVE-16413
> URL: https://issues.apache.org/jira/browse/HIVE-16413
> Project: Hive
> Issue Type: Bug
> Components: Authorization, SQLStandardAuthorization
>Affects Versions: 1.3.0, 1.2.2, 2.1.1
> Environment: hive-1.2.2, with following conf:
> hive.security.authorization.enabled: true
> hive.security.authorization.manager:
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory
> hive.security.authenticator.manager:
> org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Fix For: 1.3.0, 3.0.0, 2.4.0
>
> Attachments: HIVE-16413.patch
>
>
> 1. following statement failed:
> {code}
> create table foo(id int) location 'hdfs:///tmp/foo';
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: Principal [name=userx, type=USER] does not have following
> privileges for operation CREATETABLE [[OBJECT OWNERSHIP] on Object
> [type=DFS_URI, name=hdfs://hacluster/tmp/foo]] (state=42000,code=4)
> {code}
> 2. but when use create table as select, it successed:
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> create table foo location
> 'hdfs:///tmp/foo' as select * from xxx2;
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : number of splits:1
> INFO : Submitting tokens for job: job_1491449632882_0094
> INFO : Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:hacluster
> INFO : The url to track the job:
> https://189-39-151-44:26001/proxy/application_1491449632882_0094/
> INFO : Starting Job = job_1491449632882_0094, Tracking URL =
> https://189-39-151-44:26001/proxy/application_1491449632882_0094/
> INFO : Kill Command = /opt/hive-1.3.0/bin/..//../hadoop/bin/hadoop job
> -kill job_1491449632882_0094
> INFO : Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> INFO : 2017-04-10 09:44:49,185 Stage-1 map = 0%, reduce = 0%
> INFO : 2017-04-10 09:44:57,202 Stage-1 map = 100%, reduce = 0%, Cumulative
> CPU 1.98 sec
> INFO : MapReduce Total cumulative CPU time: 1 seconds 980 msec
> INFO : Ended Job = job_1491449632882_0094
> INFO : Stage-3 is selected by condition resolver.
> INFO : Stage-2 is filtered out by condition resolver.
> INFO : Stage-4 is filtered out by condition resolver.
> INFO : Moving data to directory
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10001
> from
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10003
> INFO : Moving data to directory hdfs:/tmp/foo from
> hdfs://hacluster/user/hive/warehouse/.hive-staging_hive_2017-04-10_09-44-32_462_4902211653847168915-1/-ext-10001
> No rows affected (26.969 seconds)
> {code}
> 3. and the table location is hdfs://hacluster/tmp/foo :
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> desc formatted foo;
> +---+---+---+--+
> | col_name| data_type
> |comment|
> +---+---+---+--+
> | # col_name| data_type
> | comment |
> | | NULL
> | NULL |
> | id| int
> | |
> | | NULL
> | NULL |
> | # Detailed Table Information | NULL
> | NULL |
> | Database: | default
> | NULL |
> | Owner:| userx
> | NULL |
> | CreateTime: | Mon Apr 10 09:44:59 CST 2017
> | NULL |
> | LastAccessTime: | UNKNOWN
> | NULL |
> | Protect Mode: | None
> | NULL
[jira] [Updated] (HIVE-16673) Test for HIVE-16413
[ https://issues.apache.org/jira/browse/HIVE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-16673: - Fix Version/s: 2.4.0 1.3.0 > Test for HIVE-16413 > --- > > Key: HIVE-16673 > URL: https://issues.apache.org/jira/browse/HIVE-16673 > Project: Hive > Issue Type: Test >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Fix For: 1.3.0, 3.0.0, 2.4.0 > > Attachments: HIVE-16673.1.patch > > > unit test for HIVE-16413 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16673) Test for HIVE-16413
[ https://issues.apache.org/jira/browse/HIVE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023662#comment-16023662 ] Thejas M Nair commented on HIVE-16673: -- Committed to master, branch-2 and branch-1 Thanks [~kgyrtkirk] > Test for HIVE-16413 > --- > > Key: HIVE-16673 > URL: https://issues.apache.org/jira/browse/HIVE-16673 > Project: Hive > Issue Type: Test >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Fix For: 1.3.0, 3.0.0, 2.4.0 > > Attachments: HIVE-16673.1.patch > > > unit test for HIVE-16413 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16755) LLAP IO: incorrect assert may trigger in tests
[ https://issues.apache.org/jira/browse/HIVE-16755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023670#comment-16023670 ] Prasanth Jayachandran commented on HIVE-16755: -- +1 > LLAP IO: incorrect assert may trigger in tests > -- > > Key: HIVE-16755 > URL: https://issues.apache.org/jira/browse/HIVE-16755 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16755.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16689) Correlated scalar subquery with comparison to constant in predicate fails
[
https://issues.apache.org/jira/browse/HIVE-16689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-16689:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Correlated scalar subquery with comparison to constant in predicate fails
> -
>
> Key: HIVE-16689
> URL: https://issues.apache.org/jira/browse/HIVE-16689
> Project: Hive
> Issue Type: Bug
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16689.1.patch, HIVE-16689.2.patch
>
>
> *Reproducer*
> {code:sql}
> CREATE TABLE `item`(
> `i_item_sk` int,
> `i_item_id` char(16),
> `i_rec_start_date` date,
> `i_rec_end_date` date,
> `i_item_desc` varchar(200),
> `i_current_price` decimal(7,2),
> `i_wholesale_cost` decimal(7,2),
> `i_brand_id` int,
> `i_brand` char(50),
> `i_class_id` int,
> `i_class` char(50),
> `i_category_id` int,
> `i_category` char(50),
> `i_manufact_id` int,
> `i_manufact` char(50),
> `i_size` char(20),
> `i_formulation` char(20),
> `i_color` char(20),
> `i_units` char(10),
> `i_container` char(10),
> `i_manager_id` int,
> `i_product_name` char(50));
> select count(*)
> from item i1
> where
>(select count(*)
>from item
>where (i_manufact = i1.i_manufact)) > 0;
> {code}
> *Error stack*
> {code}
> org.apache.calcite.util.mapping.Mappings$NoElementException: source #0 has no
> target in mapping [size=0, sourceCount=1, targetCount=1, elements=[]]
> at
> org.apache.calcite.util.mapping.Mappings$AbstractMapping.getTarget(Mappings.java:874)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at
> org.apache.calcite.sql2rel.RelFieldTrimmer$2.handle(RelFieldTrimmer.java:304)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at
> org.apache.calcite.sql2rel.CorrelationReferenceFinder$MyRexVisitor.visitFieldAccess(CorrelationReferenceFinder.java:59)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at
> org.apache.calcite.sql2rel.CorrelationReferenceFinder$MyRexVisitor.visitFieldAccess(CorrelationReferenceFinder.java:50)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at org.apache.calcite.rex.RexFieldAccess.accept(RexFieldAccess.java:81)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at org.apache.calcite.rex.RexShuttle.visitList(RexShuttle.java:148)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at org.apache.calcite.rex.RexShuttle.visitCall(RexShuttle.java:97)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at org.apache.calcite.rex.RexShuttle.visitCall(RexShuttle.java:36)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:104)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at org.apache.calcite.rex.RexShuttle.apply(RexShuttle.java:279)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at org.apache.calcite.rel.core.Filter.accept(Filter.java:103)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at
> org.apache.calcite.sql2rel.CorrelationReferenceFinder.visit(CorrelationReferenceFinder.java:44)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.accept(HiveFilter.java:116)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:55)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at
> org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:69)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:131)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at
> org.apache.calcite.sql2rel.CorrelationReferenceFinder.visit(CorrelationReferenceFinder.java:43)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.accept(HiveProject.java:198)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:55)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at
> org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:69)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:131)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at
> org.apache.calcite.sql2rel.CorrelationReferenceFinder.visit(CorrelationReferenceFinder.java:43)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at
> org.apache.calcite.rel.AbstractRelNode.accept(AbstractRelNode.java:279)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at
> org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:55)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at
> org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:69)
> ~[calcite-core-1.12.0.jar:1.12.0]
> at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImp
[jira] [Commented] (HIVE-16754) LLAP: Print hive version info on llap daemon startup
[ https://issues.apache.org/jira/browse/HIVE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023708#comment-16023708 ] Siddharth Seth commented on HIVE-16754: --- +1 > LLAP: Print hive version info on llap daemon startup > > > Key: HIVE-16754 > URL: https://issues.apache.org/jira/browse/HIVE-16754 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > Attachments: HIVE-16754.1.patch > > > For debugging purpose, print out hive version info on llap daemon startup. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16745) Syntax error in 041-HIVE-16556.mysql.sql script
[ https://issues.apache.org/jira/browse/HIVE-16745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023719#comment-16023719 ] Naveen Gangam commented on HIVE-16745: -- Looks good to me. +1 > Syntax error in 041-HIVE-16556.mysql.sql script > --- > > Key: HIVE-16745 > URL: https://issues.apache.org/jira/browse/HIVE-16745 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Minor > Attachments: HIVE-16745.01.patch > > > 041-HIVE-16556.mysql.sql has a syntax error which was introduced with > HIVE-16711 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16755) LLAP IO: incorrect assert may trigger in tests
[
https://issues.apache.org/jira/browse/HIVE-16755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023725#comment-16023725
]
Hive QA commented on HIVE-16755:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869720/HIVE-16755.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10754 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=236)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=236)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=151)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5422/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5422/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5422/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12869720 - PreCommit-HIVE-Build
> LLAP IO: incorrect assert may trigger in tests
> --
>
> Key: HIVE-16755
> URL: https://issues.apache.org/jira/browse/HIVE-16755
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16755.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15160) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15160:
---
Affects Version/s: 2.3.0
2.2.0
2.0.0
2.1.0
> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 3.0.0
>
> Attachments: HIVE-15160.01.patch, HIVE-15160.02.patch,
> HIVE-15160.04.patch, HIVE-15160.05.patch, HIVE-15160.06.patch,
> HIVE-15160.07.patch, HIVE-15160.08.patch, HIVE-15160.09.patch,
> HIVE-15160.09.patch, HIVE-15160.10.patch, HIVE-15160.11.patch,
> HIVE-15160.12.patch, HIVE-15160.13.patch, HIVE-15160.14.patch,
> HIVE-15160.15.patch, HIVE-15160.16.patch, HIVE-15160.17.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15160) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15160:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 3.0.0
>
> Attachments: HIVE-15160.01.patch, HIVE-15160.02.patch,
> HIVE-15160.04.patch, HIVE-15160.05.patch, HIVE-15160.06.patch,
> HIVE-15160.07.patch, HIVE-15160.08.patch, HIVE-15160.09.patch,
> HIVE-15160.09.patch, HIVE-15160.10.patch, HIVE-15160.11.patch,
> HIVE-15160.12.patch, HIVE-15160.13.patch, HIVE-15160.14.patch,
> HIVE-15160.15.patch, HIVE-15160.16.patch, HIVE-15160.17.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15160) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15160:
---
Fix Version/s: 3.0.0
> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 3.0.0
>
> Attachments: HIVE-15160.01.patch, HIVE-15160.02.patch,
> HIVE-15160.04.patch, HIVE-15160.05.patch, HIVE-15160.06.patch,
> HIVE-15160.07.patch, HIVE-15160.08.patch, HIVE-15160.09.patch,
> HIVE-15160.09.patch, HIVE-15160.10.patch, HIVE-15160.11.patch,
> HIVE-15160.12.patch, HIVE-15160.13.patch, HIVE-15160.14.patch,
> HIVE-15160.15.patch, HIVE-15160.16.patch, HIVE-15160.17.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HIVE-16683) ORC WriterVersion gets ArrayIndexOutOfBoundsException on newer ORC files
[ https://issues.apache.org/jira/browse/HIVE-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley resolved HIVE-16683. -- Resolution: Fixed Fix Version/s: 2.2.0 > ORC WriterVersion gets ArrayIndexOutOfBoundsException on newer ORC files > > > Key: HIVE-16683 > URL: https://issues.apache.org/jira/browse/HIVE-16683 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.1, 2.2.0 >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.2.0 > > Attachments: HIVE-16683.patch > > > This only impacts branch-2.1 and branch-2.2, because it has been fixed in the > ORC project's code base via ORC-125. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16164) Provide mechanism for passing HMS notification ID between transactional and non-transactional listeners.
[
https://issues.apache.org/jira/browse/HIVE-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023816#comment-16023816
]
Alexander Kolbasov commented on HIVE-16164:
---
[~sushanth] The tricky part with Kafka messages would be dealing with
transaction rollbacks - so the event generation should happen outside of the
transaction, but then there is an interesting question of what to do if Kafka
notification fails.
> Provide mechanism for passing HMS notification ID between transactional and
> non-transactional listeners.
>
>
> Key: HIVE-16164
> URL: https://issues.apache.org/jira/browse/HIVE-16164
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Fix For: 2.3.0, 3.0.0, 2.4.0
>
> Attachments: HIVE-16164.1.patch, HIVE-16164.2.patch,
> HIVE-16164.3.patch, HIVE-16164.6.patch, HIVE-16164.7.patch, HIVE-16164.8.patch
>
>
> The HMS DB notification listener currently stores an event ID on the HMS
> backend DB so that external applications (such as backup apps) can request
> incremental notifications based on the last event ID requested.
> The HMS DB notification and backup applications are asynchronous. However,
> there are sometimes that applications may be required to be in sync with the
> latest HMS event in order to process an action. These applications will
> provide a listener implementation that is called by the HMS after an HMS
> transaction happened.
> The problem is that the listener running after the transaction (or during the
> non-transactional context) may need the DB event ID in order to sync all
> events happened previous to that event ID, but this ID is never passed to the
> non-transactional listeners.
> We can pass this event information through the EnvironmentContext found on
> each ListenerEvent implementations (such as CreateTableEvent), and send the
> EnvironmentContext to the non-transactional listeners to get the event ID.
> The DbNotificactionListener already knows the event ID after calling the
> ObjectStore.addNotificationEvent(). We just need to set this event ID to the
> EnvironmentContext from each of the event notifications and make sure that
> this EnvironmentContext is sent to the non-transactional listeners.
> Here's the code example when creating a table on {{create_table_core}}:
> {noformat}
> ms.createTable(tbl);
> if (transactionalListeners.size() > 0) {
> CreateTableEvent createTableEvent = new CreateTableEvent(tbl, true, this);
> createTableEvent.setEnvironmentContext(envContext);
> for (MetaStoreEventListener transactionalListener :
> transactionalListeners) {
> transactionalListener.onCreateTable(createTableEvent); // <-
> Here the notification ID is generated
> }
> }
> success = ms.commitTransaction();
> } finally {
> if (!success) {
> ms.rollbackTransaction();
> if (madeDir) {
> wh.deleteDir(tblPath, true);
> }
> }
> for (MetaStoreEventListener listener : listeners) {
> CreateTableEvent createTableEvent =
> new CreateTableEvent(tbl, success, this);
> createTableEvent.setEnvironmentContext(envContext);
> listener.onCreateTable(createTableEvent);// <-
> Here we would like to consume notification ID
> }
> {noformat}
> We could use a specific key name that will be used on the EnvironmentContext,
> such as DB_NOTIFICATION_EVENT_ID.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16746) Reduce number of index lookups for same table in IndexWhereTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-16746:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Thanks for the review [~ashutoshc]. Committed to master.
> Reduce number of index lookups for same table in IndexWhereTaskDispatcher
> -
>
> Key: HIVE-16746
> URL: https://issues.apache.org/jira/browse/HIVE-16746
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-16746.1.patch
>
>
> {{IndexWhereTaskDispatcher}} is used when
> {{hive.optimize.index.filter=true}}. It lists all indices for the table and
> depending on the query complexity, this ends up being in the hotpath. For
> e.g, Q14 explain plan takes 180-200 seconds and this index querying multiple
> times for same tables take up 30-40 seconds. This function was invoked around
> 24000 times for same set of tables.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16754) LLAP: Print hive version info on llap daemon startup
[ https://issues.apache.org/jira/browse/HIVE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16754: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks Sid for the review! > LLAP: Print hive version info on llap daemon startup > > > Key: HIVE-16754 > URL: https://issues.apache.org/jira/browse/HIVE-16754 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-16754.1.patch > > > For debugging purpose, print out hive version info on llap daemon startup. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16369) Vectorization: Support PTF (Part 1: No Custom Window Framing -- Default Only)
[ https://issues.apache.org/jira/browse/HIVE-16369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16369: Status: In Progress (was: Patch Available) > Vectorization: Support PTF (Part 1: No Custom Window Framing -- Default Only) > - > > Key: HIVE-16369 > URL: https://issues.apache.org/jira/browse/HIVE-16369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16369.01.patch, HIVE-16369.02.patch, > HIVE-16369.04.patch, HIVE-16369.05.patch.tar.gz, HIVE-16369.06.patch, > HIVE-16369.07.patch, HIVE-16369.091.patch, HIVE-16369.092.patch, > HIVE-16369.093.patch, HIVE-16369.094.patch, HIVE-16369.095.patch, > HIVE-16369.09.patch > > > Vectorize a submit of current PTFOperator window function support. The first > phase doesn't include custom PRECEDING / FOLLOWING window frame clauses. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16369) Vectorization: Support PTF (Part 1: No Custom Window Framing -- Default Only)
[ https://issues.apache.org/jira/browse/HIVE-16369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16369: Status: Patch Available (was: In Progress) > Vectorization: Support PTF (Part 1: No Custom Window Framing -- Default Only) > - > > Key: HIVE-16369 > URL: https://issues.apache.org/jira/browse/HIVE-16369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16369.01.patch, HIVE-16369.02.patch, > HIVE-16369.04.patch, HIVE-16369.05.patch.tar.gz, HIVE-16369.06.patch, > HIVE-16369.07.patch, HIVE-16369.091.patch, HIVE-16369.092.patch, > HIVE-16369.093.patch, HIVE-16369.094.patch, HIVE-16369.095.patch, > HIVE-16369.097.patch, HIVE-16369.09.patch > > > Vectorize a submit of current PTFOperator window function support. The first > phase doesn't include custom PRECEDING / FOLLOWING window frame clauses. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16369) Vectorization: Support PTF (Part 1: No Custom Window Framing -- Default Only)
[ https://issues.apache.org/jira/browse/HIVE-16369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16369: Attachment: HIVE-16369.097.patch > Vectorization: Support PTF (Part 1: No Custom Window Framing -- Default Only) > - > > Key: HIVE-16369 > URL: https://issues.apache.org/jira/browse/HIVE-16369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16369.01.patch, HIVE-16369.02.patch, > HIVE-16369.04.patch, HIVE-16369.05.patch.tar.gz, HIVE-16369.06.patch, > HIVE-16369.07.patch, HIVE-16369.091.patch, HIVE-16369.092.patch, > HIVE-16369.093.patch, HIVE-16369.094.patch, HIVE-16369.095.patch, > HIVE-16369.097.patch, HIVE-16369.09.patch > > > Vectorize a submit of current PTFOperator window function support. The first > phase doesn't include custom PRECEDING / FOLLOWING window frame clauses. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15300) Reuse table information in SemanticAnalyzer::getMetaData to reduce compilation time
[
https://issues.apache.org/jira/browse/HIVE-15300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023902#comment-16023902
]
Hive QA commented on HIVE-15300:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869570/HIVE-15300.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10756 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=237)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_3] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_4] (batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_cond_pushdown2]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_onview]
(batchId=55)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
(batchId=150)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5423/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5423/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5423/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12869570 - PreCommit-HIVE-Build
> Reuse table information in SemanticAnalyzer::getMetaData to reduce
> compilation time
> ---
>
> Key: HIVE-15300
> URL: https://issues.apache.org/jira/browse/HIVE-15300
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-15300.1.patch, HIVE-15300.2.patch,
> HIVE-15300.3.patch, HIVE-15300.4.patch
>
>
> E.g Q88 in tpc-ds takes lots of time to compile and it ends up getting the
> table details for the same table repeatedly. It took 20+seconds to compile
> the query.
> It would be good to reuse the table information in
> SemanticAnalyzer::getMetadata.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15212: - Status: Open (was: Patch Available) > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15212: - Attachment: HIVE-15212.05.patch ACID integration is done. Uploading patch 5 for testing. > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-15212: Assignee: Wei Zheng (was: Sergey Shelukhin) > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15212: - Status: Patch Available (was: Open) > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16369) Vectorization: Support PTF (Part 1: No Custom Window Framing -- Default Only)
[
https://issues.apache.org/jira/browse/HIVE-16369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023955#comment-16023955
]
Hive QA commented on HIVE-16369:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869743/HIVE-16369.097.patch
{color:green}SUCCESS:{color} +1 due to 13 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 39 failed/errored test(s), 10768 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=237)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_reference_windowed]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_windowing_expressions]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_windowing_gby2]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_windowing_gby]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_windowing_multipartitioning]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_windowing_order_null]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_windowing_range_multiorder]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_windowing_rank]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_windowing_streaming]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_windowing_windowspec4]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_windowing_windowspec]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_navfn]
(batchId=65)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_navfn]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join1]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join2]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join3]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join4]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join5]
(batchId=169)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in]
(batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_count_distinct]
(batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_data_types]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_aggregate]
(batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_distinct_2]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_groupby_3]
(batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_left_outer_join]
(batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_mapjoin_reduce]
(batchId=134)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_orderby_5]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_string_concat]
(batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_0]
(batchId=135)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_5]
(batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_nested_udf]
(batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_part]
(batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_case]
(batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_nested_mapjoin]
(batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf]
(batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_timestamp_funcs]
(batchId=113)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5424/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5424/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5424/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 39 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT
[jira] [Updated] (HIVE-14990) run all tests for MM tables and fix the issues that are found
[ https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14990: - Status: Patch Available (was: Reopened) > run all tests for MM tables and fix the issues that are found > - > > Key: HIVE-14990 > URL: https://issues.apache.org/jira/browse/HIVE-14990 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch, > HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch, > HIVE-14990.05.patch, HIVE-14990.05.patch, HIVE-14990.06.patch, > HIVE-14990.06.patch, HIVE-14990.07.patch, HIVE-14990.08.patch, > HIVE-14990.09.patch, HIVE-14990.10.patch, HIVE-14990.10.patch, > HIVE-14990.10.patch, HIVE-14990.12.patch, HIVE-14990.13.patch, > HIVE-14990.14.patch, HIVE-14990.15.patch, HIVE-14990.16.patch, > HIVE-14990.17.patch, HIVE-14990.patch > > > I am running the tests with isMmTable returning true for most tables (except > ACID, temporary tables, views, etc.). > Many tests will fail because of various expected issues with such an > approach; however we can find issues in MM tables from other failures. > Expected failures > 1) All HCat tests (cannot write MM tables via the HCat writer) > 2) Almost all merge tests (alter .. concat is not supported). > 3) Tests that run dfs commands with specific paths (path changes). > 4) Truncate column (not supported). > 5) Describe formatted will have the new table fields in the output (before > merging MM with ACID). > 6) Many tests w/explain extended - diff in partition "base file name" (path > changes). > 7) TestTxnCommands - all the conversion tests, as they check for bucket count > using file lists (path changes). > 8) HBase metastore tests cause methods are not implemented. > 9) Some load and ExIm tests that export a table and then rely on specific > path for load (path changes). > 10) Bucket map join/etc. - diffs; disabled the optimization for MM tables due > to how it accounts for buckets > 11) rand - different results due to different sequence of processing. > 12) many (not all i.e. not the ones with just one insert) tests that have > stats output, such as file count, for obvious reasons > 13) materialized views, not handled by design - the test check erroneously > makes them "mm", no easy way to tell them apart, I don't want to plumb more > stuff thru just for this test > I'm filing jiras for some test failures that are not obvious and need an > investigation later -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-14990) run all tests for MM tables and fix the issues that are found
[ https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14990: - Attachment: HIVE-14990.17.patch Uploading patch 17 for testing > run all tests for MM tables and fix the issues that are found > - > > Key: HIVE-14990 > URL: https://issues.apache.org/jira/browse/HIVE-14990 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch, > HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch, > HIVE-14990.05.patch, HIVE-14990.05.patch, HIVE-14990.06.patch, > HIVE-14990.06.patch, HIVE-14990.07.patch, HIVE-14990.08.patch, > HIVE-14990.09.patch, HIVE-14990.10.patch, HIVE-14990.10.patch, > HIVE-14990.10.patch, HIVE-14990.12.patch, HIVE-14990.13.patch, > HIVE-14990.14.patch, HIVE-14990.15.patch, HIVE-14990.16.patch, > HIVE-14990.17.patch, HIVE-14990.patch > > > I am running the tests with isMmTable returning true for most tables (except > ACID, temporary tables, views, etc.). > Many tests will fail because of various expected issues with such an > approach; however we can find issues in MM tables from other failures. > Expected failures > 1) All HCat tests (cannot write MM tables via the HCat writer) > 2) Almost all merge tests (alter .. concat is not supported). > 3) Tests that run dfs commands with specific paths (path changes). > 4) Truncate column (not supported). > 5) Describe formatted will have the new table fields in the output (before > merging MM with ACID). > 6) Many tests w/explain extended - diff in partition "base file name" (path > changes). > 7) TestTxnCommands - all the conversion tests, as they check for bucket count > using file lists (path changes). > 8) HBase metastore tests cause methods are not implemented. > 9) Some load and ExIm tests that export a table and then rely on specific > path for load (path changes). > 10) Bucket map join/etc. - diffs; disabled the optimization for MM tables due > to how it accounts for buckets > 11) rand - different results due to different sequence of processing. > 12) many (not all i.e. not the ones with just one insert) tests that have > stats output, such as file count, for obvious reasons > 13) materialized views, not handled by design - the test check erroneously > makes them "mm", no easy way to tell them apart, I don't want to plumb more > stuff thru just for this test > I'm filing jiras for some test failures that are not obvious and need an > investigation later -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (HIVE-14990) run all tests for MM tables and fix the issues that are found
[ https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reopened HIVE-14990: -- Assignee: Wei Zheng (was: Sergey Shelukhin) As ACID integration is done, I want to re-run this to see what the feature parity is like. > run all tests for MM tables and fix the issues that are found > - > > Key: HIVE-14990 > URL: https://issues.apache.org/jira/browse/HIVE-14990 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Fix For: hive-14535 > > Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch, > HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch, > HIVE-14990.05.patch, HIVE-14990.05.patch, HIVE-14990.06.patch, > HIVE-14990.06.patch, HIVE-14990.07.patch, HIVE-14990.08.patch, > HIVE-14990.09.patch, HIVE-14990.10.patch, HIVE-14990.10.patch, > HIVE-14990.10.patch, HIVE-14990.12.patch, HIVE-14990.13.patch, > HIVE-14990.14.patch, HIVE-14990.15.patch, HIVE-14990.16.patch, > HIVE-14990.17.patch, HIVE-14990.patch > > > I am running the tests with isMmTable returning true for most tables (except > ACID, temporary tables, views, etc.). > Many tests will fail because of various expected issues with such an > approach; however we can find issues in MM tables from other failures. > Expected failures > 1) All HCat tests (cannot write MM tables via the HCat writer) > 2) Almost all merge tests (alter .. concat is not supported). > 3) Tests that run dfs commands with specific paths (path changes). > 4) Truncate column (not supported). > 5) Describe formatted will have the new table fields in the output (before > merging MM with ACID). > 6) Many tests w/explain extended - diff in partition "base file name" (path > changes). > 7) TestTxnCommands - all the conversion tests, as they check for bucket count > using file lists (path changes). > 8) HBase metastore tests cause methods are not implemented. > 9) Some load and ExIm tests that export a table and then rely on specific > path for load (path changes). > 10) Bucket map join/etc. - diffs; disabled the optimization for MM tables due > to how it accounts for buckets > 11) rand - different results due to different sequence of processing. > 12) many (not all i.e. not the ones with just one insert) tests that have > stats output, such as file count, for obvious reasons > 13) materialized views, not handled by design - the test check erroneously > makes them "mm", no easy way to tell them apart, I don't want to plumb more > stuff thru just for this test > I'm filing jiras for some test failures that are not obvious and need an > investigation later -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16554) ACID: Make HouseKeeperService threads daemon
[ https://issues.apache.org/jira/browse/HIVE-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-16554: Attachment: HIVE-16554.1.patch > ACID: Make HouseKeeperService threads daemon > > > Key: HIVE-16554 > URL: https://issues.apache.org/jira/browse/HIVE-16554 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.1, 2.1.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-16554.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
