[jira] [Commented] (HIVE-20508) Hive does not support user names of type "user@realm"
[
https://issues.apache.org/jira/browse/HIVE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607918#comment-16607918
]
Hive QA commented on HIVE-20508:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
24s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
21s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 21s{color}
| {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} service: The patch generated 1 new + 27 unchanged - 0
fixed = 28 total (was 27) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} service in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13658/dev-support/hive-personality.sh
|
| git revision | master / b1a917c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13658/yetus/patch-mvninstall-service.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13658/yetus/patch-compile-service.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13658/yetus/patch-compile-service.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13658/yetus/diff-checkstyle-service.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13658/yetus/patch-findbugs-service.txt
|
| modules | C: common service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13658/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive does not support user names of type "user@realm"
> -
>
> Key: HIVE-20508
> URL: https://issues.apache.org/jira/browse/HIVE-20508
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20508.1.patch, HIVE-20508.2.patch,
> HIVE-20508.3.patch, HIVE-20508.4.patch
>
>
> Hive does not support user names of type "user@realm". This causes
> authorization problems with Ranger for user names containing realms in
> Kerberos environment.
[jira] [Commented] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles
[
https://issues.apache.org/jira/browse/HIVE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607914#comment-16607914
]
Hive QA commented on HIVE-20395:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938855/HIVE-20395.11-branch-2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 10662 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=227)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_tableproperty_optimize]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[table_nonprintable]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=155)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[merge_negative_5]
(batchId=88)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=100)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf]
(batchId=125)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=176)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13657/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13657/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13657/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938855 - PreCommit-HIVE-Build
> Parallelize files move in the ql.metadata.Hive#replaceFiles
> ---
>
> Key: HIVE-20395
> URL: https://issues.apache.org/jira/browse/HIVE-20395
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Igor Dvorzhak
>Assignee: Igor Dvorzhak
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20395.10.patch, HIVE-20395.11-branch-2.patch
>
>
> Files move is not parallelized in `replaceFiles` method and could be very
> slow on Cloud Object Stores.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20521) HS2 doAs=true has permission issue with hadoop.tmp.dir, with MR and S3A filesystem
[
https://issues.apache.org/jira/browse/HIVE-20521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-20521:
-
Summary: HS2 doAs=true has permission issue with hadoop.tmp.dir, with MR
and S3A filesystem (was: HS2 doAs=true has permission issue with
hadoop.tmp.dir and MR execution engine)
> HS2 doAs=true has permission issue with hadoop.tmp.dir, with MR and S3A
> filesystem
> --
>
> Key: HIVE-20521
> URL: https://issues.apache.org/jira/browse/HIVE-20521
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Thejas M Nair
>Priority: Major
>
> This is a result of changes in HIVE-18858.
> As described by [~puneetj] in HIVE-18858 -
> {quote}
> This seems to have broken working scenarios with Hive MR. We now see
> hadoop.tmp.dir is always set to /tmp/hadoop-hive (in job.xml). This creates
> problems on a multi-tenant hadoop cluster since ownership of tmp folder is
> set to the user who executes the jobs first and other users fails to write to
> tmp folder.
> E.g. User1 run job and /tmp/hadoop-hive is created on worker node with
> ownership to user1 and sibsequently user2 tries to run a job and job fails
> due to no write permission on /tmp/hadoop-hive/
> Old behavior allowed multiple tenants to write to their respective tmp
> folders which was secure and contention free. User1 - /tmp/hadoop-user1,
> User2 - /tmp/hadoop-user2.
> {quote}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20515) Empty query results when using results cache and query temp dir, results cache dir in different filesystems
[
https://issues.apache.org/jira/browse/HIVE-20515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607898#comment-16607898
]
Hive QA commented on HIVE-20515:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938852/HIVE-20515.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14931 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13656/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13656/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13656/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938852 - PreCommit-HIVE-Build
> Empty query results when using results cache and query temp dir, results
> cache dir in different filesystems
> ---
>
> Key: HIVE-20515
> URL: https://issues.apache.org/jira/browse/HIVE-20515
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20515.1.patch, HIVE-20515.2.patch
>
>
> If the scratchdir for temporary query results and the results cache dir are
> in different filesystems, moving the query from the temp directory to results
> cache will fail.
> Looking at the moveResultsToCacheDirectory() logic in QueryResultsCache.java,
> I see the following issues:
> - FileSystem.rename() is used, which only works if the files are on the same
> filesystem. Need to use something like Hive.mvFile or something similar which
> can work between different filesystems.
> - The return code from rename() was not checked which might possibly have
> caught the error here. This may not be applicable if a different method from
> FS.rename() is used in the proper fix.
> With some filesystems (noticed this with WASB), if FileSystem.rename()
> returns false on failure rather than throwing an exception, then this results
> in empty results showing up for the query because the return code was not
> checked properly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20515) Empty query results when using results cache and query temp dir, results cache dir in different filesystems
[
https://issues.apache.org/jira/browse/HIVE-20515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607888#comment-16607888
]
Hive QA commented on HIVE-20515:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
9s{color} | {color:red} ql generated 1 new + 2310 unchanged - 0 fixed = 2311
total (was 2310) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to fs in
org.apache.hadoop.hive.ql.cache.results.QueryResultsCache.moveResultsToCacheDirectory(Path)
At
QueryResultsCache.java:org.apache.hadoop.hive.ql.cache.results.QueryResultsCache.moveResultsToCacheDirectory(Path)
At QueryResultsCache.java:[line 793] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13656/dev-support/hive-personality.sh
|
| git revision | master / b1a917c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13656/yetus/new-findbugs-ql.html
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13656/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Empty query results when using results cache and query temp dir, results
> cache dir in different filesystems
> ---
>
> Key: HIVE-20515
> URL: https://issues.apache.org/jira/browse/HIVE-20515
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20515.1.patch, HIVE-20515.2.patch
>
>
> If the scratchdir for temporary query results and the results cache dir are
> in different filesystems, moving the query from the temp directory to results
> cache will fail.
> Looking at the moveResultsToCacheDirectory() logic in QueryResultsCache.java,
> I see the following issues:
> - FileSystem.rename() is used, which only works if the files are on the same
> filesystem. Need to use
[jira] [Commented] (HIVE-20020) Hive contrib jar should not be in lib
[
https://issues.apache.org/jira/browse/HIVE-20020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607883#comment-16607883
]
Hive QA commented on HIVE-20020:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938848/HIVE-20020.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14930 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsHttp.testOpenConnectionMetrics
(batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13655/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13655/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13655/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938848 - PreCommit-HIVE-Build
> Hive contrib jar should not be in lib
> -
>
> Key: HIVE-20020
> URL: https://issues.apache.org/jira/browse/HIVE-20020
> Project: Hive
> Issue Type: Improvement
> Components: Contrib
>Reporter: Johndee Burks
>Assignee: Alice Fan
>Priority: Trivial
> Fix For: 4.0.0
>
> Attachments: HIVE-20020.3.patch, HIVE-20020.4.patch
>
>
> Currently the way hive is packaged it includes hive-contrib-.jar in
> lib, we should not include it here because it is picked up by services like
> HS2. This creates a situation in which experimental features such as the
> [MultiDelimitSerDe|https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java]
> are accessible without understanding how to really install and use it. For
> example you can create a table using HS2 via beeline with the aforementioned
> SerDe and it will work as long you do not do M/R jobs. The M/R jobs do not
> work because the SerDe is not in aux to get shipped into distcache. I propose
> we do not package it this way and if someone would like to leverage an
> experimental feature they can add it manually to their environment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20020) Hive contrib jar should not be in lib
[
https://issues.apache.org/jira/browse/HIVE-20020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607872#comment-16607872
]
Hive QA commented on HIVE-20020:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13655/dev-support/hive-personality.sh
|
| git revision | master / b1a917c |
| Default Java | 1.8.0_111 |
| modules | C: packaging U: packaging |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13655/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive contrib jar should not be in lib
> -
>
> Key: HIVE-20020
> URL: https://issues.apache.org/jira/browse/HIVE-20020
> Project: Hive
> Issue Type: Improvement
> Components: Contrib
>Reporter: Johndee Burks
>Assignee: Alice Fan
>Priority: Trivial
> Fix For: 4.0.0
>
> Attachments: HIVE-20020.3.patch, HIVE-20020.4.patch
>
>
> Currently the way hive is packaged it includes hive-contrib-.jar in
> lib, we should not include it here because it is picked up by services like
> HS2. This creates a situation in which experimental features such as the
> [MultiDelimitSerDe|https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java]
> are accessible without understanding how to really install and use it. For
> example you can create a table using HS2 via beeline with the aforementioned
> SerDe and it will work as long you do not do M/R jobs. The M/R jobs do not
> work because the SerDe is not in aux to get shipped into distcache. I propose
> we do not package it this way and if someone would like to leverage an
> experimental feature they can add it manually to their environment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20506) HOS times out when cluster is full while Hive-on-MR waits
[
https://issues.apache.org/jira/browse/HIVE-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607868#comment-16607868
]
Hive QA commented on HIVE-20506:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938845/HIVE-20506.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14931 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testQuotedIdentifier
(batchId=299)
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSparkSessionTimeout
(batchId=245)
org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsHttp.testOpenConnectionMetrics
(batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13654/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13654/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13654/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938845 - PreCommit-HIVE-Build
> HOS times out when cluster is full while Hive-on-MR waits
> -
>
> Key: HIVE-20506
> URL: https://issues.apache.org/jira/browse/HIVE-20506
> Project: Hive
> Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Major
> Attachments: HIVE-20506-CDH5.14.2.patch, HIVE-20506.1.patch, Screen
> Shot 2018-09-07 at 8.10.37 AM.png
>
>
> My understanding is as follows:
> Hive-on-MR when the cluster is full will wait for resources to be available
> before submitting a job. This is because the hadoop jar command is the
> primary mechanism Hive uses to know if a job is complete or failed.
>
> Hive-on-Spark will timeout after {{SPARK_RPC_CLIENT_CONNECT_TIMEOUT}} because
> the RPC client in the AppMaster doesn't connect back to the RPC Server in
> HS2.
> This is a behavior difference it'd be great to close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20522) HiveFilterSetOpTransposeRule may throw assertion error due to nullability of fields
[
https://issues.apache.org/jira/browse/HIVE-20522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20522:
---
Attachment: HIVE-20522.patch
> HiveFilterSetOpTransposeRule may throw assertion error due to nullability of
> fields
> ---
>
> Key: HIVE-20522
> URL: https://issues.apache.org/jira/browse/HIVE-20522
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Riju Trivedi
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20522.patch
>
>
> {noformat}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Cannot add expression
> of different type to set:
> set type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" column1, VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" column2, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column3)
> NOT NULL
> expression type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> COLLATE "ISO-8859-1$en_US$primary" NOT NULL column1, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column2,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" NOT NULL column3) NOT NULL
> set is rel#260:HiveFilter.HIVE.[](input=HepRelVertex#251,condition=<($2,
> _UTF-16LE'100'))
> expression is HiveFilter#262
> {noformat}
> q file contains examples that may to reproduce failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20522) HiveFilterSetOpTransposeRule may throw assertion error due to nullability of fields
[
https://issues.apache.org/jira/browse/HIVE-20522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20522:
---
Attachment: (was: HIVE-20522.patch)
> HiveFilterSetOpTransposeRule may throw assertion error due to nullability of
> fields
> ---
>
> Key: HIVE-20522
> URL: https://issues.apache.org/jira/browse/HIVE-20522
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Riju Trivedi
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20522.patch
>
>
> {noformat}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Cannot add expression
> of different type to set:
> set type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" column1, VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" column2, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column3)
> NOT NULL
> expression type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> COLLATE "ISO-8859-1$en_US$primary" NOT NULL column1, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column2,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" NOT NULL column3) NOT NULL
> set is rel#260:HiveFilter.HIVE.[](input=HepRelVertex#251,condition=<($2,
> _UTF-16LE'100'))
> expression is HiveFilter#262
> {noformat}
> q file contains examples that may to reproduce failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20522) HiveFilterSetOpTransposeRule may throw assertion error due to nullability of fields
[
https://issues.apache.org/jira/browse/HIVE-20522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20522:
---
Reporter: Riju Trivedi (was: Jesus Camacho Rodriguez)
> HiveFilterSetOpTransposeRule may throw assertion error due to nullability of
> fields
> ---
>
> Key: HIVE-20522
> URL: https://issues.apache.org/jira/browse/HIVE-20522
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Riju Trivedi
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20522.patch
>
>
> {noformat}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Cannot add expression
> of different type to set:
> set type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" column1, VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" column2, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column3)
> NOT NULL
> expression type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> COLLATE "ISO-8859-1$en_US$primary" NOT NULL column1, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column2,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" NOT NULL column3) NOT NULL
> set is rel#260:HiveFilter.HIVE.[](input=HepRelVertex#251,condition=<($2,
> _UTF-16LE'100'))
> expression is HiveFilter#262
> {noformat}
> q file contains examples that may to reproduce failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20506) HOS times out when cluster is full while Hive-on-MR waits
[
https://issues.apache.org/jira/browse/HIVE-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607858#comment-16607858
]
Hive QA commented on HIVE-20506:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
14s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} spark-client in master has 10 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
55s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} spark-client: The patch generated 7 new + 45 unchanged
- 0 fixed = 52 total (was 45) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
35s{color} | {color:red} spark-client generated 2 new + 10 unchanged - 0 fixed
= 12 total (was 10) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:spark-client |
| | Boxing/unboxing to parse a primitive
org.apache.hive.spark.client.rpc.RpcServer.getApplicationIDFromString(String)
At
RpcServer.java:org.apache.hive.spark.client.rpc.RpcServer.getApplicationIDFromString(String)
At RpcServer.java:[line 193] |
| | Boxing/unboxing to parse a primitive
org.apache.hive.spark.client.rpc.RpcServer.getApplicationIDFromString(String)
At
RpcServer.java:org.apache.hive.spark.client.rpc.RpcServer.getApplicationIDFromString(String)
At RpcServer.java:[line 192] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13654/dev-support/hive-personality.sh
|
| git revision | master / b1a917c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13654/yetus/patch-mvninstall-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13654/yetus/diff-checkstyle-spark-client.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13654/yetus/new-findbugs-spark-client.html
|
| modules | C: ql spark-client U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13654/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HOS times out when cluster is full while Hive-on-MR waits
> -
>
> Key: HIVE-20506
> URL: https://issues.apache.org/jira/browse/HIVE-20506
>
[jira] [Updated] (HIVE-20522) HiveFilterSetOpTransposeRule may throw assertion error due to nullability of fields
[
https://issues.apache.org/jira/browse/HIVE-20522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20522:
---
Attachment: HIVE-20522.patch
> HiveFilterSetOpTransposeRule may throw assertion error due to nullability of
> fields
> ---
>
> Key: HIVE-20522
> URL: https://issues.apache.org/jira/browse/HIVE-20522
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20522.patch
>
>
> {noformat}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Cannot add expression
> of different type to set:
> set type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" column1, VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" column2, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column3)
> NOT NULL
> expression type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> COLLATE "ISO-8859-1$en_US$primary" NOT NULL column1, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column2,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" NOT NULL column3) NOT NULL
> set is rel#260:HiveFilter.HIVE.[](input=HepRelVertex#251,condition=<($2,
> _UTF-16LE'100'))
> expression is HiveFilter#262
> {noformat}
> q file contains examples that may to reproduce failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20522) HiveFilterSetOpTransposeRule may throw assertion error due to nullability of fields
[
https://issues.apache.org/jira/browse/HIVE-20522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20522:
---
Attachment: (was: HIVE-20522.patch)
> HiveFilterSetOpTransposeRule may throw assertion error due to nullability of
> fields
> ---
>
> Key: HIVE-20522
> URL: https://issues.apache.org/jira/browse/HIVE-20522
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20522.patch
>
>
> {noformat}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Cannot add expression
> of different type to set:
> set type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" column1, VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" column2, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column3)
> NOT NULL
> expression type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> COLLATE "ISO-8859-1$en_US$primary" NOT NULL column1, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column2,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" NOT NULL column3) NOT NULL
> set is rel#260:HiveFilter.HIVE.[](input=HepRelVertex#251,condition=<($2,
> _UTF-16LE'100'))
> expression is HiveFilter#262
> {noformat}
> q file contains examples that may to reproduce failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20522) HiveFilterSetOpTransposeRule may throw assertion error due to nullability of fields
[
https://issues.apache.org/jira/browse/HIVE-20522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20522:
---
Attachment: HIVE-20522.patch
> HiveFilterSetOpTransposeRule may throw assertion error due to nullability of
> fields
> ---
>
> Key: HIVE-20522
> URL: https://issues.apache.org/jira/browse/HIVE-20522
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20522.patch
>
>
> {noformat}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Cannot add expression
> of different type to set:
> set type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" column1, VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" column2, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column3)
> NOT NULL
> expression type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> COLLATE "ISO-8859-1$en_US$primary" NOT NULL column1, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column2,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" NOT NULL column3) NOT NULL
> set is rel#260:HiveFilter.HIVE.[](input=HepRelVertex#251,condition=<($2,
> _UTF-16LE'100'))
> expression is HiveFilter#262
> {noformat}
> q file contains examples that may to reproduce failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-20522) HiveFilterSetOpTransposeRule may throw assertion error due to nullability of fields
[
https://issues.apache.org/jira/browse/HIVE-20522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-20522 started by Jesus Camacho Rodriguez.
--
> HiveFilterSetOpTransposeRule may throw assertion error due to nullability of
> fields
> ---
>
> Key: HIVE-20522
> URL: https://issues.apache.org/jira/browse/HIVE-20522
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> {noformat}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Cannot add expression
> of different type to set:
> set type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" column1, VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" column2, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column3)
> NOT NULL
> expression type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> COLLATE "ISO-8859-1$en_US$primary" NOT NULL column1, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column2,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" NOT NULL column3) NOT NULL
> set is rel#260:HiveFilter.HIVE.[](input=HepRelVertex#251,condition=<($2,
> _UTF-16LE'100'))
> expression is HiveFilter#262
> {noformat}
> q file contains examples that may to reproduce failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20522) HiveFilterSetOpTransposeRule may throw assertion error due to nullability of fields
[
https://issues.apache.org/jira/browse/HIVE-20522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20522:
---
Status: Patch Available (was: In Progress)
> HiveFilterSetOpTransposeRule may throw assertion error due to nullability of
> fields
> ---
>
> Key: HIVE-20522
> URL: https://issues.apache.org/jira/browse/HIVE-20522
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> {noformat}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Cannot add expression
> of different type to set:
> set type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" column1, VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" column2, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column3)
> NOT NULL
> expression type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> COLLATE "ISO-8859-1$en_US$primary" NOT NULL column1, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column2,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" NOT NULL column3) NOT NULL
> set is rel#260:HiveFilter.HIVE.[](input=HepRelVertex#251,condition=<($2,
> _UTF-16LE'100'))
> expression is HiveFilter#262
> {noformat}
> q file contains examples that may to reproduce failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20522) HiveFilterSetOpTransposeRule may throw assertion error due to nullability of fields
[
https://issues.apache.org/jira/browse/HIVE-20522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-20522:
--
> HiveFilterSetOpTransposeRule may throw assertion error due to nullability of
> fields
> ---
>
> Key: HIVE-20522
> URL: https://issues.apache.org/jira/browse/HIVE-20522
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> {noformat}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Cannot add expression
> of different type to set:
> set type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" column1, VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" column2, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column3)
> NOT NULL
> expression type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> COLLATE "ISO-8859-1$en_US$primary" NOT NULL column1, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column2,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" NOT NULL column3) NOT NULL
> set is rel#260:HiveFilter.HIVE.[](input=HepRelVertex#251,condition=<($2,
> _UTF-16LE'100'))
> expression is HiveFilter#262
> {noformat}
> q file contains examples that may to reproduce failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17300) WebUI query plan graphs
[
https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607833#comment-16607833
]
Hive QA commented on HIVE-17300:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938843/HIVE-17300.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14932 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsBinary.testOpenConnectionMetrics
(batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13653/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13653/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13653/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938843 - PreCommit-HIVE-Build
> WebUI query plan graphs
> ---
>
> Key: HIVE-17300
> URL: https://issues.apache.org/jira/browse/HIVE-17300
> Project: Hive
> Issue Type: Sub-task
> Components: Web UI
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: beginner, features, patch
> Attachments: HIVE-17300.3.patch, HIVE-17300.4.patch,
> HIVE-17300.5.patch, HIVE-17300.6.patch, HIVE-17300.7.patch, HIVE-17300.patch,
> complete_success.png, full_mapred_stats.png, graph_with_mapred_stats.png,
> last_stage_error.png, last_stage_running.png, non_mapred_task_selected.png
>
>
> Hi all,
> I’m working on a feature of the Hive WebUI Query Plan tab that would provide
> the option to display the query plan as a nice graph (scroll down for
> screenshots). If you click on one of the graph’s stages, the plan for that
> stage appears as text below.
> Stages are color-coded if they have a status (Success, Error, Running), and
> the rest are grayed out. Coloring is based on status already available in the
> WebUI, under the Stages tab.
> There is an additional option to display stats for MapReduce tasks. This
> includes the job’s ID, tracking URL (where the logs are found), and mapper
> and reducer numbers/progress, among other info.
> The library I’m using for the graph is called vis.js (http://visjs.org/). It
> has an Apache license, and the only necessary file to be included from this
> library is about 700 KB.
> I tried to keep server-side changes minimal, and graph generation is taken
> care of by the client. Plans with more than a given number of stages
> (default: 25) won't be displayed in order to preserve resources.
> I’d love to hear any and all input from the community about this feature: do
> you think it’s useful, and is there anything important I’m missing?
> Thanks,
> Karen Coppage
> Review request: https://reviews.apache.org/r/61663/
> Any input is welcome!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17300) WebUI query plan graphs
[
https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607813#comment-16607813
]
Hive QA commented on HIVE-17300:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
11s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} common: The patch generated 1 new + 449 unchanged - 0
fixed = 450 total (was 449) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 5 new + 186 unchanged - 0
fixed = 191 total (was 186) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13653/dev-support/hive-personality.sh
|
| git revision | master / b1a917c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13653/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13653/yetus/diff-checkstyle-ql.txt
|
| modules | C: common itests/hive-unit ql service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13653/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> WebUI query plan graphs
> ---
>
> Key: HIVE-17300
> URL: https://issues.apache.org/jira/browse/HIVE-17300
> Project: Hive
> Issue Type: Sub-task
> Components: Web UI
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: beginner, features, patch
>
[jira] [Updated] (HIVE-20521) HS2 doAs=true has permission issue with hadoop.tmp.dir and MR execution engine
[
https://issues.apache.org/jira/browse/HIVE-20521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-20521:
-
Affects Version/s: 3.0.0
3.1.0
> HS2 doAs=true has permission issue with hadoop.tmp.dir and MR execution engine
> --
>
> Key: HIVE-20521
> URL: https://issues.apache.org/jira/browse/HIVE-20521
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Thejas M Nair
>Priority: Major
>
> This is a result of changes in HIVE-18858.
> As described by [~puneetj] in HIVE-18858 -
> {quote}
> This seems to have broken working scenarios with Hive MR. We now see
> hadoop.tmp.dir is always set to /tmp/hadoop-hive (in job.xml). This creates
> problems on a multi-tenant hadoop cluster since ownership of tmp folder is
> set to the user who executes the jobs first and other users fails to write to
> tmp folder.
> E.g. User1 run job and /tmp/hadoop-hive is created on worker node with
> ownership to user1 and sibsequently user2 tries to run a job and job fails
> due to no write permission on /tmp/hadoop-hive/
> Old behavior allowed multiple tenants to write to their respective tmp
> folders which was secure and contention free. User1 - /tmp/hadoop-user1,
> User2 - /tmp/hadoop-user2.
> {quote}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20521) HS2 doAs=true has permission issue with hadoop.tmp.dir and MR execution engine
[
https://issues.apache.org/jira/browse/HIVE-20521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-20521:
-
Summary: HS2 doAs=true has permission issue with hadoop.tmp.dir and MR
execution engine (was: HS2 doAs=true has permission issue with hadoop.tmp.dir)
> HS2 doAs=true has permission issue with hadoop.tmp.dir and MR execution engine
> --
>
> Key: HIVE-20521
> URL: https://issues.apache.org/jira/browse/HIVE-20521
> Project: Hive
> Issue Type: Improvement
>Reporter: Thejas M Nair
>Priority: Major
>
> This is a result of changes in HIVE-18858.
> As described by [~puneetj] in HIVE-18858 -
> {quote}
> This seems to have broken working scenarios with Hive MR. We now see
> hadoop.tmp.dir is always set to /tmp/hadoop-hive (in job.xml). This creates
> problems on a multi-tenant hadoop cluster since ownership of tmp folder is
> set to the user who executes the jobs first and other users fails to write to
> tmp folder.
> E.g. User1 run job and /tmp/hadoop-hive is created on worker node with
> ownership to user1 and sibsequently user2 tries to run a job and job fails
> due to no write permission on /tmp/hadoop-hive/
> Old behavior allowed multiple tenants to write to their respective tmp
> folders which was secure and contention free. User1 - /tmp/hadoop-user1,
> User2 - /tmp/hadoop-user2.
> {quote}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20520) length(CHAR) doesn't consider trailing space
[
https://issues.apache.org/jira/browse/HIVE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20520:
---
Reporter: Naresh P R (was: Jesus Camacho Rodriguez)
> length(CHAR) doesn't consider trailing space
>
>
> Key: HIVE-20520
> URL: https://issues.apache.org/jira/browse/HIVE-20520
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Naresh P R
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20520.patch
>
>
> Reproduce steps:
> {code:java}
> create table test(a char(2), b varchar(2));
> insert into test values('L ', 'L ');
> select length(a),length(b) from test;
> +--+--+
> | _c0 | _c1 |
> +--+--+
> | 1| 2|
> +--+--+
> 1 row selected (0.185 seconds)
> {code}
> Here char with trailing spaces are trimmed, whereas leading spaces are not
> trimmed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20503) Use datastructure aware estimations during mapjoin selection
[ https://issues.apache.org/jira/browse/HIVE-20503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607786#comment-16607786 ] Ashutosh Chauhan commented on HIVE-20503: - private boolean useFastHashTable; Do we still need it? I think we said compiler's logic will be agnostic to fast vs optimized and will do computation assuming fast. > Use datastructure aware estimations during mapjoin selection > > > Key: HIVE-20503 > URL: https://issues.apache.org/jira/browse/HIVE-20503 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20503.01.patch, HIVE-20503.01wip01.patch, > HIVE-20503.01wip01.patch, HIVE-20503.02.patch, HIVE-20503.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20360) QTest: ignore driver/qtest exclusions if -Dqfile param is set
[
https://issues.apache.org/jira/browse/HIVE-20360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607785#comment-16607785
]
Hive QA commented on HIVE-20360:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938811/HIVE-20360.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 647 failed/errored test(s), 15439 tests
executed
*Failed tests:*
{noformat}
TestEncryptedHDFSCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=186)
[encryption_join_with_different_encryption_keys.q,encryption_join_unencrypted_tbl.q]
TestHBaseCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=102)
[hbase_ppd_key_range.q,hbasestats.q,hbase_custom_key2.q,hbase_viewjoins.q,hbase_pushdown.q]
TestHBaseCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=104)
[hbase_handler_snapshot.q,hbase_binary_binary.q,hbase_binary_external_table_queries.q,hbase_timestamp.q,hbase_binary_storage_queries.q]
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=156)
[orc_merge2.q,multi_count_distinct_null.q]
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=161)
[insert_values_non_partitioned.q,retry_failure_oom.q,colstats_date_min_max.q,vectorized_math_funcs.q,subquery_multi.q,check_constraint.q,cbo_simple_select.q,cbo_rp_udf_udaf_stats_opt.q,tez_union_decimal.q,vector_parquet_nested_two_level_complex.q,multi_column_in.q,groupby1.q,order_null.q,cbo_rp_udf_udaf.q,cbo_rp_views.q,vector_decimal_aggregate.q,lvj_mapjoin.q,update_all_types.q,retry_failure_stat_changes.q,kill_query.q,materialized_view_rewrite_part_1.q,vector_llap_text_1.q,vector_decimal_3.q,update_two_cols.q,ptf.q,update_where_no_match.q,vectorized_nested_mapjoin.q,authorization_2.q,schema_evol_text_nonvec_part.q,enforce_constraint_notnull.q]
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=168)
[load_dyn_part5.q,vector_groupby_sort_8.q,enforce_order.q,vector_windowing_windowspec4.q,vector_groupby_grouping_sets5.q,vector_complex_join.q,vectorization_7.q,orc_ppd_schema_evol_1a.q,cbo_gby.q,vectorized_dynamic_semijoin_reduction2.q,subquery_exists.q,schema_evol_orc_vec_table.q,lineage3.q,mergejoin_3way.q,materialized_view_rewrite_empty.q,vectorization_16.q,orc_ppd_schema_evol_2a.q,partition_ctas.q,vector_windowing_multipartitioning.q,orc_ppd_date.q,create_merge_compressed.q,vector_outer_join1.q,dynpart_sort_optimization_acid.q,vectorization_not.q,windowing_gby.q,having.q,vector_orderby_5.q,vector_outer_join6.q,vector_topnkey.q,union9.q]
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=174)
[auto_sortmerge_join_13.q,schema_evol_orc_acid_table_update.q,stats_date.q,vector_left_outer_join2.q,metadataonly1.q,materialized_view_rewrite_10.q,constprog_semijoin.q,vector_retry_failure.q,insert1_overwrite_partitions.q,schema_evol_orc_vec_part_llap_io.q,vectorized_ptf.q,alter_merge_orc.q,vector_char_cast.q,join_filters.q,udaf_collect_set_2.q,orc_merge_incompat1.q,vectorized_parquet_types.q,vector_windowing_gby2.q,results_cache_1.q,vectorization_15.q,union7.q,materialized_view_rewrite_9.q,vectorization_nested_udf.q,schema_evol_orc_acidvec_part.q,vector_groupby_3.q,materialized_view_create_rewrite_multi_db.q,insert_values_orig_table_use_metadata.q,vector_mr_diff_schema_alias.q,cbo_rp_gby.q,auto_sortmerge_join_9.q]
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=179)
[vector_case_when_1.q,escape2.q,vector_udf_character_length.q,tez_dynpart_hashjoin_1.q,correlationoptimizer3.q,runtime_stats_merge.q,vector_groupby_grouping_sets1.q,autoColumnStats_2.q,vector_binary_join_groupby.q,runtime_stats_hs2.q,schema_evol_orc_acid_part_llap_io.q,semijoin6.q,materialized_view_rebuild.q,vectorization_0.q,orc_merge8.q,orc_merge_incompat2.q,nonmr_fetch_threshold.q,vectorized_casts.q,vector_grouping_sets.q,schema_evol_orc_acidvec_part_update_llap_io.q,groupby_groupingset_bug.q,schema_evol_text_vecrow_part_all_complex.q,bucketmapjoin1.q,truncate_external_force.q,stats11.q,tez_join_tests.q,schema_evol_text_vecrow_part.q,load_dyn_part1.q,results_cache_2.q,udf_coalesce.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_globallimit]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_no_buckets]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_missing_cols]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_original]
(batchId=84)
[jira] [Updated] (HIVE-20520) length(CHAR) doesn't consider trailing space
[
https://issues.apache.org/jira/browse/HIVE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20520:
---
Attachment: HIVE-20520.patch
> length(CHAR) doesn't consider trailing space
>
>
> Key: HIVE-20520
> URL: https://issues.apache.org/jira/browse/HIVE-20520
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20520.patch
>
>
> Reproduce steps:
> {code:java}
> create table test(a char(2), b varchar(2));
> insert into test values('L ', 'L ');
> select length(a),length(b) from test;
> +--+--+
> | _c0 | _c1 |
> +--+--+
> | 1| 2|
> +--+--+
> 1 row selected (0.185 seconds)
> {code}
> Here char with trailing spaces are trimmed, whereas leading spaces are not
> trimmed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-20520) length(CHAR) doesn't consider trailing space
[
https://issues.apache.org/jira/browse/HIVE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-20520 started by Jesus Camacho Rodriguez.
--
> length(CHAR) doesn't consider trailing space
>
>
> Key: HIVE-20520
> URL: https://issues.apache.org/jira/browse/HIVE-20520
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Reproduce steps:
> {code:java}
> create table test(a char(2), b varchar(2));
> insert into test values('L ', 'L ');
> select length(a),length(b) from test;
> +--+--+
> | _c0 | _c1 |
> +--+--+
> | 1| 2|
> +--+--+
> 1 row selected (0.185 seconds)
> {code}
> Here char with trailing spaces are trimmed, whereas leading spaces are not
> trimmed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20520) length(CHAR) doesn't consider trailing space
[
https://issues.apache.org/jira/browse/HIVE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-20520:
--
> length(CHAR) doesn't consider trailing space
>
>
> Key: HIVE-20520
> URL: https://issues.apache.org/jira/browse/HIVE-20520
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Reproduce steps:
> {code:java}
> create table test(a char(2), b varchar(2));
> insert into test values('L ', 'L ');
> select length(a),length(b) from test;
> +--+--+
> | _c0 | _c1 |
> +--+--+
> | 1| 2|
> +--+--+
> 1 row selected (0.185 seconds)
> {code}
> Here char with trailing spaces are trimmed, whereas leading spaces are not
> trimmed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20520) length(CHAR) doesn't consider trailing space
[
https://issues.apache.org/jira/browse/HIVE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20520:
---
Status: Patch Available (was: In Progress)
> length(CHAR) doesn't consider trailing space
>
>
> Key: HIVE-20520
> URL: https://issues.apache.org/jira/browse/HIVE-20520
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Reproduce steps:
> {code:java}
> create table test(a char(2), b varchar(2));
> insert into test values('L ', 'L ');
> select length(a),length(b) from test;
> +--+--+
> | _c0 | _c1 |
> +--+--+
> | 1| 2|
> +--+--+
> 1 row selected (0.185 seconds)
> {code}
> Here char with trailing spaces are trimmed, whereas leading spaces are not
> trimmed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20360) QTest: ignore driver/qtest exclusions if -Dqfile param is set
[
https://issues.apache.org/jira/browse/HIVE-20360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607746#comment-16607746
]
Hive QA commented on HIVE-20360:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
51s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} itests/util: The patch generated 1 new + 72 unchanged
- 6 fixed = 73 total (was 78) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13652/dev-support/hive-personality.sh
|
| git revision | master / b1a917c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13652/yetus/diff-checkstyle-itests_util.txt
|
| modules | C: itests/util U: itests/util |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13652/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> QTest: ignore driver/qtest exclusions if -Dqfile param is set
> -
>
> Key: HIVE-20360
> URL: https://issues.apache.org/jira/browse/HIVE-20360
> Project: Hive
> Issue Type: Improvement
> Components: Test
>Affects Versions: 3.1.0
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20360.01.patch, HIVE-20360.02.patch
>
>
> Sometimes I need to run qtests with another driver for testing purposes. In
> this case I have to edit testconfiguration.properties which seems a bit
> hacky, even if it's temporary.
> In this case, no tests will run (however there's a log message):
> {code:java}
> mvn test -Pitests -pl itests/qtest -pl itests/util -Dtest=TestCliDriver
> -Dqfile=bucketizedhiveinputformat.q
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20511) REPL DUMP is leaking metastore connections
[
https://issues.apache.org/jira/browse/HIVE-20511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607729#comment-16607729
]
Hive QA commented on HIVE-20511:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938785/HIVE-20511.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14930 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.repl.TestReplDumpTask.removeDBPropertyToPreventRenameWhenBootstrapDumpOfTableFails
(batchId=308)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13651/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13651/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13651/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938785 - PreCommit-HIVE-Build
> REPL DUMP is leaking metastore connections
> --
>
> Key: HIVE-20511
> URL: https://issues.apache.org/jira/browse/HIVE-20511
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20511.01.patch, HIVE-20511.02.patch
>
>
> With remote metastore, REPL DUMP leaking connections. Each repl dump task is
> leaking one connection due to the usage of stale hive object.
> {code}
> 18/09/04 16:01:46 INFO ReplState: REPL::EVENT_DUMP:
> {"dbName":"*","eventId":"566","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"1/0","dumpTime":1536076906}
> 18/09/04 16:01:46 INFO events.AbstractEventHandler: Processing#567 OPEN_TXN
> message :
> {"txnIds":null,"timestamp":1536076905,"fromTxnId":269,"toTxnId":269,"server":"thrift://metastore-service.warehouse-1536062326-s74h.svc.cluster.local:9083","servicePrincipal":""}
> 18/09/04 16:01:46 INFO ReplState: REPL::EVENT_DUMP:
> {"dbName":"*","eventId":"567","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"2/0","dumpTime":1536076906}
> 18/09/04 16:01:46 INFO metastore.HiveMetaStoreClient: Trying to connect to
> metastore with URI
> thrift://metastore-service.warehouse-1536062326-s74h.svc.cluster.local:9083
> 18/09/04 16:01:46 INFO metastore.HiveMetaStoreClient: Opened a connection to
> metastore, current connections: 471
> 18/09/04 16:01:46 INFO metastore.HiveMetaStoreClient: Connected to metastore.
> 18/09/04 16:01:46 INFO metastore.RetryingMetaStoreClient:
> RetryingMetaStoreClient proxy=class
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive
> (auth:SIMPLE) retries=24 delay=5 lifetime=0
> 18/09/04 16:01:46 INFO ReplState: REPL::END:
> {"dbName":"*","dumpType":"INCREMENTAL","actualNumEvents":2,"dumpEndTime":1536076906,"dumpDir":"/user/hive/repl/e45bde27-74dc-45cd-9823-400a8fc1aea3","lastReplId":"567"}
> 18/09/04 16:01:46 INFO repl.ReplDumpTask: Done dumping events, preparing to
> return /user/hive/repl/e45bde27-74dc-45cd-9823-400a8fc1aea3,567
> 18/09/04 16:01:46 INFO ql.Driver: Completed executing
> command(queryId=hive_20180904160145_30f9570a-44e0-4f3b-b961-1906d3972fc4);
> Time taken: 0.585 seconds
> OK
> 18/09/04 16:01:46 INFO ql.Driver: OK
> 18/09/04 16:01:46 INFO lockmgr.DbTxnManager: Stopped heartbeat for query:
> hive_20180904160145_30f9570a-44e0-4f3b-b961-1906d3972fc4
> 18/09/04 16:01:46 INFO metastore.HiveMetaStoreClient: Trying to connect to
> metastore with URI
> thrift://metastore-service.warehouse-1536062326-s74h.svc.cluster.local:9083
> 18/09/04 16:01:46 INFO metastore.HiveMetaStoreClient: Opened a connection to
> metastore, current connections: 472
> 18/09/04 16:01:46 INFO metastore.HiveMetaStoreClient: Connected to metastore.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20511) REPL DUMP is leaking metastore connections
[
https://issues.apache.org/jira/browse/HIVE-20511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607696#comment-16607696
]
Hive QA commented on HIVE-20511:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 11 unchanged - 0 fixed
= 12 total (was 11) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13651/dev-support/hive-personality.sh
|
| git revision | master / b1a917c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13651/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13651/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> REPL DUMP is leaking metastore connections
> --
>
> Key: HIVE-20511
> URL: https://issues.apache.org/jira/browse/HIVE-20511
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20511.01.patch, HIVE-20511.02.patch
>
>
> With remote metastore, REPL DUMP leaking connections. Each repl dump task is
> leaking one connection due to the usage of stale hive object.
> {code}
> 18/09/04 16:01:46 INFO ReplState: REPL::EVENT_DUMP:
> {"dbName":"*","eventId":"566","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"1/0","dumpTime":1536076906}
> 18/09/04 16:01:46 INFO events.AbstractEventHandler: Processing#567 OPEN_TXN
> message :
> {"txnIds":null,"timestamp":1536076905,"fromTxnId":269,"toTxnId":269,"server":"thrift://metastore-service.warehouse-1536062326-s74h.svc.cluster.local:9083","servicePrincipal":""}
> 18/09/04 16:01:46 INFO ReplState: REPL::EVENT_DUMP:
> {"dbName":"*","eventId":"567","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"2/0","dumpTime":1536076906}
> 18/09/04 16:01:46 INFO metastore.HiveMetaStoreClient: Trying to connect to
> metastore with URI
> thrift://metastore-service.warehouse-1536062326-s74h.svc.cluster.local:9083
> 18/09/04 16:01:46 INFO metastore.HiveMetaStoreClient: Opened
[jira] [Commented] (HIVE-18724) Improve error handling for subqueries referencing columns(correlated) of its grand-parent query
[
https://issues.apache.org/jira/browse/HIVE-18724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607667#comment-16607667
]
Hive QA commented on HIVE-18724:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938807/HIVE-18724.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14931 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13650/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13650/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13650/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938807 - PreCommit-HIVE-Build
> Improve error handling for subqueries referencing columns(correlated) of its
> grand-parent query
> ---
>
> Key: HIVE-18724
> URL: https://issues.apache.org/jira/browse/HIVE-18724
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-18724.01.patch, HIVE-18724.02.patch
>
>
> Currently Hive doesn't allow correlated subqueries which refer to columns of
> grand parent query e.g.
> {code:sql}
> select t1.a from t t1 where t1.b IN (select t2.c from t t2 where t2.a IN
> (select max(t3.b) from t t3, t t4 where t3.a=t2.a and t3.b=t1.b))
> {code}
> In this example inner most query has column reference {{t3.b=t1.b}} where
> {{t1.b}} references outer most query.
> The error which Hive compiler throw is
> {noformat}
> FAILED: SemanticException Line 0:-1 Unsupported SubQuery Expression 'a':
> Nested SubQuery expressions are not supported.
> {noformat}
> This is very misleading since Hive does support Nested subqueries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607622#comment-16607622
]
Sahil Takiar commented on HIVE-17684:
-
[~mi...@cloudera.com] yes we can configure it that way. We just have to add a
new config to {{HiveConf.java}} and then we can set a lower value for all of
our tests (we can do this by modifying the {{hive-site.xml}} files under
{{data/conf}}). The default value can be somewhere around 50.
Let me know if you need help adding the new config variable.
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-17684.01.patch, HIVE-17684.02.patch,
> HIVE-17684.03.patch, HIVE-17684.04.patch, HIVE-17684.05.patch,
> HIVE-17684.06.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect
> scenarios where the small table is taking too much space in memory, in which
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic
> to estimate how much memory the {{HashMap}} is consuming:
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() /
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be
> inaccurate. The value returned by this method returns all reachable and
> unreachable memory on the heap, so there may be a bunch of garbage data, and
> the JVM just hasn't taken the time to reclaim it all. This can lead to
> intermittent failures of this check even though a simple GC would have
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS.
> In Hive-on-MR this probably made sense to use because every Hive task was run
> in a dedicated container, so a Hive Task could assume it created most of the
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks
> running in a single executor, each doing different things.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20519) Remove 30m min value for hive.spark.session.timeout
[
https://issues.apache.org/jira/browse/HIVE-20519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar reassigned HIVE-20519:
---
> Remove 30m min value for hive.spark.session.timeout
> ---
>
> Key: HIVE-20519
> URL: https://issues.apache.org/jira/browse/HIVE-20519
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> In HIVE-14162 we added the config \{{hive.spark.session.timeout}} which
> provided a way to time out Spark sessions that are active for a long period
> of time. The config has a lower bound of 30m which we should remove. It
> should be possible for users to configure this value so the HoS session is
> closed as soon as the query is complete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20506) HOS times out when cluster is full while Hive-on-MR waits
[
https://issues.apache.org/jira/browse/HIVE-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607614#comment-16607614
]
Sahil Takiar commented on HIVE-20506:
-
Yeah, I'll file a follow up JIRA to do the re-factoring. Otherwise +1 pending
tests.
> HOS times out when cluster is full while Hive-on-MR waits
> -
>
> Key: HIVE-20506
> URL: https://issues.apache.org/jira/browse/HIVE-20506
> Project: Hive
> Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Major
> Attachments: HIVE-20506-CDH5.14.2.patch, HIVE-20506.1.patch, Screen
> Shot 2018-09-07 at 8.10.37 AM.png
>
>
> My understanding is as follows:
> Hive-on-MR when the cluster is full will wait for resources to be available
> before submitting a job. This is because the hadoop jar command is the
> primary mechanism Hive uses to know if a job is complete or failed.
>
> Hive-on-Spark will timeout after {{SPARK_RPC_CLIENT_CONNECT_TIMEOUT}} because
> the RPC client in the AppMaster doesn't connect back to the RPC Server in
> HS2.
> This is a behavior difference it'd be great to close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20489) Explain plan of query hangs
[ https://issues.apache.org/jira/browse/HIVE-20489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607613#comment-16607613 ] Naveen Gangam commented on HIVE-20489: -- Review posted to RB at https://reviews.apache.org/r/68574/. The latest patch looks good. I have approved it on RB. So +1 for me. > Explain plan of query hangs > --- > > Key: HIVE-20489 > URL: https://issues.apache.org/jira/browse/HIVE-20489 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20489.1.patch, HIVE-20489.2.patch, > HIVE-20489.3.patch, HIVE-20489.4.patch > > > Explain on a query that joins 47 views, in effect around 94 joins after view > expansion seems to take forever. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18724) Improve error handling for subqueries referencing columns(correlated) of its grand-parent query
[
https://issues.apache.org/jira/browse/HIVE-18724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607608#comment-16607608
]
Hive QA commented on HIVE-18724:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13650/dev-support/hive-personality.sh
|
| git revision | master / b1a917c |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13650/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve error handling for subqueries referencing columns(correlated) of its
> grand-parent query
> ---
>
> Key: HIVE-18724
> URL: https://issues.apache.org/jira/browse/HIVE-18724
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-18724.01.patch, HIVE-18724.02.patch
>
>
> Currently Hive doesn't allow correlated subqueries which refer to columns of
> grand parent query e.g.
> {code:sql}
> select t1.a from t t1 where t1.b IN (select t2.c from t t2 where t2.a IN
> (select max(t3.b) from t t3, t t4 where t3.a=t2.a and t3.b=t1.b))
> {code}
> In this example inner most query has column reference {{t3.b=t1.b}} where
> {{t1.b}} references outer most query.
> The error which Hive compiler throw is
> {noformat}
> FAILED: SemanticException Line 0:-1 Unsupported SubQuery Expression 'a':
> Nested SubQuery expressions are not supported.
> {noformat}
> This is very misleading since Hive does support Nested subqueries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20503) Use datastructure aware estimations during mapjoin selection
[
https://issues.apache.org/jira/browse/HIVE-20503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607605#comment-16607605
]
Hive QA commented on HIVE-20503:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938806/HIVE-20503.03.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14930 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsBinary.testOpenConnectionMetrics
(batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13649/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13649/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13649/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938806 - PreCommit-HIVE-Build
> Use datastructure aware estimations during mapjoin selection
>
>
> Key: HIVE-20503
> URL: https://issues.apache.org/jira/browse/HIVE-20503
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20503.01.patch, HIVE-20503.01wip01.patch,
> HIVE-20503.01wip01.patch, HIVE-20503.02.patch, HIVE-20503.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20510) Vectorization : Support loading bucketed tables using sorted dynamic partition optimizer
[ https://issues.apache.org/jira/browse/HIVE-20510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-20510: -- Attachment: HIVE-20510.3.patch > Vectorization : Support loading bucketed tables using sorted dynamic > partition optimizer > > > Key: HIVE-20510 > URL: https://issues.apache.org/jira/browse/HIVE-20510 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20510.1.patch, HIVE-20510.2.patch, > HIVE-20510.3.patch > > > sorted dynamic partition optimizer does not work on bucketed tables when > vectorization is enabled. > > cc [~mmccline] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20506) HOS times out when cluster is full while Hive-on-MR waits
[
https://issues.apache.org/jira/browse/HIVE-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607602#comment-16607602
]
Brock Noland commented on HIVE-20506:
-
bq. Is that correct?
Correct!
bq. Rather than extending the timeout, why not just create two separate ones?
.. We probably don't want to change the meaning of the current timeout for
backwards compatibility, so maybe we could deprecate the existing one and
replace it with two new ones?
Hmm this seems complex.100% of my HOS customers that are complaining about this
behavior so I don't see it as a backwards incompatible change but an
improvement on the existing timeout.
bq. Is there any way to avoid creating a YarnClient? I guess this is mitigated
slightly by the fact that you only create the client if the timeout is triggered
I don't think creating a YarnClient once a minute is a huge deal. We could of
course share a YarnClient. Perhaps that could be a follow-on patch?
bq. would this work on a secure cluster
My test above is on a kerberos + TLS cluster. Worked great.
bq. perhaps we can parse the state from the logs
In the patch we only parse the output after spark-submit has exited so I don't
think spark-submit sticks around forever like it used to.
bq. Can we move all the changes in RpcServer to a separate class? That class is
really meant to act as a generic RPC framework that is relatively independent
of the HoS logic
We could move the {{YarnApplicationStateFinder}} out of there, but otherwise I
don't see how we can implement this without some change to {{RPCServer}}. Maybe
you could take that on as follow-on change?
> HOS times out when cluster is full while Hive-on-MR waits
> -
>
> Key: HIVE-20506
> URL: https://issues.apache.org/jira/browse/HIVE-20506
> Project: Hive
> Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Major
> Attachments: HIVE-20506-CDH5.14.2.patch, HIVE-20506.1.patch, Screen
> Shot 2018-09-07 at 8.10.37 AM.png
>
>
> My understanding is as follows:
> Hive-on-MR when the cluster is full will wait for resources to be available
> before submitting a job. This is because the hadoop jar command is the
> primary mechanism Hive uses to know if a job is complete or failed.
>
> Hive-on-Spark will timeout after {{SPARK_RPC_CLIENT_CONNECT_TIMEOUT}} because
> the RPC client in the AppMaster doesn't connect back to the RPC Server in
> HS2.
> This is a behavior difference it'd be great to close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20513) Vectorization: Improve Fast Vector MapJoin Bytes Hash Tables
[ https://issues.apache.org/jira/browse/HIVE-20513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20513: Status: Patch Available (was: In Progress) Some changes from Zoltan's code review. > Vectorization: Improve Fast Vector MapJoin Bytes Hash Tables > > > Key: HIVE-20513 > URL: https://issues.apache.org/jira/browse/HIVE-20513 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20513.01.patch, HIVE-20513.02.patch, > HIVE-20513.03.patch > > > Based on HIVE-20491 / HIVE-20503 discussions, improve Fast Vector MapJoin > Bytes Hash Tables by only storing a one word slot entry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20513) Vectorization: Improve Fast Vector MapJoin Bytes Hash Tables
[ https://issues.apache.org/jira/browse/HIVE-20513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20513: Attachment: HIVE-20513.03.patch > Vectorization: Improve Fast Vector MapJoin Bytes Hash Tables > > > Key: HIVE-20513 > URL: https://issues.apache.org/jira/browse/HIVE-20513 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20513.01.patch, HIVE-20513.02.patch, > HIVE-20513.03.patch > > > Based on HIVE-20491 / HIVE-20503 discussions, improve Fast Vector MapJoin > Bytes Hash Tables by only storing a one word slot entry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20513) Vectorization: Improve Fast Vector MapJoin Bytes Hash Tables
[ https://issues.apache.org/jira/browse/HIVE-20513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20513: Status: In Progress (was: Patch Available) > Vectorization: Improve Fast Vector MapJoin Bytes Hash Tables > > > Key: HIVE-20513 > URL: https://issues.apache.org/jira/browse/HIVE-20513 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20513.01.patch, HIVE-20513.02.patch, > HIVE-20513.03.patch > > > Based on HIVE-20491 / HIVE-20503 discussions, improve Fast Vector MapJoin > Bytes Hash Tables by only storing a one word slot entry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20503) Use datastructure aware estimations during mapjoin selection
[
https://issues.apache.org/jira/browse/HIVE-20503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607550#comment-16607550
]
Hive QA commented on HIVE-20503:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
20s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13649/dev-support/hive-personality.sh
|
| git revision | master / b1a917c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13649/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use datastructure aware estimations during mapjoin selection
>
>
> Key: HIVE-20503
> URL: https://issues.apache.org/jira/browse/HIVE-20503
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20503.01.patch, HIVE-20503.01wip01.patch,
> HIVE-20503.01wip01.patch, HIVE-20503.02.patch, HIVE-20503.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Status: Patch Available (was: In Progress)
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.10.patch,
> HIVE-20225.11.patch, HIVE-20225.12.patch, HIVE-20225.13.patch,
> HIVE-20225.14-branch-2.patch, HIVE-20225.15.patch,
> HIVE-20225.16-branch-2.patch, HIVE-20225.17-branch-3.patch,
> HIVE-20225.18-branch-3.patch, HIVE-20225.19-branch-3.patch,
> HIVE-20225.2.patch, HIVE-20225.3.patch, HIVE-20225.4.patch,
> HIVE-20225.5-branch-2.patch, HIVE-20225.6.patch, HIVE-20225.7.patch,
> HIVE-20225.8.patch, HIVE-20225.9.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Attachment: HIVE-20225.19-branch-3.patch
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.10.patch,
> HIVE-20225.11.patch, HIVE-20225.12.patch, HIVE-20225.13.patch,
> HIVE-20225.14-branch-2.patch, HIVE-20225.15.patch,
> HIVE-20225.16-branch-2.patch, HIVE-20225.17-branch-3.patch,
> HIVE-20225.18-branch-3.patch, HIVE-20225.19-branch-3.patch,
> HIVE-20225.2.patch, HIVE-20225.3.patch, HIVE-20225.4.patch,
> HIVE-20225.5-branch-2.patch, HIVE-20225.6.patch, HIVE-20225.7.patch,
> HIVE-20225.8.patch, HIVE-20225.9.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format
[
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lu Li updated HIVE-20225:
-
Status: In Progress (was: Patch Available)
> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.10.patch,
> HIVE-20225.11.patch, HIVE-20225.12.patch, HIVE-20225.13.patch,
> HIVE-20225.14-branch-2.patch, HIVE-20225.15.patch,
> HIVE-20225.16-branch-2.patch, HIVE-20225.17-branch-3.patch,
> HIVE-20225.18-branch-3.patch, HIVE-20225.19-branch-3.patch,
> HIVE-20225.2.patch, HIVE-20225.3.patch, HIVE-20225.4.patch,
> HIVE-20225.5-branch-2.patch, HIVE-20225.6.patch, HIVE-20225.7.patch,
> HIVE-20225.8.patch, HIVE-20225.9.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT.
> However, the Hive could not directly utilize/generate these binary format
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata
> Binary Format file transparently
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16370) Avro data type null not supported on partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-16370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-16370:
-
Attachment: HIVE-16370.1.patch
> Avro data type null not supported on partitioned tables
> ---
>
> Key: HIVE-16370
> URL: https://issues.apache.org/jira/browse/HIVE-16370
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.1.1
>Reporter: rui miranda
>Assignee: Alice Fan
>Priority: Minor
> Attachments: HIVE-16370.01-branch-1.patch, HIVE-16370.1.patch
>
>
> I was attempting to create hive tables over some partitioned Avro files. It
> seems the void data type (Avro null) is not supported on partitioned tables
> (i could not replicate the bug on an un-partitioned table).
> ---
> i managed to replicate the bug on two different hive versions.
> Hive 1.1.0-cdh5.10.0
> Hive 2.1.1-amzn-0
>
> how to replicate (avro tools are required to create the avro files):
> $ wget
> http://mirror.serversupportforum.de/apache/avro/avro-1.8.1/java/avro-tools-1.8.1.jar
> $ mkdir /tmp/avro
> $ mkdir /tmp/avro/null
> $ echo "{ \
> \"type\" : \"record\", \
> \"name\" : \"null_failure\", \
> \"namespace\" : \"org.apache.avro.null_failure\", \
> \"doc\":\"the purpose of this schema is to replicate the hive avro null
> failure\", \
> \"fields\" : [{\"name\":\"one\", \"type\":\"null\",\"default\":null}] \
> } " > /tmp/avro/null/schema.avsc
> $ echo "{\"one\":null}" > /tmp/avro/null/data.json
> $ java -jar avro-tools-1.8.1.jar fromjson --schema-file
> /tmp/avro/null/schema.avsc /tmp/avro/null/data.json > /tmp/avro/null/data.avro
> $ hdfs dfs -mkdir /tmp/avro
> $ hdfs dfs -mkdir /tmp/avro/null
> $ hdfs dfs -mkdir /tmp/avro/null/schema
> $ hdfs dfs -mkdir /tmp/avro/null/data
> $ hdfs dfs -mkdir /tmp/avro/null/data/foo=bar
> $ hdfs dfs -copyFromLocal /tmp/avro/null/schema.avsc
> /tmp/avro/null/schema/schema.avsc
> $ hdfs dfs -copyFromLocal /tmp/avro/null/data.avro
> /tmp/avro/null/data/foo=bar/data.avro
> $ hive
> hive> CREATE EXTERNAL TABLE avro_null
> PARTITIONED BY (foo string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED as INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> LOCATION
> '/tmp/avro/null/data/'
> TBLPROPERTIES (
> 'avro.schema.url'='/tmp/avro/null/schema/schema.avsc')
> ;
> OK
> Time taken: 3.127 seconds
> hive> msck repair table avro_null;
> OK
> Partitions not in metastore: avro_null:foo=bar
> Repair: Added partition to metastore avro_null:foo=bar
> Time taken: 0.712 seconds, Fetched: 2 row(s)
> hive> select * from avro_null;
> FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException:
> Failed with exception Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported
> yet.java.lang.RuntimeException: Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported yet.
> hive> select foo, count(1) from avro_null group by foo;
> OK
> bar 1
> Time taken: 29.806 seconds, Fetched: 1 row(s)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20502) Fix NPE while running skewjoin_mapjoin10.q when column stats is used.
[
https://issues.apache.org/jira/browse/HIVE-20502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607509#comment-16607509
]
Hive QA commented on HIVE-20502:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938804/HIVE-20502.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14930 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSessionSparkSessionTimeout
(batchId=245)
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSparkSessionTimeout
(batchId=245)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13648/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13648/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13648/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938804 - PreCommit-HIVE-Build
> Fix NPE while running skewjoin_mapjoin10.q when column stats is used.
> -
>
> Key: HIVE-20502
> URL: https://issues.apache.org/jira/browse/HIVE-20502
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-20502.1.patch, HIVE-20502.2.patch,
> HIVE-20502.2.patch
>
>
> Enabling {{hive.stats.fetch.column.stats}} makes this test fail during:
> {code}
> EXPLAIN
> SELECT a.*, b.* FROM T1_n151 a RIGHT OUTER JOIN T2_n88 b ON a.key = b.key
> {code}
> Seems like joinKeys is null at [this
> point|https://github.com/apache/hive/blob/48f92c31dee3983f573f2e66baaa213a0196f1ba/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L2169]
> Exception:
> {code}
> 2018-09-04T23:47:02,398 DEBUG [fef236ce-e62e-4c20-b0c0-3b15d2b336f7 main]
> annotation.StatsRulesProcFactory: STATS-JOIN[15]: detects none/multiple PK
> parents.
> 2018-09-04T23:47:02,409 ERROR [fef236ce-e62e-4c20-b0c0-3b15d2b336f7 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$JoinStatsRule.isJoinKey(StatsRulesProcFactory.java:2169)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$JoinStatsRule.updateNumNulls(StatsRulesProcFactory.java:2210)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$JoinStatsRule.updateColStats(StatsRulesProcFactory.java:2276)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$JoinStatsRule.process(StatsRulesProcFactory.java:1785)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.LevelOrderWalker.walk(LevelOrderWalker.java:143)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20514) Query with outer join filter is failing with dynamic partition join
[
https://issues.apache.org/jira/browse/HIVE-20514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607505#comment-16607505
]
Jason Dere commented on HIVE-20514:
---
+1. Interesting that this also makes a vertex vectorizable in
tez_dynpart_hashjoin_3
> Query with outer join filter is failing with dynamic partition join
> ---
>
> Key: HIVE-20514
> URL: https://issues.apache.org/jira/browse/HIVE-20514
> Project: Hive
> Issue Type: Bug
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20514.1.patch, HIVE-20514.2.patch
>
>
> *Reproducer*
> Copy the following query in {{tez_dynpart_hashjoin_1.q}} and run the test
> {code:sql}
> select
> *
> from alltypesorc a left outer join alltypesorc b on a.cint = b.cint and
> a.csmallint != a.cint
> where
> a.cint between 100 and 300
> order by a.cint;
> {code}
> *Exception*
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1536275581088_0001_5_02,
> diagnostics=[Task failed, taskId=task_1536275581088_0001_5_02_09,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1536275581088_0001_5_02_09_0:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field _col1 from [0:key, 1:value]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: cannot find field _col1 from [0:key,
> 1:value]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:537)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:56)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.initializeOp(CommonJoinOperator.java:266)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:78)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:155)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:193)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 15 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1536275581088_0001_5_02_09_1:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field _col1 from [0:key, 1:value]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
[jira] [Updated] (HIVE-20510) Vectorization : Support loading bucketed tables using sorted dynamic partition optimizer
[ https://issues.apache.org/jira/browse/HIVE-20510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-20510: -- Attachment: HIVE-20510.2.patch > Vectorization : Support loading bucketed tables using sorted dynamic > partition optimizer > > > Key: HIVE-20510 > URL: https://issues.apache.org/jira/browse/HIVE-20510 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20510.1.patch, HIVE-20510.2.patch > > > sorted dynamic partition optimizer does not work on bucketed tables when > vectorization is enabled. > > cc [~mmccline] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20481) Add the Kafka Key record as part of the row.
[
https://issues.apache.org/jira/browse/HIVE-20481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607467#comment-16607467
]
Vineet Garg commented on HIVE-20481:
Just one small comment on RB. Pending its answer +1
> Add the Kafka Key record as part of the row.
>
>
> Key: HIVE-20481
> URL: https://issues.apache.org/jira/browse/HIVE-20481
> Project: Hive
> Issue Type: Sub-task
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20481.patch
>
>
> Kafka records are keyed, most of the case this key is null or used to route
> records to the same partition. This patch adds this column as a binary column
> {code} __key{code}.
> New table layout is as follow
> {code}
> POSTHOOK: type: CREATETABLE
> POSTHOOK: Output: database:default
> POSTHOOK: Output: default@wiki_kafka_avro_table
> PREHOOK: query: describe extended wiki_kafka_avro_table
> PREHOOK: type: DESCTABLE
> PREHOOK: Input: default@wiki_kafka_avro_table
> POSTHOOK: query: describe extended wiki_kafka_avro_table
> POSTHOOK: type: DESCTABLE
> POSTHOOK: Input: default@wiki_kafka_avro_table
> isrobot boolean from deserializer
> channel string from deserializer
> timestamp string from deserializer
> flags string from deserializer
> isunpatrolled boolean from deserializer
> page string from deserializer
> diffurl string from deserializer
> added bigint from deserializer
> comment string from deserializer
> commentlength bigint from deserializer
> isnew boolean from deserializer
> isminor boolean from deserializer
> delta bigint from deserializer
> isanonymous boolean from deserializer
> user string from deserializer
> deltabucket double from deserializer
> deleted bigint from deserializer
> namespace string from deserializer
> __key binary from deserializer
> __partition int from deserializer
> __offset bigint from deserializer
> __timestamp bigint from deserializer
> __start_offsetbigint from deserializer
> __end_offset bigint from deserializer
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20502) Fix NPE while running skewjoin_mapjoin10.q when column stats is used.
[
https://issues.apache.org/jira/browse/HIVE-20502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607458#comment-16607458
]
Hive QA commented on HIVE-20502:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13648/dev-support/hive-personality.sh
|
| git revision | master / e74eb35 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13648/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix NPE while running skewjoin_mapjoin10.q when column stats is used.
> -
>
> Key: HIVE-20502
> URL: https://issues.apache.org/jira/browse/HIVE-20502
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-20502.1.patch, HIVE-20502.2.patch,
> HIVE-20502.2.patch
>
>
> Enabling {{hive.stats.fetch.column.stats}} makes this test fail during:
> {code}
> EXPLAIN
> SELECT a.*, b.* FROM T1_n151 a RIGHT OUTER JOIN T2_n88 b ON a.key = b.key
> {code}
> Seems like joinKeys is null at [this
> point|https://github.com/apache/hive/blob/48f92c31dee3983f573f2e66baaa213a0196f1ba/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L2169]
> Exception:
> {code}
> 2018-09-04T23:47:02,398 DEBUG [fef236ce-e62e-4c20-b0c0-3b15d2b336f7 main]
> annotation.StatsRulesProcFactory: STATS-JOIN[15]: detects none/multiple PK
> parents.
> 2018-09-04T23:47:02,409 ERROR [fef236ce-e62e-4c20-b0c0-3b15d2b336f7 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$JoinStatsRule.isJoinKey(StatsRulesProcFactory.java:2169)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$JoinStatsRule.updateNumNulls(StatsRulesProcFactory.java:2210)
> at
>
[jira] [Updated] (HIVE-20514) Query with outer join filter is failing with dynamic partition join
[
https://issues.apache.org/jira/browse/HIVE-20514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20514:
---
Attachment: HIVE-20514.2.patch
> Query with outer join filter is failing with dynamic partition join
> ---
>
> Key: HIVE-20514
> URL: https://issues.apache.org/jira/browse/HIVE-20514
> Project: Hive
> Issue Type: Bug
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20514.1.patch, HIVE-20514.2.patch
>
>
> *Reproducer*
> Copy the following query in {{tez_dynpart_hashjoin_1.q}} and run the test
> {code:sql}
> select
> *
> from alltypesorc a left outer join alltypesorc b on a.cint = b.cint and
> a.csmallint != a.cint
> where
> a.cint between 100 and 300
> order by a.cint;
> {code}
> *Exception*
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1536275581088_0001_5_02,
> diagnostics=[Task failed, taskId=task_1536275581088_0001_5_02_09,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1536275581088_0001_5_02_09_0:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field _col1 from [0:key, 1:value]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: cannot find field _col1 from [0:key,
> 1:value]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:537)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:56)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.initializeOp(CommonJoinOperator.java:266)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:78)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:155)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:193)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 15 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1536275581088_0001_5_02_09_1:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field _col1 from [0:key, 1:value]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
>
[jira] [Updated] (HIVE-20514) Query with outer join filter is failing with dynamic partition join
[
https://issues.apache.org/jira/browse/HIVE-20514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20514:
---
Status: Patch Available (was: Open)
> Query with outer join filter is failing with dynamic partition join
> ---
>
> Key: HIVE-20514
> URL: https://issues.apache.org/jira/browse/HIVE-20514
> Project: Hive
> Issue Type: Bug
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20514.1.patch, HIVE-20514.2.patch
>
>
> *Reproducer*
> Copy the following query in {{tez_dynpart_hashjoin_1.q}} and run the test
> {code:sql}
> select
> *
> from alltypesorc a left outer join alltypesorc b on a.cint = b.cint and
> a.csmallint != a.cint
> where
> a.cint between 100 and 300
> order by a.cint;
> {code}
> *Exception*
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1536275581088_0001_5_02,
> diagnostics=[Task failed, taskId=task_1536275581088_0001_5_02_09,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1536275581088_0001_5_02_09_0:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field _col1 from [0:key, 1:value]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: cannot find field _col1 from [0:key,
> 1:value]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:537)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:56)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.initializeOp(CommonJoinOperator.java:266)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:78)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:155)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:193)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 15 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1536275581088_0001_5_02_09_1:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field _col1 from [0:key, 1:value]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
>
[jira] [Updated] (HIVE-20514) Query with outer join filter is failing with dynamic partition join
[
https://issues.apache.org/jira/browse/HIVE-20514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20514:
---
Status: Open (was: Patch Available)
> Query with outer join filter is failing with dynamic partition join
> ---
>
> Key: HIVE-20514
> URL: https://issues.apache.org/jira/browse/HIVE-20514
> Project: Hive
> Issue Type: Bug
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20514.1.patch, HIVE-20514.2.patch
>
>
> *Reproducer*
> Copy the following query in {{tez_dynpart_hashjoin_1.q}} and run the test
> {code:sql}
> select
> *
> from alltypesorc a left outer join alltypesorc b on a.cint = b.cint and
> a.csmallint != a.cint
> where
> a.cint between 100 and 300
> order by a.cint;
> {code}
> *Exception*
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1536275581088_0001_5_02,
> diagnostics=[Task failed, taskId=task_1536275581088_0001_5_02_09,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1536275581088_0001_5_02_09_0:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field _col1 from [0:key, 1:value]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: cannot find field _col1 from [0:key,
> 1:value]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:537)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:56)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.initializeOp(CommonJoinOperator.java:266)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:78)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:155)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:193)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 15 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1536275581088_0001_5_02_09_1:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field _col1 from [0:key, 1:value]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
>
[jira] [Commented] (HIVE-20514) Query with outer join filter is failing with dynamic partition join
[
https://issues.apache.org/jira/browse/HIVE-20514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607457#comment-16607457
]
Vineet Garg commented on HIVE-20514:
[~jdere] can you take a look please?
> Query with outer join filter is failing with dynamic partition join
> ---
>
> Key: HIVE-20514
> URL: https://issues.apache.org/jira/browse/HIVE-20514
> Project: Hive
> Issue Type: Bug
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20514.1.patch, HIVE-20514.2.patch
>
>
> *Reproducer*
> Copy the following query in {{tez_dynpart_hashjoin_1.q}} and run the test
> {code:sql}
> select
> *
> from alltypesorc a left outer join alltypesorc b on a.cint = b.cint and
> a.csmallint != a.cint
> where
> a.cint between 100 and 300
> order by a.cint;
> {code}
> *Exception*
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1536275581088_0001_5_02,
> diagnostics=[Task failed, taskId=task_1536275581088_0001_5_02_09,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1536275581088_0001_5_02_09_0:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field _col1 from [0:key, 1:value]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: cannot find field _col1 from [0:key,
> 1:value]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:537)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:56)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.initializeOp(CommonJoinOperator.java:266)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:78)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:155)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:193)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 15 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1536275581088_0001_5_02_09_1:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field _col1 from [0:key, 1:value]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
>
[jira] [Updated] (HIVE-20080) TxnHandler checkLock direct sql fail with ORA-01795 , if the table has more than 1000 partitions
[
https://issues.apache.org/jira/browse/HIVE-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-20080:
Resolution: Fixed
Fix Version/s: 2.4.0
Status: Resolved (was: Patch Available)
Pushed to branch-2. Thanks, Rajkumar!
> TxnHandler checkLock direct sql fail with ORA-01795 , if the table has more
> than 1000 partitions
>
>
> Key: HIVE-20080
> URL: https://issues.apache.org/jira/browse/HIVE-20080
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.4.0
>
> Attachments: HIVE-20080.01.branch-2.patch,
> HIVE-20080.02.branch-2.patch, HIVE-20080.patch
>
>
> with Oracle as Metastore, txnhandler checkLock fail with
> "checkLockWithRetry(181398,34773) : ORA-01795: maximum number of expressions
> in a list is 1000" if the write table has more than 1000 partitions.
> complete stacktrace
> {code}
> txn.TxnHandler (TxnHandler.java:checkRetryable(2099)) - Non-retryable error
> in checkLockWithRetry(181398,34773) : ORA-01795: maximum number of
> expressions in a list is 1000
> (SQLState=42000, ErrorCode=1795)
> 2018-06-25 15:09:35,999 ERROR [pool-7-thread-197]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(203)) -
> MetaException(message:Unable to update transaction database
> java.sql.SQLSyntaxErrorException: ORA-01795: maximum number of expressions in
> a list is 1000
>
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447)
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
> at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
> at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:195)
> at
> oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:876)
> at
> oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1175)
> at
> oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1296)
> at
> oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1498)
> at
> oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:406)
> at
> com.jolbox.bonecp.StatementHandle.executeQuery(StatementHandle.java:464)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:2649)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLockWithRetry(TxnHandler.java:1126)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:895)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:6123)
> at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
> at com.sun.proxy.$Proxy11.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:12012)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:11996)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:551)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:546)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:546)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
[jira] [Commented] (HIVE-20518) TxnHandler checkLock direct sql fail with ORA-01795 , if the table has more than 1000 partitions
[
https://issues.apache.org/jira/browse/HIVE-20518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607441#comment-16607441
]
Ashutosh Chauhan commented on HIVE-20518:
-
HIVE-20080 provided fix for same issue for branch-2. This jira is for master.
> TxnHandler checkLock direct sql fail with ORA-01795 , if the table has more
> than 1000 partitions
>
>
> Key: HIVE-20518
> URL: https://issues.apache.org/jira/browse/HIVE-20518
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Labels: pull-request-available
>
> with Oracle as Metastore, txnhandler checkLock fail with
> "checkLockWithRetry(181398,34773) : ORA-01795: maximum number of expressions
> in a list is 1000" if the write table has more than 1000 partitions.
> complete stacktrace
> {code}
> txn.TxnHandler (TxnHandler.java:checkRetryable(2099)) - Non-retryable error
> in checkLockWithRetry(181398,34773) : ORA-01795: maximum number of
> expressions in a list is 1000
> (SQLState=42000, ErrorCode=1795)
> 2018-06-25 15:09:35,999 ERROR [pool-7-thread-197]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(203)) -
> MetaException(message:Unable to update transaction database
> java.sql.SQLSyntaxErrorException: ORA-01795: maximum number of expressions in
> a list is 1000
>
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447)
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
> at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
> at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:195)
> at
> oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:876)
> at
> oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1175)
> at
> oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1296)
> at
> oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1498)
> at
> oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:406)
> at
> com.jolbox.bonecp.StatementHandle.executeQuery(StatementHandle.java:464)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:2649)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLockWithRetry(TxnHandler.java:1126)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:895)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:6123)
> at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
> at com.sun.proxy.$Proxy11.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:12012)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:11996)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:551)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:546)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:546)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> )
> at
>
[jira] [Assigned] (HIVE-20518) TxnHandler checkLock direct sql fail with ORA-01795 , if the table has more than 1000 partitions
[
https://issues.apache.org/jira/browse/HIVE-20518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan reassigned HIVE-20518:
---
> TxnHandler checkLock direct sql fail with ORA-01795 , if the table has more
> than 1000 partitions
>
>
> Key: HIVE-20518
> URL: https://issues.apache.org/jira/browse/HIVE-20518
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Labels: pull-request-available
>
> with Oracle as Metastore, txnhandler checkLock fail with
> "checkLockWithRetry(181398,34773) : ORA-01795: maximum number of expressions
> in a list is 1000" if the write table has more than 1000 partitions.
> complete stacktrace
> {code}
> txn.TxnHandler (TxnHandler.java:checkRetryable(2099)) - Non-retryable error
> in checkLockWithRetry(181398,34773) : ORA-01795: maximum number of
> expressions in a list is 1000
> (SQLState=42000, ErrorCode=1795)
> 2018-06-25 15:09:35,999 ERROR [pool-7-thread-197]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(203)) -
> MetaException(message:Unable to update transaction database
> java.sql.SQLSyntaxErrorException: ORA-01795: maximum number of expressions in
> a list is 1000
>
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447)
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
> at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
> at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:195)
> at
> oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:876)
> at
> oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1175)
> at
> oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1296)
> at
> oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1498)
> at
> oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:406)
> at
> com.jolbox.bonecp.StatementHandle.executeQuery(StatementHandle.java:464)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:2649)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLockWithRetry(TxnHandler.java:1126)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:895)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:6123)
> at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
> at com.sun.proxy.$Proxy11.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:12012)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:11996)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:551)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:546)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:546)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> )
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLockWithRetry(TxnHandler.java:1131)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:895)
>
[jira] [Updated] (HIVE-20499) GetTablesOperation pull all the tables meta irrespective of auth.
[
https://issues.apache.org/jira/browse/HIVE-20499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-20499:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
> GetTablesOperation pull all the tables meta irrespective of auth.
> -
>
> Key: HIVE-20499
> URL: https://issues.apache.org/jira/browse/HIVE-20499
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
> Environment: hive-3,java-8,sqlstdauth/ranger auth enabled.
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20499.patch
>
>
> GetTablesOperation pull all the tables meta irrespective of auth.
> dbvisualizer and other ui based jdbc client pull tableemta similar to
> following operation:
> {code}
> ResultSet res = con.getMetaData().getTables("", "", "%", new String[] {
> "TABLE", "VIEW" });
> {code}
> https://github.com/rajkrrsingh/HiveServer2JDBCSample/blob/master/src/main/java/TestConnection.java#L20
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20506) HOS times out when cluster is full while Hive-on-MR waits
[
https://issues.apache.org/jira/browse/HIVE-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607431#comment-16607431
]
Sahil Takiar commented on HIVE-20506:
-
The general idea makes sense to me. To confirm my understanding this change
will essentially do the following:
* Parse the {{spark-submit}} logs and look for the YARN application id
* Create a {{YarnClient}} and check the state of the YARN app
* If the app is in {{ACCEPTED}} state (which means it has been acknowledged by
YARN, but hasn't actually been started yet)
* As long as the app is in {{ACCEPTED}} state, extend the timeout until it
transitions out of this state
Is that correct?
If thats the case, then I just have a few comments:
* Rather than extending the timeout, why not just create two separate ones? One
timeout for launching {{bin/spark-submit}} --> app = ACCEPTED and another from
app = RUNNING --> connection established.
** We probably don't want to change the meaning of the current timeout for
backwards compatibility, so maybe we could deprecate the existing one and
replace it with two new ones?
* Is there any way to avoid creating a {{YarnClient}}? I guess this is
mitigated slightly by the fact that you only create the client if the timeout
is triggered
** Just concerned about the overhead of creating a {{YarnClient}} + would this
work on a secure cluster?
** {{bin/spark-submit}} should print out something like {{Application report
for ... (state: ACCEPTED)}} perhaps we can parse the state from the logs?
* Can we move all the changes in {{RpcServer}} to a separate class? That class
is really meant to act as a generic RPC framework that is relatively
independent of the HoS logic
> HOS times out when cluster is full while Hive-on-MR waits
> -
>
> Key: HIVE-20506
> URL: https://issues.apache.org/jira/browse/HIVE-20506
> Project: Hive
> Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Major
> Attachments: HIVE-20506-CDH5.14.2.patch, HIVE-20506.1.patch, Screen
> Shot 2018-09-07 at 8.10.37 AM.png
>
>
> My understanding is as follows:
> Hive-on-MR when the cluster is full will wait for resources to be available
> before submitting a job. This is because the hadoop jar command is the
> primary mechanism Hive uses to know if a job is complete or failed.
>
> Hive-on-Spark will timeout after {{SPARK_RPC_CLIENT_CONNECT_TIMEOUT}} because
> the RPC client in the AppMaster doesn't connect back to the RPC Server in
> HS2.
> This is a behavior difference it'd be great to close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20516) alter table drop partition should be compatible with old metastore, as partition pruner does
[
https://issues.apache.org/jira/browse/HIVE-20516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607416#comment-16607416
]
Hive QA commented on HIVE-20516:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938790/temp.diff
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13646/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13646/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13646/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-09-07 17:37:24.088
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-13646/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-09-07 17:37:24.092
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
cbda0f5..e74eb35 master -> origin/master
+ git reset --hard HEAD
HEAD is now at cbda0f5 HIVE-20505: upgrade org.openjdk.jmh:jmh-core to 1.21
(Laszlo Pinter, reviewed by Adam Szita)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at e74eb35 HIVE-20432: Rewrite BETWEEN to IN for integer types for
stats estimation(Vineet Garg, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-09-07 17:37:25.217
+ rm -rf ../yetus_PreCommit-HIVE-Build-13646
+ mkdir ../yetus_PreCommit-HIVE-Build-13646
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-13646
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-13646/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc2865617867014695323.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc2865617867014695323.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc437404518118992487.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
log4j:WARN No appenders could be found for logger (DataNucleus.Persistence).
log4j:WARN Please initialize the log4j system properly.
DataNucleus Enhancer (version 4.1.17) for API "JDO"
DataNucleus Enhancer
[jira] [Commented] (HIVE-20020) Hive contrib jar should not be in lib
[ https://issues.apache.org/jira/browse/HIVE-20020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607415#comment-16607415 ] Aihua Xu commented on HIVE-20020: - The new patch looks good to me. +1. The test failure doesn't seem related but let's retry to get the tests all passed. > Hive contrib jar should not be in lib > - > > Key: HIVE-20020 > URL: https://issues.apache.org/jira/browse/HIVE-20020 > Project: Hive > Issue Type: Improvement > Components: Contrib >Reporter: Johndee Burks >Assignee: Alice Fan >Priority: Trivial > Fix For: 4.0.0 > > Attachments: HIVE-20020.3.patch, HIVE-20020.4.patch > > > Currently the way hive is packaged it includes hive-contrib-.jar in > lib, we should not include it here because it is picked up by services like > HS2. This creates a situation in which experimental features such as the > [MultiDelimitSerDe|https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java] > are accessible without understanding how to really install and use it. For > example you can create a table using HS2 via beeline with the aforementioned > SerDe and it will work as long you do not do M/R jobs. The M/R jobs do not > work because the SerDe is not in aux to get shipped into distcache. I propose > we do not package it this way and if someone would like to leverage an > experimental feature they can add it manually to their environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20513) Vectorization: Improve Fast Vector MapJoin Bytes Hash Tables
[
https://issues.apache.org/jira/browse/HIVE-20513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607411#comment-16607411
]
Hive QA commented on HIVE-20513:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938782/HIVE-20513.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14926 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parallel_orderby]
(batchId=57)
org.apache.hadoop.hive.ql.exec.tez.TestVectorMapJoinFastHashTable.checkFast3estimations
(batchId=306)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13645/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13645/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13645/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938782 - PreCommit-HIVE-Build
> Vectorization: Improve Fast Vector MapJoin Bytes Hash Tables
>
>
> Key: HIVE-20513
> URL: https://issues.apache.org/jira/browse/HIVE-20513
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20513.01.patch, HIVE-20513.02.patch
>
>
> Based on HIVE-20491 / HIVE-20503 discussions, improve Fast Vector MapJoin
> Bytes Hash Tables by only storing a one word slot entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20080) TxnHandler checkLock direct sql fail with ORA-01795 , if the table has more than 1000 partitions
[
https://issues.apache.org/jira/browse/HIVE-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607403#comment-16607403
]
Ashutosh Chauhan commented on HIVE-20080:
-
+1
[~Rajkumar Singh] Can you please create a jira for master. Lets use this one
for branch-2.
> TxnHandler checkLock direct sql fail with ORA-01795 , if the table has more
> than 1000 partitions
>
>
> Key: HIVE-20080
> URL: https://issues.apache.org/jira/browse/HIVE-20080
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20080.01.branch-2.patch,
> HIVE-20080.02.branch-2.patch, HIVE-20080.patch
>
>
> with Oracle as Metastore, txnhandler checkLock fail with
> "checkLockWithRetry(181398,34773) : ORA-01795: maximum number of expressions
> in a list is 1000" if the write table has more than 1000 partitions.
> complete stacktrace
> {code}
> txn.TxnHandler (TxnHandler.java:checkRetryable(2099)) - Non-retryable error
> in checkLockWithRetry(181398,34773) : ORA-01795: maximum number of
> expressions in a list is 1000
> (SQLState=42000, ErrorCode=1795)
> 2018-06-25 15:09:35,999 ERROR [pool-7-thread-197]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(203)) -
> MetaException(message:Unable to update transaction database
> java.sql.SQLSyntaxErrorException: ORA-01795: maximum number of expressions in
> a list is 1000
>
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447)
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
> at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
> at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:195)
> at
> oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:876)
> at
> oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1175)
> at
> oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1296)
> at
> oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1498)
> at
> oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:406)
> at
> com.jolbox.bonecp.StatementHandle.executeQuery(StatementHandle.java:464)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:2649)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLockWithRetry(TxnHandler.java:1126)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:895)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:6123)
> at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
> at com.sun.proxy.$Proxy11.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:12012)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:11996)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:551)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:546)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:546)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
[jira] [Commented] (HIVE-20499) GetTablesOperation pull all the tables meta irrespective of auth.
[
https://issues.apache.org/jira/browse/HIVE-20499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607399#comment-16607399
]
Ashutosh Chauhan commented on HIVE-20499:
-
+1
> GetTablesOperation pull all the tables meta irrespective of auth.
> -
>
> Key: HIVE-20499
> URL: https://issues.apache.org/jira/browse/HIVE-20499
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
> Environment: hive-3,java-8,sqlstdauth/ranger auth enabled.
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-20499.patch
>
>
> GetTablesOperation pull all the tables meta irrespective of auth.
> dbvisualizer and other ui based jdbc client pull tableemta similar to
> following operation:
> {code}
> ResultSet res = con.getMetaData().getTables("", "", "%", new String[] {
> "TABLE", "VIEW" });
> {code}
> https://github.com/rajkrrsingh/HiveServer2JDBCSample/blob/master/src/main/java/TestConnection.java#L20
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20508) Hive does not support user names of type "user@realm"
[ https://issues.apache.org/jira/browse/HIVE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-20508: -- Attachment: HIVE-20508.4.patch > Hive does not support user names of type "user@realm" > - > > Key: HIVE-20508 > URL: https://issues.apache.org/jira/browse/HIVE-20508 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20508.1.patch, HIVE-20508.2.patch, > HIVE-20508.3.patch, HIVE-20508.4.patch > > > Hive does not support user names of type "user@realm". This causes > authorization problems with Ranger for user names containing realms in > Kerberos environment. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20432) Rewrite BETWEEN to IN for integer types for stats estimation
[
https://issues.apache.org/jira/browse/HIVE-20432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20432:
---
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
> Rewrite BETWEEN to IN for integer types for stats estimation
>
>
> Key: HIVE-20432
> URL: https://issues.apache.org/jira/browse/HIVE-20432
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20432.1.patch, HIVE-20432.2.patch,
> HIVE-20432.3.patch, HIVE-20432.4.patch, HIVE-20432.5.patch
>
>
> IN clause provides better statistics estimation since predicates are equality
> based and it keeps into account selectivity of all values. BETWEEN is
> rewritten into {{>= AND <=}} i.e. non-equality predicates which cause worse
> estimation. For integer types BETWEEN could be rewritten into IN e.g. BETWEEN
> 4 and 7 is equivalent to IN (4,5,6,7).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20437) Handle schema evolution from float, double and decimal
[ https://issues.apache.org/jira/browse/HIVE-20437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-20437: -- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~janulatha] for the patch > Handle schema evolution from float, double and decimal > -- > > Key: HIVE-20437 > URL: https://issues.apache.org/jira/browse/HIVE-20437 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20437.1.patch, HIVE-20437.2.patch, > HIVE-20437.3.patch > > > When data created as float, double or decimal in parquet format is read back > using some other type, errors are seen. Parquet should behave just like any > other format. If the value is valid for the new type, data is retuned > otherwise null has to be returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20507) Beeline: Add a utility command to retrieve all uris from beeline-site.xml
[ https://issues.apache.org/jira/browse/HIVE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-20507: Attachment: HIVE-20507.1.patch > Beeline: Add a utility command to retrieve all uris from beeline-site.xml > - > > Key: HIVE-20507 > URL: https://issues.apache.org/jira/browse/HIVE-20507 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-20507.1.patch > > > It will be useful for some clients to get the url list when beeline-site is > present. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20513) Vectorization: Improve Fast Vector MapJoin Bytes Hash Tables
[
https://issues.apache.org/jira/browse/HIVE-20513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607369#comment-16607369
]
Hive QA commented on HIVE-20513:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} serde in master has 195 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 46 new + 25 unchanged - 13
fixed = 71 total (was 38) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13645/dev-support/hive-personality.sh
|
| git revision | master / cbda0f5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13645/yetus/diff-checkstyle-ql.txt
|
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13645/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Improve Fast Vector MapJoin Bytes Hash Tables
>
>
> Key: HIVE-20513
> URL: https://issues.apache.org/jira/browse/HIVE-20513
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20513.01.patch, HIVE-20513.02.patch
>
>
> Based on HIVE-20491 / HIVE-20503 discussions, improve Fast Vector MapJoin
> Bytes Hash Tables by only storing a one word slot entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles
[ https://issues.apache.org/jira/browse/HIVE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated HIVE-20395: - Attachment: HIVE-20395.11-branch-2.patch > Parallelize files move in the ql.metadata.Hive#replaceFiles > --- > > Key: HIVE-20395 > URL: https://issues.apache.org/jira/browse/HIVE-20395 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20395.10.patch, HIVE-20395.11-branch-2.patch > > > Files move is not parallelized in `replaceFiles` method and could be very > slow on Cloud Object Stores. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20395) Parallelize files move in the ql.metadata.Hive#replaceFiles
[ https://issues.apache.org/jira/browse/HIVE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated HIVE-20395: - Attachment: (was: HIVE-20395.10-branch-2.patch) > Parallelize files move in the ql.metadata.Hive#replaceFiles > --- > > Key: HIVE-20395 > URL: https://issues.apache.org/jira/browse/HIVE-20395 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20395.10.patch, HIVE-20395.11-branch-2.patch > > > Files move is not parallelized in `replaceFiles` method and could be very > slow on Cloud Object Stores. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-15608) Improve avg function performance for Range based window
[ https://issues.apache.org/jira/browse/HIVE-15608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu resolved HIVE-15608. - Resolution: Duplicate > Improve avg function performance for Range based window > --- > > Key: HIVE-15608 > URL: https://issues.apache.org/jira/browse/HIVE-15608 > Project: Hive > Issue Type: Improvement > Components: PTF-Windowing >Affects Versions: 2.1.1 >Reporter: Aihua Xu >Priority: Major > > Similar to HIVE-15520, we need to improve the performance for avg() function. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20515) Empty query results when using results cache and query temp dir, results cache dir in different filesystems
[ https://issues.apache.org/jira/browse/HIVE-20515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20515: -- Attachment: HIVE-20515.2.patch > Empty query results when using results cache and query temp dir, results > cache dir in different filesystems > --- > > Key: HIVE-20515 > URL: https://issues.apache.org/jira/browse/HIVE-20515 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20515.1.patch, HIVE-20515.2.patch > > > If the scratchdir for temporary query results and the results cache dir are > in different filesystems, moving the query from the temp directory to results > cache will fail. > Looking at the moveResultsToCacheDirectory() logic in QueryResultsCache.java, > I see the following issues: > - FileSystem.rename() is used, which only works if the files are on the same > filesystem. Need to use something like Hive.mvFile or something similar which > can work between different filesystems. > - The return code from rename() was not checked which might possibly have > caught the error here. This may not be applicable if a different method from > FS.rename() is used in the proper fix. > With some filesystems (noticed this with WASB), if FileSystem.rename() > returns false on failure rather than throwing an exception, then this results > in empty results showing up for the query because the return code was not > checked properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20020) Hive contrib jar should not be in lib
[ https://issues.apache.org/jira/browse/HIVE-20020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan updated HIVE-20020: - Attachment: HIVE-20020.4.patch > Hive contrib jar should not be in lib > - > > Key: HIVE-20020 > URL: https://issues.apache.org/jira/browse/HIVE-20020 > Project: Hive > Issue Type: Improvement > Components: Contrib >Reporter: Johndee Burks >Assignee: Alice Fan >Priority: Trivial > Fix For: 4.0.0 > > Attachments: HIVE-20020.3.patch, HIVE-20020.4.patch > > > Currently the way hive is packaged it includes hive-contrib-.jar in > lib, we should not include it here because it is picked up by services like > HS2. This creates a situation in which experimental features such as the > [MultiDelimitSerDe|https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java] > are accessible without understanding how to really install and use it. For > example you can create a table using HS2 via beeline with the aforementioned > SerDe and it will work as long you do not do M/R jobs. The M/R jobs do not > work because the SerDe is not in aux to get shipped into distcache. I propose > we do not package it this way and if someone would like to leverage an > experimental feature they can add it manually to their environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20515) Empty query results when using results cache and query temp dir, results cache dir in different filesystems
[ https://issues.apache.org/jira/browse/HIVE-20515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607334#comment-16607334 ] Thejas M Nair commented on HIVE-20515: -- +1 pending tests > Empty query results when using results cache and query temp dir, results > cache dir in different filesystems > --- > > Key: HIVE-20515 > URL: https://issues.apache.org/jira/browse/HIVE-20515 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20515.1.patch > > > If the scratchdir for temporary query results and the results cache dir are > in different filesystems, moving the query from the temp directory to results > cache will fail. > Looking at the moveResultsToCacheDirectory() logic in QueryResultsCache.java, > I see the following issues: > - FileSystem.rename() is used, which only works if the files are on the same > filesystem. Need to use something like Hive.mvFile or something similar which > can work between different filesystems. > - The return code from rename() was not checked which might possibly have > caught the error here. This may not be applicable if a different method from > FS.rename() is used in the proper fix. > With some filesystems (noticed this with WASB), if FileSystem.rename() > returns false on failure rather than throwing an exception, then this results > in empty results showing up for the query because the return code was not > checked properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20020) Hive contrib jar should not be in lib
[ https://issues.apache.org/jira/browse/HIVE-20020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan updated HIVE-20020: - Attachment: (was: HIVE-20020.4.patch) > Hive contrib jar should not be in lib > - > > Key: HIVE-20020 > URL: https://issues.apache.org/jira/browse/HIVE-20020 > Project: Hive > Issue Type: Improvement > Components: Contrib >Reporter: Johndee Burks >Assignee: Alice Fan >Priority: Trivial > Fix For: 4.0.0 > > Attachments: HIVE-20020.3.patch > > > Currently the way hive is packaged it includes hive-contrib-.jar in > lib, we should not include it here because it is picked up by services like > HS2. This creates a situation in which experimental features such as the > [MultiDelimitSerDe|https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java] > are accessible without understanding how to really install and use it. For > example you can create a table using HS2 via beeline with the aforementioned > SerDe and it will work as long you do not do M/R jobs. The M/R jobs do not > work because the SerDe is not in aux to get shipped into distcache. I propose > we do not package it this way and if someone would like to leverage an > experimental feature they can add it manually to their environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20508) Hive does not support user names of type "user@realm"
[
https://issues.apache.org/jira/browse/HIVE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607324#comment-16607324
]
Hive QA commented on HIVE-20508:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938771/HIVE-20508.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14928 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=324)
org.apache.hadoop.hive.ql.exec.tez.TestVectorMapJoinFastHashTable.checkFast3estimations
(batchId=306)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13644/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13644/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13644/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938771 - PreCommit-HIVE-Build
> Hive does not support user names of type "user@realm"
> -
>
> Key: HIVE-20508
> URL: https://issues.apache.org/jira/browse/HIVE-20508
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20508.1.patch, HIVE-20508.2.patch,
> HIVE-20508.3.patch
>
>
> Hive does not support user names of type "user@realm". This causes
> authorization problems with Ranger for user names containing realms in
> Kerberos environment.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20506) HOS times out when cluster is full while Hive-on-MR waits
[
https://issues.apache.org/jira/browse/HIVE-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607287#comment-16607287
]
Brock Noland commented on HIVE-20506:
-
Patch for master attached.
> HOS times out when cluster is full while Hive-on-MR waits
> -
>
> Key: HIVE-20506
> URL: https://issues.apache.org/jira/browse/HIVE-20506
> Project: Hive
> Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Major
> Attachments: HIVE-20506-CDH5.14.2.patch, HIVE-20506.1.patch, Screen
> Shot 2018-09-07 at 8.10.37 AM.png
>
>
> My understanding is as follows:
> Hive-on-MR when the cluster is full will wait for resources to be available
> before submitting a job. This is because the hadoop jar command is the
> primary mechanism Hive uses to know if a job is complete or failed.
>
> Hive-on-Spark will timeout after {{SPARK_RPC_CLIENT_CONNECT_TIMEOUT}} because
> the RPC client in the AppMaster doesn't connect back to the RPC Server in
> HS2.
> This is a behavior difference it'd be great to close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20506) HOS times out when cluster is full while Hive-on-MR waits
[
https://issues.apache.org/jira/browse/HIVE-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-20506:
Attachment: HIVE-20506.1.patch
> HOS times out when cluster is full while Hive-on-MR waits
> -
>
> Key: HIVE-20506
> URL: https://issues.apache.org/jira/browse/HIVE-20506
> Project: Hive
> Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Major
> Attachments: HIVE-20506-CDH5.14.2.patch, HIVE-20506.1.patch, Screen
> Shot 2018-09-07 at 8.10.37 AM.png
>
>
> My understanding is as follows:
> Hive-on-MR when the cluster is full will wait for resources to be available
> before submitting a job. This is because the hadoop jar command is the
> primary mechanism Hive uses to know if a job is complete or failed.
>
> Hive-on-Spark will timeout after {{SPARK_RPC_CLIENT_CONNECT_TIMEOUT}} because
> the RPC client in the AppMaster doesn't connect back to the RPC Server in
> HS2.
> This is a behavior difference it'd be great to close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20506) HOS times out when cluster is full while Hive-on-MR waits
[
https://issues.apache.org/jira/browse/HIVE-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-20506:
Status: Patch Available (was: Open)
> HOS times out when cluster is full while Hive-on-MR waits
> -
>
> Key: HIVE-20506
> URL: https://issues.apache.org/jira/browse/HIVE-20506
> Project: Hive
> Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Major
> Attachments: HIVE-20506-CDH5.14.2.patch, HIVE-20506.1.patch, Screen
> Shot 2018-09-07 at 8.10.37 AM.png
>
>
> My understanding is as follows:
> Hive-on-MR when the cluster is full will wait for resources to be available
> before submitting a job. This is because the hadoop jar command is the
> primary mechanism Hive uses to know if a job is complete or failed.
>
> Hive-on-Spark will timeout after {{SPARK_RPC_CLIENT_CONNECT_TIMEOUT}} because
> the RPC client in the AppMaster doesn't connect back to the RPC Server in
> HS2.
> This is a behavior difference it'd be great to close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20508) Hive does not support user names of type "user@realm"
[
https://issues.apache.org/jira/browse/HIVE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607247#comment-16607247
]
Hive QA commented on HIVE-20508:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
23s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
25s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 25s{color}
| {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} service: The patch generated 1 new + 27 unchanged - 0
fixed = 28 total (was 27) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
24s{color} | {color:red} service in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13644/dev-support/hive-personality.sh
|
| git revision | master / cbda0f5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13644/yetus/patch-mvninstall-service.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13644/yetus/patch-compile-service.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13644/yetus/patch-compile-service.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13644/yetus/diff-checkstyle-service.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13644/yetus/patch-findbugs-service.txt
|
| modules | C: common service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13644/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive does not support user names of type "user@realm"
> -
>
> Key: HIVE-20508
> URL: https://issues.apache.org/jira/browse/HIVE-20508
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20508.1.patch, HIVE-20508.2.patch,
> HIVE-20508.3.patch
>
>
> Hive does not support user names of type "user@realm". This causes
> authorization problems with Ranger for user names containing realms in
> Kerberos environment.
>
--
This
[jira] [Updated] (HIVE-17300) WebUI query plan graphs
[ https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-17300: - Attachment: (was: HIVE-17300.7.patch) > WebUI query plan graphs > --- > > Key: HIVE-17300 > URL: https://issues.apache.org/jira/browse/HIVE-17300 > Project: Hive > Issue Type: Sub-task > Components: Web UI >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: beginner, features, patch > Attachments: HIVE-17300.3.patch, HIVE-17300.4.patch, > HIVE-17300.5.patch, HIVE-17300.6.patch, HIVE-17300.7.patch, HIVE-17300.patch, > complete_success.png, full_mapred_stats.png, graph_with_mapred_stats.png, > last_stage_error.png, last_stage_running.png, non_mapred_task_selected.png > > > Hi all, > I’m working on a feature of the Hive WebUI Query Plan tab that would provide > the option to display the query plan as a nice graph (scroll down for > screenshots). If you click on one of the graph’s stages, the plan for that > stage appears as text below. > Stages are color-coded if they have a status (Success, Error, Running), and > the rest are grayed out. Coloring is based on status already available in the > WebUI, under the Stages tab. > There is an additional option to display stats for MapReduce tasks. This > includes the job’s ID, tracking URL (where the logs are found), and mapper > and reducer numbers/progress, among other info. > The library I’m using for the graph is called vis.js (http://visjs.org/). It > has an Apache license, and the only necessary file to be included from this > library is about 700 KB. > I tried to keep server-side changes minimal, and graph generation is taken > care of by the client. Plans with more than a given number of stages > (default: 25) won't be displayed in order to preserve resources. > I’d love to hear any and all input from the community about this feature: do > you think it’s useful, and is there anything important I’m missing? > Thanks, > Karen Coppage > Review request: https://reviews.apache.org/r/61663/ > Any input is welcome! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17300) WebUI query plan graphs
[ https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-17300: - Labels: beginner features patch (was: ) Affects Version/s: 4.0.0 Attachment: HIVE-17300.7.patch Target Version/s: 4.0.0 Status: Patch Available (was: Open) > WebUI query plan graphs > --- > > Key: HIVE-17300 > URL: https://issues.apache.org/jira/browse/HIVE-17300 > Project: Hive > Issue Type: Sub-task > Components: Web UI >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: patch, beginner, features > Attachments: HIVE-17300.3.patch, HIVE-17300.4.patch, > HIVE-17300.5.patch, HIVE-17300.6.patch, HIVE-17300.7.patch, HIVE-17300.patch, > complete_success.png, full_mapred_stats.png, graph_with_mapred_stats.png, > last_stage_error.png, last_stage_running.png, non_mapred_task_selected.png > > > Hi all, > I’m working on a feature of the Hive WebUI Query Plan tab that would provide > the option to display the query plan as a nice graph (scroll down for > screenshots). If you click on one of the graph’s stages, the plan for that > stage appears as text below. > Stages are color-coded if they have a status (Success, Error, Running), and > the rest are grayed out. Coloring is based on status already available in the > WebUI, under the Stages tab. > There is an additional option to display stats for MapReduce tasks. This > includes the job’s ID, tracking URL (where the logs are found), and mapper > and reducer numbers/progress, among other info. > The library I’m using for the graph is called vis.js (http://visjs.org/). It > has an Apache license, and the only necessary file to be included from this > library is about 700 KB. > I tried to keep server-side changes minimal, and graph generation is taken > care of by the client. Plans with more than a given number of stages > (default: 25) won't be displayed in order to preserve resources. > I’d love to hear any and all input from the community about this feature: do > you think it’s useful, and is there anything important I’m missing? > Thanks, > Karen Coppage > Review request: https://reviews.apache.org/r/61663/ > Any input is welcome! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17300) WebUI query plan graphs
[ https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-17300: - Status: Open (was: Patch Available) > WebUI query plan graphs > --- > > Key: HIVE-17300 > URL: https://issues.apache.org/jira/browse/HIVE-17300 > Project: Hive > Issue Type: Sub-task > Components: Web UI >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-17300.3.patch, HIVE-17300.4.patch, > HIVE-17300.5.patch, HIVE-17300.6.patch, HIVE-17300.7.patch, HIVE-17300.patch, > complete_success.png, full_mapred_stats.png, graph_with_mapred_stats.png, > last_stage_error.png, last_stage_running.png, non_mapred_task_selected.png > > > Hi all, > I’m working on a feature of the Hive WebUI Query Plan tab that would provide > the option to display the query plan as a nice graph (scroll down for > screenshots). If you click on one of the graph’s stages, the plan for that > stage appears as text below. > Stages are color-coded if they have a status (Success, Error, Running), and > the rest are grayed out. Coloring is based on status already available in the > WebUI, under the Stages tab. > There is an additional option to display stats for MapReduce tasks. This > includes the job’s ID, tracking URL (where the logs are found), and mapper > and reducer numbers/progress, among other info. > The library I’m using for the graph is called vis.js (http://visjs.org/). It > has an Apache license, and the only necessary file to be included from this > library is about 700 KB. > I tried to keep server-side changes minimal, and graph generation is taken > care of by the client. Plans with more than a given number of stages > (default: 25) won't be displayed in order to preserve resources. > I’d love to hear any and all input from the community about this feature: do > you think it’s useful, and is there anything important I’m missing? > Thanks, > Karen Coppage > Review request: https://reviews.apache.org/r/61663/ > Any input is welcome! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17300) WebUI query plan graphs
[ https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-17300: - Attachment: HIVE-17300.7.patch > WebUI query plan graphs > --- > > Key: HIVE-17300 > URL: https://issues.apache.org/jira/browse/HIVE-17300 > Project: Hive > Issue Type: Sub-task > Components: Web UI >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-17300.3.patch, HIVE-17300.4.patch, > HIVE-17300.5.patch, HIVE-17300.6.patch, HIVE-17300.7.patch, HIVE-17300.patch, > complete_success.png, full_mapred_stats.png, graph_with_mapred_stats.png, > last_stage_error.png, last_stage_running.png, non_mapred_task_selected.png > > > Hi all, > I’m working on a feature of the Hive WebUI Query Plan tab that would provide > the option to display the query plan as a nice graph (scroll down for > screenshots). If you click on one of the graph’s stages, the plan for that > stage appears as text below. > Stages are color-coded if they have a status (Success, Error, Running), and > the rest are grayed out. Coloring is based on status already available in the > WebUI, under the Stages tab. > There is an additional option to display stats for MapReduce tasks. This > includes the job’s ID, tracking URL (where the logs are found), and mapper > and reducer numbers/progress, among other info. > The library I’m using for the graph is called vis.js (http://visjs.org/). It > has an Apache license, and the only necessary file to be included from this > library is about 700 KB. > I tried to keep server-side changes minimal, and graph generation is taken > care of by the client. Plans with more than a given number of stages > (default: 25) won't be displayed in order to preserve resources. > I’d love to hear any and all input from the community about this feature: do > you think it’s useful, and is there anything important I’m missing? > Thanks, > Karen Coppage > Review request: https://reviews.apache.org/r/61663/ > Any input is welcome! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19653) Incorrect predicate pushdown for groupby with grouping sets
[
https://issues.apache.org/jira/browse/HIVE-19653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607213#comment-16607213
]
Hive QA commented on HIVE-19653:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12927286/HIVE-19653.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 14930 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_grouping_sets_pushdown1]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets4]
(batchId=164)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_grouping_sets_pushdown1]
(batchId=147)
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSparkSessionTimeout
(batchId=245)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13643/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13643/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13643/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12927286 - PreCommit-HIVE-Build
> Incorrect predicate pushdown for groupby with grouping sets
> ---
>
> Key: HIVE-19653
> URL: https://issues.apache.org/jira/browse/HIVE-19653
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Zhang Li
>Assignee: Zhang Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-19653.1.patch, HIVE-19653.patch
>
>
> Consider the following query:
> {code:java}
> CREATE TABLE T1(a STRING, b STRING, s BIGINT);
> INSERT OVERWRITE TABLE T1 VALUES ('', '', 123456);
> SELECT * FROM (
> SELECT a, b, sum(s)
> FROM T1
> GROUP BY a, b GROUPING SETS ((), (a), (b), (a, b))
> ) t WHERE a IS NOT NULL;
> {code}
> When hive.optimize.ppd is enabled (and hive.cbo.enable=false), the query will
> output:
> {code:java}
> NULL NULL123456
> NULL 123456
> NULL123456
> 123456
> {code}
> We can see the predicate "a IS NOT NULL" takes no effect, which is incorrect.
> When performing PPD optimization for a GBY operator, we should make sure all
> grouping sets contains the processing expr before pushdown. otherwise the
> expr value after GBY is changed and the result is wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19653) Incorrect predicate pushdown for groupby with grouping sets
[
https://issues.apache.org/jira/browse/HIVE-19653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607163#comment-16607163
]
Hive QA commented on HIVE-19653:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
11s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13643/dev-support/hive-personality.sh
|
| git revision | master / cbda0f5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13643/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Incorrect predicate pushdown for groupby with grouping sets
> ---
>
> Key: HIVE-19653
> URL: https://issues.apache.org/jira/browse/HIVE-19653
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Zhang Li
>Assignee: Zhang Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-19653.1.patch, HIVE-19653.patch
>
>
> Consider the following query:
> {code:java}
> CREATE TABLE T1(a STRING, b STRING, s BIGINT);
> INSERT OVERWRITE TABLE T1 VALUES ('', '', 123456);
> SELECT * FROM (
> SELECT a, b, sum(s)
> FROM T1
> GROUP BY a, b GROUPING SETS ((), (a), (b), (a, b))
> ) t WHERE a IS NOT NULL;
> {code}
> When hive.optimize.ppd is enabled (and hive.cbo.enable=false), the query will
> output:
> {code:java}
> NULL NULL123456
> NULL 123456
> NULL123456
> 123456
> {code}
> We can see the predicate "a IS NOT NULL" takes no effect, which is incorrect.
> When performing PPD optimization for a GBY operator, we should make sure all
> grouping sets contains the processing expr before pushdown. otherwise the
> expr value after GBY is changed and the result is wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607157#comment-16607157
]
BELUGA BEHR commented on HIVE-20255:
Failures are not related. Please accept latest patch.
[~kgyrtkirk]
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.1.patch, HIVE-20255.2.patch,
> HIVE-20255.3.patch, HIVE-20255.4.patch, HIVE-20255.5.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20020) Hive contrib jar should not be in lib
[
https://issues.apache.org/jira/browse/HIVE-20020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607142#comment-16607142
]
Hive QA commented on HIVE-20020:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938770/HIVE-20020.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14928 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.tez.TestVectorMapJoinFastHashTable.checkFast3estimations
(batchId=306)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13642/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13642/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13642/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938770 - PreCommit-HIVE-Build
> Hive contrib jar should not be in lib
> -
>
> Key: HIVE-20020
> URL: https://issues.apache.org/jira/browse/HIVE-20020
> Project: Hive
> Issue Type: Improvement
> Components: Contrib
>Reporter: Johndee Burks
>Assignee: Alice Fan
>Priority: Trivial
> Fix For: 4.0.0
>
> Attachments: HIVE-20020.3.patch, HIVE-20020.4.patch
>
>
> Currently the way hive is packaged it includes hive-contrib-.jar in
> lib, we should not include it here because it is picked up by services like
> HS2. This creates a situation in which experimental features such as the
> [MultiDelimitSerDe|https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java]
> are accessible without understanding how to really install and use it. For
> example you can create a table using HS2 via beeline with the aforementioned
> SerDe and it will work as long you do not do M/R jobs. The M/R jobs do not
> work because the SerDe is not in aux to get shipped into distcache. I propose
> we do not package it this way and if someone would like to leverage an
> experimental feature they can add it manually to their environment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20506) HOS times out when cluster is full while Hive-on-MR waits
[
https://issues.apache.org/jira/browse/HIVE-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607111#comment-16607111
]
Brock Noland commented on HIVE-20506:
-
I had a HOS query pending all night long with a 90 second handshake timeout. I
finally cleared the workload this morning and the query started just fine.
{noformat}
2018-09-07 07:11:23,771 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-3]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:12:53,801 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-4]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:14:23,832 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-5]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:15:53,862 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-6]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:17:23,892 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-7]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:18:53,922 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-0]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:20:23,951 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-1]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:21:53,981 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-2]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:23:24,011 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-3]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:24:54,042 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-4]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:26:24,072 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-5]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:27:54,103 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-6]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:29:24,133 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-7]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:30:54,163 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-0]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:32:24,193 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-1]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:33:54,222 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-2]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:35:24,252 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-3]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:36:54,282 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-4]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:38:24,311 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-5]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:39:54,342 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-6]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:41:24,372 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-7]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:42:54,401 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-0]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:44:24,445 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-1]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:45:54,474 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-2]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:47:24,504 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-3]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:48:54,536 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-4]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:50:24,565 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-5]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:51:54,595 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-6]: Extending timeout for client
4c0592e0-b1e4-4f5e-9f42-d31de39995e2
2018-09-07 07:53:24,625 INFO org.apache.hive.spark.client.rpc.RpcServer:
[RPC-Handler-7]: Extending timeout for client
[jira] [Updated] (HIVE-20506) HOS times out when cluster is full while Hive-on-MR waits
[
https://issues.apache.org/jira/browse/HIVE-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-20506:
Attachment: Screen Shot 2018-09-07 at 8.10.37 AM.png
> HOS times out when cluster is full while Hive-on-MR waits
> -
>
> Key: HIVE-20506
> URL: https://issues.apache.org/jira/browse/HIVE-20506
> Project: Hive
> Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Major
> Attachments: HIVE-20506-CDH5.14.2.patch, Screen Shot 2018-09-07 at
> 8.10.37 AM.png
>
>
> My understanding is as follows:
> Hive-on-MR when the cluster is full will wait for resources to be available
> before submitting a job. This is because the hadoop jar command is the
> primary mechanism Hive uses to know if a job is complete or failed.
>
> Hive-on-Spark will timeout after {{SPARK_RPC_CLIENT_CONNECT_TIMEOUT}} because
> the RPC client in the AppMaster doesn't connect back to the RPC Server in
> HS2.
> This is a behavior difference it'd be great to close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18945) Support "analyze table T"
[
https://issues.apache.org/jira/browse/HIVE-18945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor reassigned HIVE-18945:
---
Assignee: Laszlo Bodor
> Support "analyze table T"
> -
>
> Key: HIVE-18945
> URL: https://issues.apache.org/jira/browse/HIVE-18945
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
>
> I think it would be good to have it behave the same as
> {code}
> analyze table T compute statistics for columns
> {code}
> this could help people who not yet know the different analyze commands ; to
> run the probably most appropriate one
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20020) Hive contrib jar should not be in lib
[
https://issues.apache.org/jira/browse/HIVE-20020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607092#comment-16607092
]
Hive QA commented on HIVE-20020:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13642/dev-support/hive-personality.sh
|
| git revision | master / cbda0f5 |
| Default Java | 1.8.0_111 |
| modules | C: packaging U: packaging |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13642/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive contrib jar should not be in lib
> -
>
> Key: HIVE-20020
> URL: https://issues.apache.org/jira/browse/HIVE-20020
> Project: Hive
> Issue Type: Improvement
> Components: Contrib
>Reporter: Johndee Burks
>Assignee: Alice Fan
>Priority: Trivial
> Fix For: 4.0.0
>
> Attachments: HIVE-20020.3.patch, HIVE-20020.4.patch
>
>
> Currently the way hive is packaged it includes hive-contrib-.jar in
> lib, we should not include it here because it is picked up by services like
> HS2. This creates a situation in which experimental features such as the
> [MultiDelimitSerDe|https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java]
> are accessible without understanding how to really install and use it. For
> example you can create a table using HS2 via beeline with the aforementioned
> SerDe and it will work as long you do not do M/R jobs. The M/R jobs do not
> work because the SerDe is not in aux to get shipped into distcache. I propose
> we do not package it this way and if someone would like to leverage an
> experimental feature they can add it manually to their environment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20432) Rewrite BETWEEN to IN for integer types for stats estimation
[
https://issues.apache.org/jira/browse/HIVE-20432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607062#comment-16607062
]
Hive QA commented on HIVE-20432:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938742/HIVE-20432.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14928 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13641/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13641/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13641/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12938742 - PreCommit-HIVE-Build
> Rewrite BETWEEN to IN for integer types for stats estimation
>
>
> Key: HIVE-20432
> URL: https://issues.apache.org/jira/browse/HIVE-20432
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20432.1.patch, HIVE-20432.2.patch,
> HIVE-20432.3.patch, HIVE-20432.4.patch, HIVE-20432.5.patch
>
>
> IN clause provides better statistics estimation since predicates are equality
> based and it keeps into account selectivity of all values. BETWEEN is
> rewritten into {{>= AND <=}} i.e. non-equality predicates which cause worse
> estimation. For integer types BETWEEN could be rewritten into IN e.g. BETWEEN
> 4 and 7 is equivalent to IN (4,5,6,7).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20484) Disable Block Cache By Default With HBase SerDe
[
https://issues.apache.org/jira/browse/HIVE-20484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607060#comment-16607060
]
BELUGA BEHR commented on HIVE-20484:
{{org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[allcolref_in_udf]}} is
unrelated as I see it failing for another patch request as well.
> Disable Block Cache By Default With HBase SerDe
> ---
>
> Key: HIVE-20484
> URL: https://issues.apache.org/jira/browse/HIVE-20484
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler
>Affects Versions: 1.2.3, 2.4.0, 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-20484.1.patch, HIVE-20484.2.patch
>
>
> {quote}
> Scan instances can be set to use the block cache in the RegionServer via the
> setCacheBlocks method. For input Scans to MapReduce jobs, this should be
> false.
> https://hbase.apache.org/book.html#perf.hbase.client.blockcache
> {quote}
> However, from the Hive code, we can see that this is not the case.
> {code}
> public static final String HBASE_SCAN_CACHEBLOCKS = "hbase.scan.cacheblock";
> ...
> String scanCacheBlocks =
> tableProperties.getProperty(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS);
> if (scanCacheBlocks != null) {
> jobProperties.put(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS, scanCacheBlocks);
> }
> ...
> String scanCacheBlocks = jobConf.get(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS);
> if (scanCacheBlocks != null) {
> scan.setCacheBlocks(Boolean.parseBoolean(scanCacheBlocks));
> }
> {code}
> In the Hive code, we can see that if {{hbase.scan.cacheblock}} is not
> specified in the {{SERDEPROPERTIES}} then {{setCacheBlocks}} is not called
> and the default value of the HBase {{Scan}} class is used.
> {code:java|title=Scan.java}
> /**
>* Set whether blocks should be cached for this Scan.
>*
>* This is true by default. When true, default settings of the table and
>* family are used (this will never override caching blocks if the block
>* cache is disabled for that family or entirely).
>*
>* @param cacheBlocks if false, default settings are overridden and blocks
>* will not be cached
>*/
> public Scan setCacheBlocks(boolean cacheBlocks) {
> this.cacheBlocks = cacheBlocks;
> return this;
> }
> {code}
> Hive is doing full scans of the table with MapReduce/Spark and therefore,
> according to the HBase docs, the default behavior here should be that blocks
> are not cached. Hive should set this value to "false" by default unless the
> table {{SERDEPROPERTIES}} override this.
> {code:sql}
> -- Commands for HBase
> -- create 'test', 't'
> CREATE EXTERNAL TABLE test(value map, row_key string)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> "hbase.columns.mapping" = "t:,:key",
> "hbase.scan.cacheblock" = "false"
> );
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20360) QTest: ignore driver/qtest exclusions if -Dqfile param is set
[
https://issues.apache.org/jira/browse/HIVE-20360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607047#comment-16607047
]
Laszlo Bodor commented on HIVE-20360:
-
[~kgyrtkirk] : patch 02 uploaded, which includes some other refactoring too
(I've put masking things to QOutProcessor, where it belongs IMO)
> QTest: ignore driver/qtest exclusions if -Dqfile param is set
> -
>
> Key: HIVE-20360
> URL: https://issues.apache.org/jira/browse/HIVE-20360
> Project: Hive
> Issue Type: Improvement
> Components: Test
>Affects Versions: 3.1.0
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20360.01.patch, HIVE-20360.02.patch
>
>
> Sometimes I need to run qtests with another driver for testing purposes. In
> this case I have to edit testconfiguration.properties which seems a bit
> hacky, even if it's temporary.
> In this case, no tests will run (however there's a log message):
> {code:java}
> mvn test -Pitests -pl itests/qtest -pl itests/util -Dtest=TestCliDriver
> -Dqfile=bucketizedhiveinputformat.q
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
