[jira] [Commented] (HIVE-20385) Date: date + int fails to add days
[
https://issues.apache.org/jira/browse/HIVE-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614404#comment-16614404
]
Hive QA commented on HIVE-20385:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 21 new + 97 unchanged - 1
fixed = 118 total (was 98) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13771/dev-support/hive-personality.sh
|
| git revision | master / 35f86c7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13771/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13771/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Date: date + int fails to add days
> --
>
> Key: HIVE-20385
> URL: https://issues.apache.org/jira/browse/HIVE-20385
> Project: Hive
> Issue Type: Bug
> Components: Types
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Bruno Pusztahazi
>Priority: Minor
> Attachments: HIVE-20385.1.patch
>
>
> {code}
> select current_date + 5;
> FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments '5': No
> matching method for class
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPDTIPlus with (date, int)
> {code}
> This works in Postgres 9.6 - http://sqlfiddle.com/#!17/9eecb/19253/0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20202) Add profiler endpoint to HS2 and LLAP
[
https://issues.apache.org/jira/browse/HIVE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614381#comment-16614381
]
Hive QA commented on HIVE-20202:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939526/HIVE-20202.9.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14939 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13770/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13770/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13770/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939526 - PreCommit-HIVE-Build
> Add profiler endpoint to HS2 and LLAP
> -
>
> Key: HIVE-20202
> URL: https://issues.apache.org/jira/browse/HIVE-20202
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20202.1.patch, HIVE-20202.2.patch,
> HIVE-20202.3.patch, HIVE-20202.4.patch, HIVE-20202.5.patch,
> HIVE-20202.6.patch, HIVE-20202.7.patch, HIVE-20202.8.patch, HIVE-20202.9.patch
>
>
> Add a web endpoint for profiling based on async-profiler. This servlet should
> be added to httpserver so that HS2 and LLAP daemons can output flamegraphs
> when their /prof endpoint is hit. Since this will be based on
> [https://github.com/jvm-profiling-tools/async-profiler] heap allocation, lock
> contentions, HW counters etc. will also be supported in addition to cpu
> profiling. In most cases the profiling overhead is pretty low and is safe to
> run on production. More analysis on CPU and memory overhead here
> [https://github.com/jvm-profiling-tools/async-profiler/issues/14] and
> [https://github.com/jvm-profiling-tools/async-profiler/issues/131]
>
> For the impatient, here is the usage doc and the sample output
> [https://github.com/prasanthj/nightswatch/blob/master/README.md]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19302) Logging Too Verbose For TableNotFound
[ https://issues.apache.org/jira/browse/HIVE-19302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan updated HIVE-19302: - Attachment: HIVE-19302.1.patch Status: Patch Available (was: Open) > Logging Too Verbose For TableNotFound > - > > Key: HIVE-19302 > URL: https://issues.apache.org/jira/browse/HIVE-19302 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 3.0.0, 2.2.0 >Reporter: BELUGA BEHR >Assignee: Alice Fan >Priority: Minor > Attachments: HIVE-19302.1.patch, table_not_found_cdh6.txt > > > There is way too much logging when a user submits a query against a table > which does not exist. In an ad-hoc setting, it is quite normal that a user > fat-fingers a table name. Yet, from the perspective of the Hive > administrator, there was perhaps a major issue based on the volume and > severity of logging. Please change the logging to INFO level, and do not > present a stack trace, for such a trivial error. > > See the attached file for a sample of what logging a single "table not found" > query generates. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20558) Change default of hive.hashtable.key.count.adjustment to 0.99
[ https://issues.apache.org/jira/browse/HIVE-20558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-20558: Attachment: HIVE-20558.patch > Change default of hive.hashtable.key.count.adjustment to 0.99 > -- > > Key: HIVE-20558 > URL: https://issues.apache.org/jira/browse/HIVE-20558 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > Attachments: HIVE-20558.patch > > > Current default is 2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20202) Add profiler endpoint to HS2 and LLAP
[
https://issues.apache.org/jira/browse/HIVE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614369#comment-16614369
]
Hive QA commented on HIVE-20202:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
36s{color} | {color:red} root in master failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
42s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 42s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} common: The patch generated 12 new + 19 unchanged - 0
fixed = 31 total (was 19) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} root: The patch generated 12 new + 19 unchanged - 0
fixed = 31 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red} common generated 1 new + 64 unchanged - 0 fixed = 65
total (was 64) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| | Class org.apache.hive.http.ProfileServlet defines non-transient
non-serializable instance field process In ProfileServlet.java:instance field
process In ProfileServlet.java |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13770/dev-support/hive-personality.sh
|
| git revision | master / 35f86c7 |
| Default Java | 1.8.0_111 |
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13770/yetus/branch-compile-root.txt
|
| findbugs | v3.0.0 |
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13770/yetus/patch-compile-root.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13770/yetus/patch-compile-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13770/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13770/yetus/diff-checkstyle-root.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13770/yetus/new-findbugs-common.html
|
| modules | C: common . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13770/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add profiler endpoint to HS2 and LLAP
> -
>
> Key: HIVE-20202
> URL: https://issues.apache.org/jira/browse/HIVE-20202
> Project: Hive
> Issue Type: New Feature
>Affects Versions:
[jira] [Updated] (HIVE-20558) Change default of hive.hashtable.key.count.adjustment to 0.99
[ https://issues.apache.org/jira/browse/HIVE-20558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-20558: Status: Patch Available (was: Open) > Change default of hive.hashtable.key.count.adjustment to 0.99 > -- > > Key: HIVE-20558 > URL: https://issues.apache.org/jira/browse/HIVE-20558 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > Attachments: HIVE-20558.patch > > > Current default is 2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20558) Change default of hive.hashtable.key.count.adjustment to 0.99
[ https://issues.apache.org/jira/browse/HIVE-20558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reassigned HIVE-20558: --- > Change default of hive.hashtable.key.count.adjustment to 0.99 > -- > > Key: HIVE-20558 > URL: https://issues.apache.org/jira/browse/HIVE-20558 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > > Current default is 2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20555) HiveServer2: Preauthenticated subject for http transport is not retained for entire duration of http communication in some cases
[ https://issues.apache.org/jira/browse/HIVE-20555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reassigned HIVE-20555: --- Assignee: Vaibhav Gumashta > HiveServer2: Preauthenticated subject for http transport is not retained for > entire duration of http communication in some cases > > > Key: HIVE-20555 > URL: https://issues.apache.org/jira/browse/HIVE-20555 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.3.2, 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > As implemented in HIVE-8705, for http transport, we add the logged in > subject's credentials in the http header via a request interceptor. The > request interceptor doesn't seem to be getting used for some http traffic > (e.g. knox ssl in the same rpc). It would also be better to cache the logged > in subject for the duration of the whole session. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18876) Remove Superfluous Logging in Driver
[
https://issues.apache.org/jira/browse/HIVE-18876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-18876:
-
Issue Type: Sub-task (was: Improvement)
Parent: HIVE-19318
> Remove Superfluous Logging in Driver
>
>
> Key: HIVE-18876
> URL: https://issues.apache.org/jira/browse/HIVE-18876
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Trivial
> Labels: noob
>
> [https://github.com/apache/hive/blob/a4198f584aa0792a16d1e1eeb2ef3147403b8acb/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L2188-L2190]
> {code:java}
> if (console != null) {
> console.printInfo("OK");
> }
> {code}
> # Console can never be 'null'
> # OK is not an informative logging message, and in the HiveServer2 logs, it
> is often interwoven into the logging and is pretty much useless on its own,
> without having to trace back through the logs to see what happened before it.
> This is also printed out, even if an error occurred
> Please remove this block of code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20267) Expanding WebUI to include form to dynamically config log levels
[
https://issues.apache.org/jira/browse/HIVE-20267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614336#comment-16614336
]
Hive QA commented on HIVE-20267:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939527/HIVE-20267.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 14939 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=251)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testDataTypes (batchId=251)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testEscapedStrings (batchId=251)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testLlapInputFormatEndToEnd
(batchId=251)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testNonAsciiStrings (batchId=251)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13769/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13769/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13769/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939527 - PreCommit-HIVE-Build
> Expanding WebUI to include form to dynamically config log levels
> -
>
> Key: HIVE-20267
> URL: https://issues.apache.org/jira/browse/HIVE-20267
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-20267.1.patch, HIVE-20267.2.patch,
> HIVE-20267.3.patch
>
>
> Expanding the possibility to change the log levels during runtime, the webUI
> can be extended to interact with the Log4j2ConfiguratorServlet, this way it
> can be directly used and users/admins don't need to execute curl commands
> from commandline.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20557) Constant Folding: null > (expr) is not constant folded to NULL
[
https://issues.apache.org/jira/browse/HIVE-20557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-20557:
---
Component/s: CBO
> Constant Folding: null > (expr) is not constant folded to NULL
> --
>
> Key: HIVE-20557
> URL: https://issues.apache.org/jira/browse/HIVE-20557
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Gopal V
>Priority: Major
>
> {code}
> create temporary table foo(col1 int, col2 int, col3 int, col5 int, col9 int)
> stored as orc;
> explain select count(1) from foo where
> CASE WHEN (col5 is not null) THEN (((col1 / col5) > (col9 / col3))) ELSE
> ((null > (col9 / col3))) END;
> explain select count(1), max((col9/col3) > null) from foo where
> CASE WHEN (col5 is not null) THEN (((col1 / col5) > (col9 / col3))) ELSE
> (((col9 / col3) > null)) END;
> {code}
> There's no sane way to write this query, but this is actually generated by
> CBO for an expression which looks like
> {code}
> case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total /
> t_w_firstyear.year_total else null end
>> case when t_s_firstyear.year_total > 0 then
> t_s_secyear.year_total / t_s_firstyear.year_total else null end
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20557) Constant Folding: null > (expr) is not constant folded to NULL
[
https://issues.apache.org/jira/browse/HIVE-20557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-20557:
---
Affects Version/s: 4.0.0
3.1.0
> Constant Folding: null > (expr) is not constant folded to NULL
> --
>
> Key: HIVE-20557
> URL: https://issues.apache.org/jira/browse/HIVE-20557
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Gopal V
>Priority: Major
>
> {code}
> create temporary table foo(col1 int, col2 int, col3 int, col5 int, col9 int)
> stored as orc;
> explain select count(1) from foo where
> CASE WHEN (col5 is not null) THEN (((col1 / col5) > (col9 / col3))) ELSE
> ((null > (col9 / col3))) END;
> explain select count(1), max((col9/col3) > null) from foo where
> CASE WHEN (col5 is not null) THEN (((col1 / col5) > (col9 / col3))) ELSE
> (((col9 / col3) > null)) END;
> {code}
> There's no sane way to write this query, but this is actually generated by
> CBO for an expression which looks like
> {code}
> case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total /
> t_w_firstyear.year_total else null end
>> case when t_s_firstyear.year_total > 0 then
> t_s_secyear.year_total / t_s_firstyear.year_total else null end
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20267) Expanding WebUI to include form to dynamically config log levels
[
https://issues.apache.org/jira/browse/HIVE-20267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614299#comment-16614299
]
Hive QA commented on HIVE-20267:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
56s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13769/dev-support/hive-personality.sh
|
| git revision | master / 35f86c7 |
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13769/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Expanding WebUI to include form to dynamically config log levels
> -
>
> Key: HIVE-20267
> URL: https://issues.apache.org/jira/browse/HIVE-20267
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Minor
> Attachments: HIVE-20267.1.patch, HIVE-20267.2.patch,
> HIVE-20267.3.patch
>
>
> Expanding the possibility to change the log levels during runtime, the webUI
> can be extended to interact with the Log4j2ConfiguratorServlet, this way it
> can be directly used and users/admins don't need to execute curl commands
> from commandline.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20552) Get Schema from LogicalPlan faster
[
https://issues.apache.org/jira/browse/HIVE-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614293#comment-16614293
]
Hive QA commented on HIVE-20552:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939550/HIVE-20552.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14939 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13768/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13768/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13768/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939550 - PreCommit-HIVE-Build
> Get Schema from LogicalPlan faster
> --
>
> Key: HIVE-20552
> URL: https://issues.apache.org/jira/browse/HIVE-20552
> Project: Hive
> Issue Type: Improvement
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20552.1.patch, HIVE-20552.2.patch
>
>
> To get the schema of a query faster, it currently needs to compile, optimize,
> and generate a TezPlan, which creates extra overhead when only the
> LogicalPlan is needed.
> 1. Copy the method \{{HiveMaterializedViewsRegistry.parseQuery}}, making it
> \{{public static}} and putting it in a utility class.
> 2. Change the return statement of the method to \{{return
> analyzer.getResultSchema();}}
> 3. Change the return type of the method to \{{List}}
> 4. Call the new method from \{{GenericUDTFGetSplits.createPlanFragment}}
> replacing the current code which does this:
> {code}
> if(num == 0) {
> //Schema only
> return new PlanFragment(null, schema, null);
> }
> {code}
> moving the call earlier in \{{getPlanFragment}} ... right after the HiveConf
> is created ... bypassing the code that uses \{{HiveTxnManager}} and
> \{{Driver}}.
> 5. Convert the \{{List}} to
> \{{org.apache.hadoop.hive.llap.Schema}}.
> 6. return from \{{getPlanFragment}} by returning \{{new PlanFragment(null,
> schema, null)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20552) Get Schema from LogicalPlan faster
[
https://issues.apache.org/jira/browse/HIVE-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614260#comment-16614260
]
Hive QA commented on HIVE-20552:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 22 unchanged - 1 fixed
= 23 total (was 23) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13768/dev-support/hive-personality.sh
|
| git revision | master / 35f86c7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13768/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13768/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Get Schema from LogicalPlan faster
> --
>
> Key: HIVE-20552
> URL: https://issues.apache.org/jira/browse/HIVE-20552
> Project: Hive
> Issue Type: Improvement
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20552.1.patch, HIVE-20552.2.patch
>
>
> To get the schema of a query faster, it currently needs to compile, optimize,
> and generate a TezPlan, which creates extra overhead when only the
> LogicalPlan is needed.
> 1. Copy the method \{{HiveMaterializedViewsRegistry.parseQuery}}, making it
> \{{public static}} and putting it in a utility class.
> 2. Change the return statement of the method to \{{return
> analyzer.getResultSchema();}}
> 3. Change the return type of the method to \{{List}}
> 4. Call the new method from \{{GenericUDTFGetSplits.createPlanFragment}}
> replacing the current code which does this:
> {code}
> if(num == 0) {
> //Schema only
> return new PlanFragment(null, schema, null);
> }
> {code}
> moving the call earlier in \{{getPlanFragment}} ... right after the HiveConf
> is created ... bypassing the code that uses \{{HiveTxnManager}} and
> \{{Driver}}.
> 5. Convert the \{{List}} to
> \{{org.apache.hadoop.hive.llap.Schema}}.
> 6. return from \{{getPlanFragment}} by returning \{{new PlanFragment(null,
> schema, null)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614242#comment-16614242
]
Misha Dmitriev commented on HIVE-17684:
---
[~stakiar] this looks generally good. I noticed one or two typos; will take a
deeper look into the code tomorrow.
The main question is: did you actually see the old
{{MapJoinMemoryExhaustionHandler}} working correctly, at least with Hive-on-MR?
My impression so far was that it's just broken, because the JVM cannot report
the exact amount of used memory (i.e. occupied by non-garbage objects), except
maybe after a full GC. If so, then keeping this code around will mean adding
more "cruft" to Hive, which is already not the cleanest code base, and make the
job of maintainters, yourself included, harder. Please let me know what you
think.
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-17684.01.patch, HIVE-17684.02.patch,
> HIVE-17684.03.patch, HIVE-17684.04.patch, HIVE-17684.05.patch,
> HIVE-17684.06.patch, HIVE-17684.07.patch, HIVE-17684.08.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect
> scenarios where the small table is taking too much space in memory, in which
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic
> to estimate how much memory the {{HashMap}} is consuming:
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() /
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be
> inaccurate. The value returned by this method returns all reachable and
> unreachable memory on the heap, so there may be a bunch of garbage data, and
> the JVM just hasn't taken the time to reclaim it all. This can lead to
> intermittent failures of this check even though a simple GC would have
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS.
> In Hive-on-MR this probably made sense to use because every Hive task was run
> in a dedicated container, so a Hive Task could assume it created most of the
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks
> running in a single executor, each doing different things.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20550) Switch WebHCat to use beeline to submit Hive queries
[
https://issues.apache.org/jira/browse/HIVE-20550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614241#comment-16614241
]
Hive QA commented on HIVE-20550:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939510/HIVE-20550.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14939 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13767/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13767/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13767/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939510 - PreCommit-HIVE-Build
> Switch WebHCat to use beeline to submit Hive queries
>
>
> Key: HIVE-20550
> URL: https://issues.apache.org/jira/browse/HIVE-20550
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20550.1.patch
>
>
> Since hive cli is deprecated, we shall switch WebHCat to use beeline instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20273) Spark jobs aren't cancelled if getSparkJobInfo or getSparkStagesInfo
[
https://issues.apache.org/jira/browse/HIVE-20273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-20273:
Description:
HIVE-19053 and HIVE-19733 added handling of {{InterruptedException}} to
{{RemoteSparkJobStatus#getSparkJobInfo}} and
{{RemoteSparkJobStatus#getSparkStagesInfo}}. Now, these methods catch
{{InterruptedException}} and wrap the exception in a {{HiveException}} and then
throw the new {{HiveException}}.
This new {{HiveException}} is then caught in
{{RemoteSparkJobMonitor#startMonitor}} which then looks for exceptions that
match the condition:
{code:java}
if (e instanceof InterruptedException ||
(e instanceof HiveException && e.getCause() instanceof
InterruptedException))
{code}
If this condition is met (in this case it is), the exception will again be
wrapped in another {{HiveException}} and is thrown again. So the final
exception is a {{HiveException}} that wraps a {{HiveException}} that wraps an
{{InterruptedException}}.
The double nesting of hive exception causes the logic in
{{SparkTask#setSparkException}} to break, and doesn't cause {{killJob}} to get
triggered.
This causes interrupted Hive queries to not kill their corresponding Spark jobs.
was:
HIVE-19053 and HIVE-19733 added handling of {{InterruptedException}} to
{{RemoteSparkJobStatus#getSparkJobInfo}} and
{{RemoteSparkJobStatus#getSparkStagesInfo}}. These methods catch
{{InterruptedException}} and wrap the exception in a {{HiveException}} and then
throw the new {{HiveException}}.
This new {{HiveException}} is then caught in
{{RemoteSparkJobMonitor#startMonitor}} which then looks for exceptions that
match the condition:
{code}
if (e instanceof InterruptedException ||
(e instanceof HiveException && e.getCause() instanceof
InterruptedException))
{code}
If this condition is met (in this case it is), the exception will again be
wrapped in another {{HiveException}} and is thrown again. So the final
exception is a {{HiveException}} that wraps a {{HiveException}} that wraps an
{{InterruptedException}}.
The double nesting of hive exception causes the logic in
{{SparkTask#setSparkException}} to break, and doesn't cause {{killJob}} to get
triggered.
> Spark jobs aren't cancelled if getSparkJobInfo or getSparkStagesInfo
>
>
> Key: HIVE-20273
> URL: https://issues.apache.org/jira/browse/HIVE-20273
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-20273.1.patch
>
>
> HIVE-19053 and HIVE-19733 added handling of {{InterruptedException}} to
> {{RemoteSparkJobStatus#getSparkJobInfo}} and
> {{RemoteSparkJobStatus#getSparkStagesInfo}}. Now, these methods catch
> {{InterruptedException}} and wrap the exception in a {{HiveException}} and
> then throw the new {{HiveException}}.
> This new {{HiveException}} is then caught in
> {{RemoteSparkJobMonitor#startMonitor}} which then looks for exceptions that
> match the condition:
> {code:java}
> if (e instanceof InterruptedException ||
> (e instanceof HiveException && e.getCause() instanceof
> InterruptedException))
> {code}
> If this condition is met (in this case it is), the exception will again be
> wrapped in another {{HiveException}} and is thrown again. So the final
> exception is a {{HiveException}} that wraps a {{HiveException}} that wraps an
> {{InterruptedException}}.
> The double nesting of hive exception causes the logic in
> {{SparkTask#setSparkException}} to break, and doesn't cause {{killJob}} to
> get triggered.
> This causes interrupted Hive queries to not kill their corresponding Spark
> jobs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20273) Spark jobs aren't cancelled if getSparkJobInfo or getSparkStagesInfo
[
https://issues.apache.org/jira/browse/HIVE-20273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-20273:
Description:
HIVE-19053 and HIVE-19733 added handling of {{InterruptedException}} to
{{RemoteSparkJobStatus#getSparkJobInfo}} and
{{RemoteSparkJobStatus#getSparkStagesInfo}}. These methods catch
{{InterruptedException}} and wrap the exception in a {{HiveException}} and then
throw the new {{HiveException}}.
This new {{HiveException}} is then caught in
{{RemoteSparkJobMonitor#startMonitor}} which then looks for exceptions that
match the condition:
{code}
if (e instanceof InterruptedException ||
(e instanceof HiveException && e.getCause() instanceof
InterruptedException))
{code}
If this condition is met (in this case it is), the exception will again be
wrapped in another {{HiveException}} and is thrown again. So the final
exception is a {{HiveException}} that wraps a {{HiveException}} that wraps an
{{InterruptedException}}.
The double nesting of hive exception causes the logic in
{{SparkTask#setSparkException}} to break, and doesn't cause {{killJob}} to get
triggered.
was:HIVE-19053 and HIVE-19733 add handling of {{InterruptedException}} to
{{#getSparkJobInfo}} and {{#getSparkStagesInfo}} in {{RemoteSparkJobStatus}},
but that means the {{InterruptedException}} is wrapped in a {{HiveException}}
and then thrown. The {{HiveException}} is then cause in
{{RemoteSparkJobMonitor}} and then wrapped in another Hive exception. The
double nesting of hive exception causes the logic in
{{SparkTask#setSparkException}} to break, and it doesn't kill the job if an
interrupted exception is thrown.
> Spark jobs aren't cancelled if getSparkJobInfo or getSparkStagesInfo
>
>
> Key: HIVE-20273
> URL: https://issues.apache.org/jira/browse/HIVE-20273
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-20273.1.patch
>
>
> HIVE-19053 and HIVE-19733 added handling of {{InterruptedException}} to
> {{RemoteSparkJobStatus#getSparkJobInfo}} and
> {{RemoteSparkJobStatus#getSparkStagesInfo}}. These methods catch
> {{InterruptedException}} and wrap the exception in a {{HiveException}} and
> then throw the new {{HiveException}}.
> This new {{HiveException}} is then caught in
> {{RemoteSparkJobMonitor#startMonitor}} which then looks for exceptions that
> match the condition:
> {code}
> if (e instanceof InterruptedException ||
> (e instanceof HiveException && e.getCause() instanceof
> InterruptedException))
> {code}
> If this condition is met (in this case it is), the exception will again be
> wrapped in another {{HiveException}} and is thrown again. So the final
> exception is a {{HiveException}} that wraps a {{HiveException}} that wraps an
> {{InterruptedException}}.
> The double nesting of hive exception causes the logic in
> {{SparkTask#setSparkException}} to break, and doesn't cause {{killJob}} to
> get triggered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20550) Switch WebHCat to use beeline to submit Hive queries
[
https://issues.apache.org/jira/browse/HIVE-20550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614213#comment-16614213
]
Hive QA commented on HIVE-20550:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
40s{color} | {color:red} branch/hcatalog no findbugs output file
(hcatalog/target/findbugsXml.xml) {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} hcatalog/webhcat/svr in master has 96 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} hcatalog: The patch generated 9 new + 130 unchanged -
1 fixed = 139 total (was 131) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} hcatalog/webhcat/svr: The patch generated 9 new + 130
unchanged - 1 fixed = 139 total (was 131) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
45s{color} | {color:red} patch/hcatalog no findbugs output file
(hcatalog/target/findbugsXml.xml) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13767/dev-support/hive-personality.sh
|
| git revision | master / 35f86c7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13767/yetus/branch-findbugs-hcatalog.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13767/yetus/diff-checkstyle-hcatalog.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13767/yetus/diff-checkstyle-hcatalog_webhcat_svr.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13767/yetus/patch-findbugs-hcatalog.txt
|
| modules | C: hcatalog hcatalog/webhcat/svr U: hcatalog |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13767/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Switch WebHCat to use beeline to submit Hive queries
>
>
> Key: HIVE-20550
> URL: https://issues.apache.org/jira/browse/HIVE-20550
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20550.1.patch
>
>
> Since hive cli is
[jira] [Updated] (HIVE-20519) Remove 30m min value for hive.spark.session.timeout
[
https://issues.apache.org/jira/browse/HIVE-20519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-20519:
Attachment: HIVE-20519.2.patch
> Remove 30m min value for hive.spark.session.timeout
> ---
>
> Key: HIVE-20519
> URL: https://issues.apache.org/jira/browse/HIVE-20519
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-20519.1.patch, HIVE-20519.2.patch
>
>
> In HIVE-14162 we added the config \{{hive.spark.session.timeout}} which
> provided a way to time out Spark sessions that are active for a long period
> of time. The config has a lower bound of 30m which we should remove. It
> should be possible for users to configure this value so the HoS session is
> closed as soon as the query is complete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20552) Get Schema from LogicalPlan faster
[
https://issues.apache.org/jira/browse/HIVE-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614194#comment-16614194
]
Jesus Camacho Rodriguez commented on HIVE-20552:
[~teddy.choi], can you move the static methods from
HiveMaterializedViewsRegistry to one of the Hive utility classes, e.g.,
ParseUtils? It does not seem right that GenericUDTFGetSplits needs to call
HiveMaterializedViewsRegistry. Thanks
> Get Schema from LogicalPlan faster
> --
>
> Key: HIVE-20552
> URL: https://issues.apache.org/jira/browse/HIVE-20552
> Project: Hive
> Issue Type: Improvement
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20552.1.patch, HIVE-20552.2.patch
>
>
> To get the schema of a query faster, it currently needs to compile, optimize,
> and generate a TezPlan, which creates extra overhead when only the
> LogicalPlan is needed.
> 1. Copy the method \{{HiveMaterializedViewsRegistry.parseQuery}}, making it
> \{{public static}} and putting it in a utility class.
> 2. Change the return statement of the method to \{{return
> analyzer.getResultSchema();}}
> 3. Change the return type of the method to \{{List}}
> 4. Call the new method from \{{GenericUDTFGetSplits.createPlanFragment}}
> replacing the current code which does this:
> {code}
> if(num == 0) {
> //Schema only
> return new PlanFragment(null, schema, null);
> }
> {code}
> moving the call earlier in \{{getPlanFragment}} ... right after the HiveConf
> is created ... bypassing the code that uses \{{HiveTxnManager}} and
> \{{Driver}}.
> 5. Convert the \{{List}} to
> \{{org.apache.hadoop.hive.llap.Schema}}.
> 6. return from \{{getPlanFragment}} by returning \{{new PlanFragment(null,
> schema, null)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17300) WebUI query plan graphs
[
https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614195#comment-16614195
]
Hive QA commented on HIVE-17300:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939504/HIVE-17300.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14940 tests
executed
*Failed tests:*
{noformat}
TestEncryptedHDFSCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=185)
[encryption_insert_partition_dynamic.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13766/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13766/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13766/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939504 - PreCommit-HIVE-Build
> WebUI query plan graphs
> ---
>
> Key: HIVE-17300
> URL: https://issues.apache.org/jira/browse/HIVE-17300
> Project: Hive
> Issue Type: Sub-task
> Components: Web UI
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: beginner, features, patch
> Attachments: HIVE-17300.3.patch, HIVE-17300.4.patch,
> HIVE-17300.5.patch, HIVE-17300.6.patch, HIVE-17300.7.patch,
> HIVE-17300.7.patch, HIVE-17300.patch, complete_success.png,
> full_mapred_stats.png, graph_with_mapred_stats.png, last_stage_error.png,
> last_stage_running.png, non_mapred_task_selected.png
>
>
> Hi all,
> I’m working on a feature of the Hive WebUI Query Plan tab that would provide
> the option to display the query plan as a nice graph (scroll down for
> screenshots). If you click on one of the graph’s stages, the plan for that
> stage appears as text below.
> Stages are color-coded if they have a status (Success, Error, Running), and
> the rest are grayed out. Coloring is based on status already available in the
> WebUI, under the Stages tab.
> There is an additional option to display stats for MapReduce tasks. This
> includes the job’s ID, tracking URL (where the logs are found), and mapper
> and reducer numbers/progress, among other info.
> The library I’m using for the graph is called vis.js (http://visjs.org/). It
> has an Apache license, and the only necessary file to be included from this
> library is about 700 KB.
> I tried to keep server-side changes minimal, and graph generation is taken
> care of by the client. Plans with more than a given number of stages
> (default: 25) won't be displayed in order to preserve resources.
> I’d love to hear any and all input from the community about this feature: do
> you think it’s useful, and is there anything important I’m missing?
> Thanks,
> Karen Coppage
> Review request: https://reviews.apache.org/r/61663/
> Any input is welcome!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17300) WebUI query plan graphs
[
https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614182#comment-16614182
]
Hive QA commented on HIVE-17300:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
0s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} common: The patch generated 1 new + 449 unchanged - 0
fixed = 450 total (was 449) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 5 new + 186 unchanged - 0
fixed = 191 total (was 186) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13766/dev-support/hive-personality.sh
|
| git revision | master / 35f86c7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13766/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13766/yetus/diff-checkstyle-ql.txt
|
| modules | C: common itests/hive-unit ql service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13766/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> WebUI query plan graphs
> ---
>
> Key: HIVE-17300
> URL: https://issues.apache.org/jira/browse/HIVE-17300
> Project: Hive
> Issue Type: Sub-task
> Components: Web UI
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: beginner, features, patch
>
[jira] [Commented] (HIVE-20420) Provide a fallback authorizer when no other authorizer is in use
[ https://issues.apache.org/jira/browse/HIVE-20420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614181#comment-16614181 ] Daniel Dai commented on HIVE-20420: --- Retrigger test. > Provide a fallback authorizer when no other authorizer is in use > > > Key: HIVE-20420 > URL: https://issues.apache.org/jira/browse/HIVE-20420 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20420.1.patch, HIVE-20420.2.patch, > HIVE-20420.3.patch, HIVE-20420.4.patch, HIVE-20420.5.patch, > HIVE-20420.6.patch, HIVE-20420.7.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20420) Provide a fallback authorizer when no other authorizer is in use
[ https://issues.apache.org/jira/browse/HIVE-20420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20420: -- Attachment: HIVE-20420.7.patch > Provide a fallback authorizer when no other authorizer is in use > > > Key: HIVE-20420 > URL: https://issues.apache.org/jira/browse/HIVE-20420 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20420.1.patch, HIVE-20420.2.patch, > HIVE-20420.3.patch, HIVE-20420.4.patch, HIVE-20420.5.patch, > HIVE-20420.6.patch, HIVE-20420.7.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20420) Provide a fallback authorizer when no other authorizer is in use
[
https://issues.apache.org/jira/browse/HIVE-20420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614162#comment-16614162
]
Hive QA commented on HIVE-20420:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939501/HIVE-20420.6.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14947 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.service.auth.TestCustomAuthentication.org.apache.hive.service.auth.TestCustomAuthentication
(batchId=247)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13765/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13765/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13765/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939501 - PreCommit-HIVE-Build
> Provide a fallback authorizer when no other authorizer is in use
>
>
> Key: HIVE-20420
> URL: https://issues.apache.org/jira/browse/HIVE-20420
> Project: Hive
> Issue Type: New Feature
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20420.1.patch, HIVE-20420.2.patch,
> HIVE-20420.3.patch, HIVE-20420.4.patch, HIVE-20420.5.patch, HIVE-20420.6.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20556) Expose an API to retrieve the TBL_ID from TBLS in the metastore tables
[ https://issues.apache.org/jira/browse/HIVE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M reassigned HIVE-20556: -- Assignee: Jaume M > Expose an API to retrieve the TBL_ID from TBLS in the metastore tables > -- > > Key: HIVE-20556 > URL: https://issues.apache.org/jira/browse/HIVE-20556 > Project: Hive > Issue Type: New Feature > Components: Metastore, Standalone Metastore >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > > We have two options to do this > 1) Use the current MTable and add a field for this value > 2) Add an independent API call to the metastore that would return the TBL_ID. > Option 1 is preferable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20420) Provide a fallback authorizer when no other authorizer is in use
[
https://issues.apache.org/jira/browse/HIVE-20420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614142#comment-16614142
]
Hive QA commented on HIVE-20420:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 12 new + 1 unchanged - 0 fixed
= 13 total (was 1) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13765/dev-support/hive-personality.sh
|
| git revision | master / 35f86c7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13765/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13765/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Provide a fallback authorizer when no other authorizer is in use
>
>
> Key: HIVE-20420
> URL: https://issues.apache.org/jira/browse/HIVE-20420
> Project: Hive
> Issue Type: New Feature
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20420.1.patch, HIVE-20420.2.patch,
> HIVE-20420.3.patch, HIVE-20420.4.patch, HIVE-20420.5.patch, HIVE-20420.6.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20549) Allow user set query tag, and kill query with tag
[
https://issues.apache.org/jira/browse/HIVE-20549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20549:
--
Attachment: HIVE-20549.2.patch
> Allow user set query tag, and kill query with tag
> -
>
> Key: HIVE-20549
> URL: https://issues.apache.org/jira/browse/HIVE-20549
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20549.1.patch, HIVE-20549.2.patch
>
>
> HIVE-19924 add capacity for replication job set a query tag and kill the
> replication distcp job with the tag. Here I make it more general, user can
> set arbitrary "hive.query.tag" in sql script, and kill query with the tag.
> Hive will cancel the corresponding operation in hs2, along with Tez/MR
> application launched for the query. For example:
> {code}
> set hive.query.tag=mytag;
> select . -- long running query
> {code}
> In another session:
> {code}
> kill query 'mytag';
> {code}
> There're limitations in the implementation:
> 1. No tag duplication check. There's nothing to prevent conflicting tag for
> same user, and kill query will kill queries share the same tag. However, kill
> query will not kill queries from different user unless admin. So different
> user might share the same tag
> 2. In multiple hs2 environment, kill statement should be issued to all hs2 to
> make sure the corresponding operation is canceled. When beeline/jdbc connects
> to hs2 using regular way (zookeeper url), the session will connect to random
> hs2, which might be different than the hs2 where query run on. User can use
> HiveConnection.getAllUrls or beeline --getUrlsFromBeelineSite (HIVE-20507) to
> get a list of all hs2 instances.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20095) Fix jdbc external table feature
[ https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Doron updated HIVE-20095: -- Status: Open (was: Patch Available) > Fix jdbc external table feature > --- > > Key: HIVE-20095 > URL: https://issues.apache.org/jira/browse/HIVE-20095 > Project: Hive > Issue Type: Bug >Reporter: Jonathan Doron >Assignee: Jonathan Doron >Priority: Major > Attachments: HIVE-20095.1.patch, HIVE-20095.2.patch, > HIVE-20095.3.patch, HIVE-20095.4.patch, HIVE-20095.5.patch, > HIVE-20095.6.patch, HIVE-20095.7.patch, HIVE-20095.7.patch > > > It seems like the committed code for HIVE-19161 > (7584b3276bebf64aa006eaa162c0a6264d8fcb56) reverted some of HIVE-18423 > updates, and therefore some of the external table queries are not working > correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20095) Fix jdbc external table feature
[ https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Doron updated HIVE-20095: -- Attachment: HIVE-20095.8.patch Status: Patch Available (was: Open) > Fix jdbc external table feature > --- > > Key: HIVE-20095 > URL: https://issues.apache.org/jira/browse/HIVE-20095 > Project: Hive > Issue Type: Bug >Reporter: Jonathan Doron >Assignee: Jonathan Doron >Priority: Major > Attachments: HIVE-20095.1.patch, HIVE-20095.2.patch, > HIVE-20095.3.patch, HIVE-20095.4.patch, HIVE-20095.5.patch, > HIVE-20095.6.patch, HIVE-20095.7.patch, HIVE-20095.7.patch, HIVE-20095.8.patch > > > It seems like the committed code for HIVE-19161 > (7584b3276bebf64aa006eaa162c0a6264d8fcb56) reverted some of HIVE-18423 > updates, and therefore some of the external table queries are not working > correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20549) Allow user set query tag, and kill query with tag
[
https://issues.apache.org/jira/browse/HIVE-20549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614122#comment-16614122
]
Hive QA commented on HIVE-20549:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939498/HIVE-20549.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 32 failed/errored test(s), 14939 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert]
(batchId=193)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_kill_query]
(batchId=99)
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSparkSessionTimeout
(batchId=245)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveAndKill
(batchId=251)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveBackKill
(batchId=251)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
(batchId=251)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerDAGTotalTasks
(batchId=249)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerSlowQueryExecutionTime
(batchId=249)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerTotalLaunchedTasks
(batchId=249)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerVertexTotalTasks
(batchId=249)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testMultipleTriggers1
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testMultipleTriggers2
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedFiles
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomReadOps
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerDagRawInputSplitsKill
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerDagTotalTasks
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerDefaultRawInputSplits
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesRead
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesWrite
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerShortQueryElapsedTime
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryElapsedTime
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryExecutionTime
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerTotalTasks
(batchId=251)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerVertexRawInputSplitsKill
(batchId=251)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13764/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13764/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13764/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 32 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939498 - PreCommit-HIVE-Build
> Allow user set query tag, and kill query with tag
> -
>
> Key: HIVE-20549
> URL: https://issues.apache.org/jira/browse/HIVE-20549
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20549.1.patch
>
>
> HIVE-19924 add capacity for replication job set a query tag and kill the
> replication distcp job with the tag. Here I make it more
[jira] [Commented] (HIVE-20549) Allow user set query tag, and kill query with tag
[
https://issues.apache.org/jira/browse/HIVE-20549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614100#comment-16614100
]
Hive QA commented on HIVE-20549:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} common: The patch generated 1 new + 425 unchanged - 0
fixed = 426 total (was 425) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 7 new + 7
unchanged - 1 fixed = 14 total (was 8) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 4 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13764/dev-support/hive-personality.sh
|
| git revision | master / 35f86c7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13764/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13764/yetus/diff-checkstyle-itests_hive-unit.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13764/yetus/whitespace-tabs.txt
|
| modules | C: common itests/hive-unit ql service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13764/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Allow user set query tag, and kill query with tag
> -
>
> Key: HIVE-20549
> URL: https://issues.apache.org/jira/browse/HIVE-20549
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>
[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614097#comment-16614097
]
Sahil Takiar commented on HIVE-17684:
-
[~mi...@cloudera.com] I attached an updated path. I did a good amount of
re-factoring to the patch. The main reason is that we only want these changes
to apply to Hive-on-Spark, not Hive-on-MR (at least for now). The reason is
that we are only seeing issues when running Hive-on-Spark, not Hive-on-MR.
The main changes are as follows:
* Created a new interface called {{MemoryExhaustionChecker}} which has two
implementations:
** {{DefaultMemoryExhaustionChecker}} preserves the old logic - e.g. uses
{{MapJoinMemoryExhaustionHandler}}
** {{SparkMemoryExhaustionChecker}} uses the new logic you added - e.g.
{{GcTimeMonitor}}
* Depending on the execution engine, {{HashTableSinkOperator}} will use one of
the above classes to check if memory has been exhausted
* I changed {{hive.mapjoin.max.gc.time.percentage}} to be a value between 0 and
1 to make the config more consistent with the rest of Hive configs.
Let me know what you think. These new changes should also fix the test failures
you were seeing.
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-17684.01.patch, HIVE-17684.02.patch,
> HIVE-17684.03.patch, HIVE-17684.04.patch, HIVE-17684.05.patch,
> HIVE-17684.06.patch, HIVE-17684.07.patch, HIVE-17684.08.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect
> scenarios where the small table is taking too much space in memory, in which
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic
> to estimate how much memory the {{HashMap}} is consuming:
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() /
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be
> inaccurate. The value returned by this method returns all reachable and
> unreachable memory on the heap, so there may be a bunch of garbage data, and
> the JVM just hasn't taken the time to reclaim it all. This can lead to
> intermittent failures of this check even though a simple GC would have
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS.
> In Hive-on-MR this probably made sense to use because every Hive task was run
> in a dedicated container, so a Hive Task could assume it created most of the
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks
> running in a single executor, each doing different things.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17684:
Attachment: HIVE-17684.08.patch
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-17684.01.patch, HIVE-17684.02.patch,
> HIVE-17684.03.patch, HIVE-17684.04.patch, HIVE-17684.05.patch,
> HIVE-17684.06.patch, HIVE-17684.07.patch, HIVE-17684.08.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect
> scenarios where the small table is taking too much space in memory, in which
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic
> to estimate how much memory the {{HashMap}} is consuming:
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() /
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be
> inaccurate. The value returned by this method returns all reachable and
> unreachable memory on the heap, so there may be a bunch of garbage data, and
> the JVM just hasn't taken the time to reclaim it all. This can lead to
> intermittent failures of this check even though a simple GC would have
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS.
> In Hive-on-MR this probably made sense to use because every Hive task was run
> in a dedicated container, so a Hive Task could assume it created most of the
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks
> running in a single executor, each doing different things.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-15868) Preauthenticated subject doesn't work for HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta resolved HIVE-15868. - Resolution: Not A Problem Tested recently, it works as designed. There is a bug for http transport though, for which I'll open a new jira > Preauthenticated subject doesn't work for HiveServer2 > - > > Key: HIVE-15868 > URL: https://issues.apache.org/jira/browse/HIVE-15868 > Project: Hive > Issue Type: Bug > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > The feature was added in HIVE-6486, HIVE-8705 but I tested recently and > doesn't seem to work as expected. I'll attach the test case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-20095) Fix jdbc external table feature
[ https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20095: --- Comment: was deleted (was: +1) > Fix jdbc external table feature > --- > > Key: HIVE-20095 > URL: https://issues.apache.org/jira/browse/HIVE-20095 > Project: Hive > Issue Type: Bug >Reporter: Jonathan Doron >Assignee: Jonathan Doron >Priority: Major > Attachments: HIVE-20095.1.patch, HIVE-20095.2.patch, > HIVE-20095.3.patch, HIVE-20095.4.patch, HIVE-20095.5.patch, > HIVE-20095.6.patch, HIVE-20095.7.patch, HIVE-20095.7.patch > > > It seems like the committed code for HIVE-19161 > (7584b3276bebf64aa006eaa162c0a6264d8fcb56) reverted some of HIVE-18423 > updates, and therefore some of the external table queries are not working > correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20095) Fix jdbc external table feature
[
https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614080#comment-16614080
]
Jesus Camacho Rodriguez commented on HIVE-20095:
[~msydoron], I was checking the patch and realized that you moved a method from
Druid to AbstractSerde. That method should not be moved as it is Druid specific
(specially to AbstractSerDe which does not contain related methods). Can you
revert that change and call {{Date.valueOf}} and {{Timestamp.valueOf}} instead?
> Fix jdbc external table feature
> ---
>
> Key: HIVE-20095
> URL: https://issues.apache.org/jira/browse/HIVE-20095
> Project: Hive
> Issue Type: Bug
>Reporter: Jonathan Doron
>Assignee: Jonathan Doron
>Priority: Major
> Attachments: HIVE-20095.1.patch, HIVE-20095.2.patch,
> HIVE-20095.3.patch, HIVE-20095.4.patch, HIVE-20095.5.patch,
> HIVE-20095.6.patch, HIVE-20095.7.patch, HIVE-20095.7.patch
>
>
> It seems like the committed code for HIVE-19161
> (7584b3276bebf64aa006eaa162c0a6264d8fcb56) reverted some of HIVE-18423
> updates, and therefore some of the external table queries are not working
> correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20095) Fix jdbc external table feature
[ https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20095: -- Assignee: Jonathan Doron (was: Jesus Camacho Rodriguez) > Fix jdbc external table feature > --- > > Key: HIVE-20095 > URL: https://issues.apache.org/jira/browse/HIVE-20095 > Project: Hive > Issue Type: Bug >Reporter: Jonathan Doron >Assignee: Jonathan Doron >Priority: Major > Attachments: HIVE-20095.1.patch, HIVE-20095.2.patch, > HIVE-20095.3.patch, HIVE-20095.4.patch, HIVE-20095.5.patch, > HIVE-20095.6.patch, HIVE-20095.7.patch, HIVE-20095.7.patch > > > It seems like the committed code for HIVE-19161 > (7584b3276bebf64aa006eaa162c0a6264d8fcb56) reverted some of HIVE-18423 > updates, and therefore some of the external table queries are not working > correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20095) Fix jdbc external table feature
[ https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20095: --- Attachment: HIVE-20095.7.patch > Fix jdbc external table feature > --- > > Key: HIVE-20095 > URL: https://issues.apache.org/jira/browse/HIVE-20095 > Project: Hive > Issue Type: Bug >Reporter: Jonathan Doron >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20095.1.patch, HIVE-20095.2.patch, > HIVE-20095.3.patch, HIVE-20095.4.patch, HIVE-20095.5.patch, > HIVE-20095.6.patch, HIVE-20095.7.patch, HIVE-20095.7.patch > > > It seems like the committed code for HIVE-19161 > (7584b3276bebf64aa006eaa162c0a6264d8fcb56) reverted some of HIVE-18423 > updates, and therefore some of the external table queries are not working > correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20095) Fix jdbc external table feature
[ https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20095: -- Assignee: Jesus Camacho Rodriguez (was: Jonathan Doron) > Fix jdbc external table feature > --- > > Key: HIVE-20095 > URL: https://issues.apache.org/jira/browse/HIVE-20095 > Project: Hive > Issue Type: Bug >Reporter: Jonathan Doron >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20095.1.patch, HIVE-20095.2.patch, > HIVE-20095.3.patch, HIVE-20095.4.patch, HIVE-20095.5.patch, > HIVE-20095.6.patch, HIVE-20095.7.patch > > > It seems like the committed code for HIVE-19161 > (7584b3276bebf64aa006eaa162c0a6264d8fcb56) reverted some of HIVE-18423 > updates, and therefore some of the external table queries are not working > correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20519) Remove 30m min value for hive.spark.session.timeout
[
https://issues.apache.org/jira/browse/HIVE-20519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614061#comment-16614061
]
Hive QA commented on HIVE-20519:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939492/HIVE-20519.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14945 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSessionSparkSessionTimeout
(batchId=245)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13763/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13763/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13763/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939492 - PreCommit-HIVE-Build
> Remove 30m min value for hive.spark.session.timeout
> ---
>
> Key: HIVE-20519
> URL: https://issues.apache.org/jira/browse/HIVE-20519
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-20519.1.patch
>
>
> In HIVE-14162 we added the config \{{hive.spark.session.timeout}} which
> provided a way to time out Spark sessions that are active for a long period
> of time. The config has a lower bound of 30m which we should remove. It
> should be possible for users to configure this value so the HoS session is
> closed as soon as the query is complete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17040) Join elimination in the presence of FK relationship
[
https://issues.apache.org/jira/browse/HIVE-17040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-17040:
---
Attachment: HIVE-17040.05.patch
> Join elimination in the presence of FK relationship
> ---
>
> Key: HIVE-17040
> URL: https://issues.apache.org/jira/browse/HIVE-17040
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-17040.01.patch, HIVE-17040.02.patch,
> HIVE-17040.04.patch, HIVE-17040.05.patch, HIVE-17040.patch
>
>
> If the PK/UK table is not filtered, we can safely remove the join.
> A simple example:
> {code:sql}
> SELECT c_current_cdemo_sk
> FROM customer, customer_address
> ON c_current_addr_sk = ca_address_sk;
> {code}
> As a Calcite rule, we could implement this rewriting by 1) matching a Project
> on top of a Join operator, 2) checking that only columns from the FK are used
> in the Project, 3) checking that the join condition matches the FK - PK/UK
> relationship, 4) pulling all the predicates from the PK/UK side and checking
> that the input is not filtered, and 5) removing the join, possibly adding a
> IS NOT NULL condition on the join column from the FK side.
> If the PK/UK table is filtered, we should still transform the Join into a
> SemiJoin operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18908) FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Status: Patch Available (was: In Progress) Try again. > FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: FULL OUTER MapJoin Code Changes.docx, > HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, > HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, > HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, > HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, > HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch, > HIVE-18908.0991.patch, HIVE-18908.0992.patch, HIVE-18908.0993.patch, > HIVE-18908.0994.patch, HIVE-18908.0995.patch, HIVE-18908.0996.patch, > HIVE-18908.0997.patch, HIVE-18908.0998.patch, HIVE-18908.0999.patch, > HIVE-18908.09991.patch, HIVE-18908.09992.patch, HIVE-18908.09993.patch, > HIVE-18908.09994.patch, HIVE-18908.09995.patch, HIVE-18908.09996.patch, JOIN > to MAPJOIN Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. > Rough TPC-DS timings run on laptop: > (NOTE: Query 51 has PTF as a bigger serial portion -- Amdahl's law at play) > FULL OUTER MapJoin OFF = MergeJoin > Query 51: > o Vectorization OFF > • FULL OUTER MapJoin OFF: 4:30 minutes > • FULL OUTER MapJoin ON: 4:37 minutes > o Vectorization ON > • FULL OUTER MapJoin OFF: 2:35 minutes > • FULL OUTER MapJoin ON: 1:47 minutes > Query 97: > o Vectorization OFF > • FULL OUTER MapJoin OFF: 2:37 minutes > • FULL OUTER MapJoin ON: 2:42 minutes > o Vectorization ON > • FULL OUTER MapJoin OFF: 1:17 minutes > • FULL OUTER MapJoin ON: 0:06 minutes > FULL OUTER Join 10,000,000 rows against 323,910 small table keys > o Vectorization ON > • FULL OUTER MapJoin OFF: 14:56 minutes > • FULL OUTER MapJoin ON: 1:45 minutes > FULL OUTER Join 10,000,000 rows against 1,000 small table keys > o Vectorization ON > • FULL OUTER MapJoin OFF: 12:37 minutes > • FULL OUTER MapJoin ON: 1:38 minutes > Hopefully, someone will do large scale cluster testing. > [DynamicPartitionedHashJoin] MapJoin should scale dramatically better than > [Sort] MergeJoin reduce-shuffle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Attachment: HIVE-18908.09996.patch > FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: FULL OUTER MapJoin Code Changes.docx, > HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, > HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, > HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, > HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, > HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch, > HIVE-18908.0991.patch, HIVE-18908.0992.patch, HIVE-18908.0993.patch, > HIVE-18908.0994.patch, HIVE-18908.0995.patch, HIVE-18908.0996.patch, > HIVE-18908.0997.patch, HIVE-18908.0998.patch, HIVE-18908.0999.patch, > HIVE-18908.09991.patch, HIVE-18908.09992.patch, HIVE-18908.09993.patch, > HIVE-18908.09994.patch, HIVE-18908.09995.patch, HIVE-18908.09996.patch, JOIN > to MAPJOIN Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. > Rough TPC-DS timings run on laptop: > (NOTE: Query 51 has PTF as a bigger serial portion -- Amdahl's law at play) > FULL OUTER MapJoin OFF = MergeJoin > Query 51: > o Vectorization OFF > • FULL OUTER MapJoin OFF: 4:30 minutes > • FULL OUTER MapJoin ON: 4:37 minutes > o Vectorization ON > • FULL OUTER MapJoin OFF: 2:35 minutes > • FULL OUTER MapJoin ON: 1:47 minutes > Query 97: > o Vectorization OFF > • FULL OUTER MapJoin OFF: 2:37 minutes > • FULL OUTER MapJoin ON: 2:42 minutes > o Vectorization ON > • FULL OUTER MapJoin OFF: 1:17 minutes > • FULL OUTER MapJoin ON: 0:06 minutes > FULL OUTER Join 10,000,000 rows against 323,910 small table keys > o Vectorization ON > • FULL OUTER MapJoin OFF: 14:56 minutes > • FULL OUTER MapJoin ON: 1:45 minutes > FULL OUTER Join 10,000,000 rows against 1,000 small table keys > o Vectorization ON > • FULL OUTER MapJoin OFF: 12:37 minutes > • FULL OUTER MapJoin ON: 1:38 minutes > Hopefully, someone will do large scale cluster testing. > [DynamicPartitionedHashJoin] MapJoin should scale dramatically better than > [Sort] MergeJoin reduce-shuffle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18583) Enable DateRangeRules
[ https://issues.apache.org/jira/browse/HIVE-18583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-18583: Attachment: HIVE-18583.5.patch > Enable DateRangeRules > -- > > Key: HIVE-18583 > URL: https://issues.apache.org/jira/browse/HIVE-18583 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-18583.2.patch, HIVE-18583.3.patch, > HIVE-18583.4.patch, HIVE-18583.5.patch, HIVE-18583.patch > > > Enable DateRangeRules to translate druid filters to date ranges. > Need calcite version to upgrade to 0.16.0 before merging this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20519) Remove 30m min value for hive.spark.session.timeout
[
https://issues.apache.org/jira/browse/HIVE-20519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614035#comment-16614035
]
Hive QA commented on HIVE-20519:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 12 new + 153 unchanged - 0
fixed = 165 total (was 153) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13763/dev-support/hive-personality.sh
|
| git revision | master / 35f86c7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13763/yetus/diff-checkstyle-ql.txt
|
| modules | C: common itests/hive-unit ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13763/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove 30m min value for hive.spark.session.timeout
> ---
>
> Key: HIVE-20519
> URL: https://issues.apache.org/jira/browse/HIVE-20519
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-20519.1.patch
>
>
> In HIVE-14162 we added the config \{{hive.spark.session.timeout}} which
> provided a way to time out Spark sessions that are active for a long period
> of time. The config has a lower bound of 30m which we should remove. It
> should be possible for users to configure this value so the HoS session is
> closed as soon as the query is complete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18908) FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Status: In Progress (was: Patch Available) > FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: FULL OUTER MapJoin Code Changes.docx, > HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, > HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, > HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, > HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, > HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch, > HIVE-18908.0991.patch, HIVE-18908.0992.patch, HIVE-18908.0993.patch, > HIVE-18908.0994.patch, HIVE-18908.0995.patch, HIVE-18908.0996.patch, > HIVE-18908.0997.patch, HIVE-18908.0998.patch, HIVE-18908.0999.patch, > HIVE-18908.09991.patch, HIVE-18908.09992.patch, HIVE-18908.09993.patch, > HIVE-18908.09994.patch, HIVE-18908.09995.patch, JOIN to MAPJOIN > Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. > Rough TPC-DS timings run on laptop: > (NOTE: Query 51 has PTF as a bigger serial portion -- Amdahl's law at play) > FULL OUTER MapJoin OFF = MergeJoin > Query 51: > o Vectorization OFF > • FULL OUTER MapJoin OFF: 4:30 minutes > • FULL OUTER MapJoin ON: 4:37 minutes > o Vectorization ON > • FULL OUTER MapJoin OFF: 2:35 minutes > • FULL OUTER MapJoin ON: 1:47 minutes > Query 97: > o Vectorization OFF > • FULL OUTER MapJoin OFF: 2:37 minutes > • FULL OUTER MapJoin ON: 2:42 minutes > o Vectorization ON > • FULL OUTER MapJoin OFF: 1:17 minutes > • FULL OUTER MapJoin ON: 0:06 minutes > FULL OUTER Join 10,000,000 rows against 323,910 small table keys > o Vectorization ON > • FULL OUTER MapJoin OFF: 14:56 minutes > • FULL OUTER MapJoin ON: 1:45 minutes > FULL OUTER Join 10,000,000 rows against 1,000 small table keys > o Vectorization ON > • FULL OUTER MapJoin OFF: 12:37 minutes > • FULL OUTER MapJoin ON: 1:38 minutes > Hopefully, someone will do large scale cluster testing. > [DynamicPartitionedHashJoin] MapJoin should scale dramatically better than > [Sort] MergeJoin reduce-shuffle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20524) Schema Evolution checking is broken in going from Hive version 2 to version 3 for ALTER TABLE VARCHAR to DECIMAL
[
https://issues.apache.org/jira/browse/HIVE-20524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20524:
Status: Patch Available (was: In Progress)
> Schema Evolution checking is broken in going from Hive version 2 to version 3
> for ALTER TABLE VARCHAR to DECIMAL
>
>
> Key: HIVE-20524
> URL: https://issues.apache.org/jira/browse/HIVE-20524
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20524.01.patch, HIVE-20524.02.patch,
> HIVE-20524.03.patch
>
>
> Issue that started this JIRA:
> {code}
> create external table varchar_decimal (c1 varchar(25));
> alter table varchar_decimal change c1 c1 decimal(31,0);
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following
> columns have types incompatible with the existing columns in their respective
> positions :
> c1
> {code}
> There appear to be 2 issues here:
> 1) When hive.metastore.disallow.incompatible.col.type.changes is true (the
> default) we only allow StringFamily (STRING, CHAR, VARCHAR) conversion to a
> number that can hold the largest numbers. The theory being we don't want
> data loss you would get by converting the StringFamily field into integers,
> etc. In Hive version 2 the hierarchy of numbers had DECIMAL at the top. At
> some point during Hive version 2 we realized this was incorrect and put
> DOUBLE the top.
> However, the Hive2 Hive version 2 TypeInfoUtils.implicitConversion method
> allows StringFamily to either DOUBLE or DECIMAL conversion.
> The new org.apache.hadoop.hive.metastore.ColumnType class under Hive version
> 3 hive-standalone-metadata-server method checkColTypeChangeCompatible only
> allows DOUBLE.
> This JIRA fixes that problem.
> 2) Also, the checkColTypeChangeCompatible method lost a version 2 series bug
> fix that drops CHAR/VARCHAR (and DECIMAL I think) type decorations when
> checking for Schema Evolution compatibility. So, when that code is checking
> if a data type "varchar(25)" is StringFamily it fails because the "(25)"
> didn't get removed properly.
> This JIRA fixes issue #2 also.
> NOTE: Hive1 version 2 did undecoratedTypeName(oldType) and Hive2 version
> performed the logic in TypeInfoUtils.implicitConvertible on the
> PrimitiveCategory not the raw type string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20524) Schema Evolution checking is broken in going from Hive version 2 to version 3 for ALTER TABLE VARCHAR to DECIMAL
[
https://issues.apache.org/jira/browse/HIVE-20524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20524:
Attachment: HIVE-20524.03.patch
> Schema Evolution checking is broken in going from Hive version 2 to version 3
> for ALTER TABLE VARCHAR to DECIMAL
>
>
> Key: HIVE-20524
> URL: https://issues.apache.org/jira/browse/HIVE-20524
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20524.01.patch, HIVE-20524.02.patch,
> HIVE-20524.03.patch
>
>
> Issue that started this JIRA:
> {code}
> create external table varchar_decimal (c1 varchar(25));
> alter table varchar_decimal change c1 c1 decimal(31,0);
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following
> columns have types incompatible with the existing columns in their respective
> positions :
> c1
> {code}
> There appear to be 2 issues here:
> 1) When hive.metastore.disallow.incompatible.col.type.changes is true (the
> default) we only allow StringFamily (STRING, CHAR, VARCHAR) conversion to a
> number that can hold the largest numbers. The theory being we don't want
> data loss you would get by converting the StringFamily field into integers,
> etc. In Hive version 2 the hierarchy of numbers had DECIMAL at the top. At
> some point during Hive version 2 we realized this was incorrect and put
> DOUBLE the top.
> However, the Hive2 Hive version 2 TypeInfoUtils.implicitConversion method
> allows StringFamily to either DOUBLE or DECIMAL conversion.
> The new org.apache.hadoop.hive.metastore.ColumnType class under Hive version
> 3 hive-standalone-metadata-server method checkColTypeChangeCompatible only
> allows DOUBLE.
> This JIRA fixes that problem.
> 2) Also, the checkColTypeChangeCompatible method lost a version 2 series bug
> fix that drops CHAR/VARCHAR (and DECIMAL I think) type decorations when
> checking for Schema Evolution compatibility. So, when that code is checking
> if a data type "varchar(25)" is StringFamily it fails because the "(25)"
> didn't get removed properly.
> This JIRA fixes issue #2 also.
> NOTE: Hive1 version 2 did undecoratedTypeName(oldType) and Hive2 version
> performed the logic in TypeInfoUtils.implicitConvertible on the
> PrimitiveCategory not the raw type string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20524) Schema Evolution checking is broken in going from Hive version 2 to version 3 for ALTER TABLE VARCHAR to DECIMAL
[
https://issues.apache.org/jira/browse/HIVE-20524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20524:
Status: In Progress (was: Patch Available)
> Schema Evolution checking is broken in going from Hive version 2 to version 3
> for ALTER TABLE VARCHAR to DECIMAL
>
>
> Key: HIVE-20524
> URL: https://issues.apache.org/jira/browse/HIVE-20524
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20524.01.patch, HIVE-20524.02.patch
>
>
> Issue that started this JIRA:
> {code}
> create external table varchar_decimal (c1 varchar(25));
> alter table varchar_decimal change c1 c1 decimal(31,0);
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following
> columns have types incompatible with the existing columns in their respective
> positions :
> c1
> {code}
> There appear to be 2 issues here:
> 1) When hive.metastore.disallow.incompatible.col.type.changes is true (the
> default) we only allow StringFamily (STRING, CHAR, VARCHAR) conversion to a
> number that can hold the largest numbers. The theory being we don't want
> data loss you would get by converting the StringFamily field into integers,
> etc. In Hive version 2 the hierarchy of numbers had DECIMAL at the top. At
> some point during Hive version 2 we realized this was incorrect and put
> DOUBLE the top.
> However, the Hive2 Hive version 2 TypeInfoUtils.implicitConversion method
> allows StringFamily to either DOUBLE or DECIMAL conversion.
> The new org.apache.hadoop.hive.metastore.ColumnType class under Hive version
> 3 hive-standalone-metadata-server method checkColTypeChangeCompatible only
> allows DOUBLE.
> This JIRA fixes that problem.
> 2) Also, the checkColTypeChangeCompatible method lost a version 2 series bug
> fix that drops CHAR/VARCHAR (and DECIMAL I think) type decorations when
> checking for Schema Evolution compatibility. So, when that code is checking
> if a data type "varchar(25)" is StringFamily it fails because the "(25)"
> didn't get removed properly.
> This JIRA fixes issue #2 also.
> NOTE: Hive1 version 2 did undecoratedTypeName(oldType) and Hive2 version
> performed the logic in TypeInfoUtils.implicitConvertible on the
> PrimitiveCategory not the raw type string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20524) Schema Evolution checking is broken in going from Hive version 2 to version 3 for ALTER TABLE VARCHAR to DECIMAL
[
https://issues.apache.org/jira/browse/HIVE-20524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614017#comment-16614017
]
Matt McCline commented on HIVE-20524:
-
Neither test fails on my laptop. Seems like they might be flakey.
> Schema Evolution checking is broken in going from Hive version 2 to version 3
> for ALTER TABLE VARCHAR to DECIMAL
>
>
> Key: HIVE-20524
> URL: https://issues.apache.org/jira/browse/HIVE-20524
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20524.01.patch, HIVE-20524.02.patch
>
>
> Issue that started this JIRA:
> {code}
> create external table varchar_decimal (c1 varchar(25));
> alter table varchar_decimal change c1 c1 decimal(31,0);
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following
> columns have types incompatible with the existing columns in their respective
> positions :
> c1
> {code}
> There appear to be 2 issues here:
> 1) When hive.metastore.disallow.incompatible.col.type.changes is true (the
> default) we only allow StringFamily (STRING, CHAR, VARCHAR) conversion to a
> number that can hold the largest numbers. The theory being we don't want
> data loss you would get by converting the StringFamily field into integers,
> etc. In Hive version 2 the hierarchy of numbers had DECIMAL at the top. At
> some point during Hive version 2 we realized this was incorrect and put
> DOUBLE the top.
> However, the Hive2 Hive version 2 TypeInfoUtils.implicitConversion method
> allows StringFamily to either DOUBLE or DECIMAL conversion.
> The new org.apache.hadoop.hive.metastore.ColumnType class under Hive version
> 3 hive-standalone-metadata-server method checkColTypeChangeCompatible only
> allows DOUBLE.
> This JIRA fixes that problem.
> 2) Also, the checkColTypeChangeCompatible method lost a version 2 series bug
> fix that drops CHAR/VARCHAR (and DECIMAL I think) type decorations when
> checking for Schema Evolution compatibility. So, when that code is checking
> if a data type "varchar(25)" is StringFamily it fails because the "(25)"
> didn't get removed properly.
> This JIRA fixes issue #2 also.
> NOTE: Hive1 version 2 did undecoratedTypeName(oldType) and Hive2 version
> performed the logic in TypeInfoUtils.implicitConvertible on the
> PrimitiveCategory not the raw type string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20535) Add new configuration to set the size of the global compile lock
[ https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] denys kuzmenko updated HIVE-20535: -- Status: Patch Available (was: Open) > Add new configuration to set the size of the global compile lock > > > Key: HIVE-20535 > URL: https://issues.apache.org/jira/browse/HIVE-20535 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20535.1.patch > > > When removing the compile lock, it is quite risky to remove it entirely. > It would be good to provide a pool size for the concurrent compilation, so > the administrator can limit the load -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20547) HS2: support Tez sessions started by someone else (part 1)
[ https://issues.apache.org/jira/browse/HIVE-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614013#comment-16614013 ] Prasanth Jayachandran commented on HIVE-20547: -- +1, pending tests > HS2: support Tez sessions started by someone else (part 1) > -- > > Key: HIVE-20547 > URL: https://issues.apache.org/jira/browse/HIVE-20547 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20547.01.patch, HIVE-20547.patch > > > The registry/configs/some code is based on a private patch by [~prasanth_j]. > The patch refactors tez pool session to use composition instead of > implementation inheritance from TezSessionState, to allow for two > implementations of TezSessionState. > For now it's blocked on getClient API in Tez that will be available after > 0.9.3 release; however I commented out that path to check that refactoring > passes tests. > When 0.9.3 becomes available, we can uncomment and commit. > In part 2, we may add some tests, and also consider other changes that are > required for external sessions (e.g. KillQuery, where we cannot assume YARN > is present). > We may also consider a WM change that allows for proportional session > distribution when the number of external sessions and the number of > admin-specified sessions doesn't match, or at least some validation to see > that the external sessions are available when applying a RP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20535) Add new configuration to set the size of the global compile lock
[ https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] denys kuzmenko updated HIVE-20535: -- Attachment: HIVE-20535.1.patch > Add new configuration to set the size of the global compile lock > > > Key: HIVE-20535 > URL: https://issues.apache.org/jira/browse/HIVE-20535 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20535.1.patch > > > When removing the compile lock, it is quite risky to remove it entirely. > It would be good to provide a pool size for the concurrent compilation, so > the administrator can limit the load -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20554) Unable to drop an external table after renaming it.
[
https://issues.apache.org/jira/browse/HIVE-20554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krishnama Raju K updated HIVE-20554:
Description:
Unable to drop an external partitioned table after renaming it. Getting the
following exception,
{noformat}
java.sql.BatchUpdateException: Cannot delete or update a parent row: a foreign
key constraint fails ("hive"."PART_COL_STATS", CONSTRAINT "PART_COL_STATS_FK"
FOREIGN KEY ("PART_ID") REFERENCES "PARTITIONS" ("PART_ID"))
at
com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2024)
at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1449)
at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:424)
at
org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:366)
at
org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:676)
at
org.datanucleus.store.rdbms.SQLController.getStatementForUpdate(SQLController.java:204)
at
org.datanucleus.store.rdbms.SQLController.getStatementForUpdate(SQLController.java:176)
at
org.datanucleus.store.rdbms.scostore.JoinMapStore.clearInternal(JoinMapStore.java:900)
at
org.datanucleus.store.rdbms.scostore.JoinMapStore.clear(JoinMapStore.java:449)
at org.datanucleus.store.types.wrappers.backed.Map.clear(Map.java:605)
at
org.datanucleus.store.rdbms.mapping.java.MapMapping.preDelete(MapMapping.java:252)
at
org.datanucleus.store.rdbms.request.DeleteRequest.execute(DeleteRequest.java:193)
at
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.deleteObjectFromTable(RDBMSPersistenceHandler.java:499)
at
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.deleteObject(RDBMSPersistenceHandler.java:470)
at
org.datanucleus.state.AbstractStateManager.internalDeletePersistent(AbstractStateManager.java:832)
at
org.datanucleus.state.StateManagerImpl.deletePersistent(StateManagerImpl.java:4244)
at
org.datanucleus.ExecutionContextImpl.deleteObjectInternal(ExecutionContextImpl.java:2395)
at
org.datanucleus.ExecutionContextImpl.deleteObjectWork(ExecutionContextImpl.java:2317)
at
org.datanucleus.ExecutionContextImpl.deleteObjects(ExecutionContextImpl.java:2209)
at
org.datanucleus.ExecutionContextThreadedImpl.deleteObjects(ExecutionContextThreadedImpl.java:259)
at
org.datanucleus.store.query.Query.performDeletePersistentAll(Query.java:2133)
at
org.datanucleus.store.query.AbstractJavaQuery.performDeletePersistentAll(AbstractJavaQuery.java:114)
at org.datanucleus.store.query.Query.deletePersistentAll(Query.java:2085)
at org.datanucleus.api.jdo.JDOQuery.deletePersistentInternal(JDOQuery.java:441)
at org.datanucleus.api.jdo.JDOQuery.deletePersistentAll(JDOQuery.java:428)
at
org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsNoTxn(ObjectStore.java:2421)
at
org.apache.hadoop.hive.metastore.ObjectStore.dropPartitions(ObjectStore.java:1805)
at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
at com.sun.proxy.$Proxy10.dropPartitions(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.dropPartitionsAndGetLocations(HiveMetaStore.java:1838)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1673)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1859)
at sun.reflect.GeneratedMethodAccessor110.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at com.sun.proxy.$Proxy12.drop_table_with_environment_context(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:9523)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:9507)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:551)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:546)
at java.security.AccessController.doPrivileged(Native Method)
at
[jira] [Updated] (HIVE-20535) Add new configuration to set the size of the global compile lock
[ https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] denys kuzmenko updated HIVE-20535: -- Attachment: (was: HIVE-20535.1.patch) > Add new configuration to set the size of the global compile lock > > > Key: HIVE-20535 > URL: https://issues.apache.org/jira/browse/HIVE-20535 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > > When removing the compile lock, it is quite risky to remove it entirely. > It would be good to provide a pool size for the concurrent compilation, so > the administrator can limit the load -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20547) HS2: support Tez sessions started by someone else (part 1)
[ https://issues.apache.org/jira/browse/HIVE-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20547: Attachment: HIVE-20547.01.patch > HS2: support Tez sessions started by someone else (part 1) > -- > > Key: HIVE-20547 > URL: https://issues.apache.org/jira/browse/HIVE-20547 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20547.01.patch, HIVE-20547.patch > > > The registry/configs/some code is based on a private patch by [~prasanth_j]. > The patch refactors tez pool session to use composition instead of > implementation inheritance from TezSessionState, to allow for two > implementations of TezSessionState. > For now it's blocked on getClient API in Tez that will be available after > 0.9.3 release; however I commented out that path to check that refactoring > passes tests. > When 0.9.3 becomes available, we can uncomment and commit. > In part 2, we may add some tests, and also consider other changes that are > required for external sessions (e.g. KillQuery, where we cannot assume YARN > is present). > We may also consider a WM change that allows for proportional session > distribution when the number of external sessions and the number of > admin-specified sessions doesn't match, or at least some validation to see > that the external sessions are available when applying a RP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20535) Add new configuration to set the size of the global compile lock
[ https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] denys kuzmenko updated HIVE-20535: -- Attachment: (was: HIVE-20535.2.patch) > Add new configuration to set the size of the global compile lock > > > Key: HIVE-20535 > URL: https://issues.apache.org/jira/browse/HIVE-20535 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20535.1.patch > > > When removing the compile lock, it is quite risky to remove it entirely. > It would be good to provide a pool size for the concurrent compilation, so > the administrator can limit the load -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20095) Fix jdbc external table feature
[
https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613992#comment-16613992
]
Hive QA commented on HIVE-20095:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939476/HIVE-20095.7.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14940 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_basic]
(batchId=264)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13762/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13762/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13762/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939476 - PreCommit-HIVE-Build
> Fix jdbc external table feature
> ---
>
> Key: HIVE-20095
> URL: https://issues.apache.org/jira/browse/HIVE-20095
> Project: Hive
> Issue Type: Bug
>Reporter: Jonathan Doron
>Assignee: Jonathan Doron
>Priority: Major
> Attachments: HIVE-20095.1.patch, HIVE-20095.2.patch,
> HIVE-20095.3.patch, HIVE-20095.4.patch, HIVE-20095.5.patch,
> HIVE-20095.6.patch, HIVE-20095.7.patch
>
>
> It seems like the committed code for HIVE-19161
> (7584b3276bebf64aa006eaa162c0a6264d8fcb56) reverted some of HIVE-18423
> updates, and therefore some of the external table queries are not working
> correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20483) Really move metastore common classes into metastore-common
[ https://issues.apache.org/jira/browse/HIVE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-20483: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Patch 6 committed. > Really move metastore common classes into metastore-common > -- > > Key: HIVE-20483 > URL: https://issues.apache.org/jira/browse/HIVE-20483 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 3.0.1, 4.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20483.01.patch, HIVE-20483.02.patch, > HIVE-20483.03.patch, HIVE-20483.04.patch, HIVE-20483.05.patch, > HIVE-20483.06.patch > > > HIVE-20388 patch was supposed to move a bunch of files from metastore-server > to metastore-common but for some reason it didn't happen, so now these files > should be moved. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20553) more acid stats tests
[ https://issues.apache.org/jira/browse/HIVE-20553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20553: -- Attachment: HIVE-20553.02.patch > more acid stats tests > - > > Key: HIVE-20553 > URL: https://issues.apache.org/jira/browse/HIVE-20553 > Project: Hive > Issue Type: Improvement > Components: Statistics, Transactions >Affects Versions: 4.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-20553.01.patch, HIVE-20553.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20095) Fix jdbc external table feature
[
https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613935#comment-16613935
]
Hive QA commented on HIVE-20095:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} serde in master has 195 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} druid-handler in master has 13 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} jdbc-handler in master has 8 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} druid-handler in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
13s{color} | {color:red} jdbc-handler in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch serde passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} druid-handler: The patch generated 0 new + 10
unchanged - 1 fixed = 10 total (was 11) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} jdbc-handler: The patch generated 2 new + 24 unchanged
- 1 fixed = 26 total (was 25) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
33s{color} | {color:red} jdbc-handler generated 3 new + 8 unchanged - 0 fixed =
11 total (was 8) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:jdbc-handler |
| | Exception is caught when Exception is not thrown in
org.apache.hive.storage.jdbc.JdbcSerDe.initialize(Configuration, Properties)
At JdbcSerDe.java:is not thrown in
org.apache.hive.storage.jdbc.JdbcSerDe.initialize(Configuration, Properties)
At JdbcSerDe.java:[line 114] |
| |
org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnTypes(Configuration)
may fail to clean up java.sql.ResultSet Obligation to clean up resource
created at GenericJdbcDatabaseAccessor.java:up java.sql.ResultSet Obligation
to clean up resource created at GenericJdbcDatabaseAccessor.java:[line 115] is
not discharged |
| |
[jira] [Assigned] (HIVE-20553) more acid stats tests
[ https://issues.apache.org/jira/browse/HIVE-20553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-20553: - > more acid stats tests > - > > Key: HIVE-20553 > URL: https://issues.apache.org/jira/browse/HIVE-20553 > Project: Hive > Issue Type: Improvement > Components: Statistics, Transactions >Affects Versions: 4.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-20553.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20553) more acid stats tests
[ https://issues.apache.org/jira/browse/HIVE-20553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20553: -- Attachment: HIVE-20553.01.patch > more acid stats tests > - > > Key: HIVE-20553 > URL: https://issues.apache.org/jira/browse/HIVE-20553 > Project: Hive > Issue Type: Improvement > Components: Statistics, Transactions >Affects Versions: 4.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-20553.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20547) HS2: support Tez sessions started by someone else (part 1)
[ https://issues.apache.org/jira/browse/HIVE-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20547: Status: Patch Available (was: Open) > HS2: support Tez sessions started by someone else (part 1) > -- > > Key: HIVE-20547 > URL: https://issues.apache.org/jira/browse/HIVE-20547 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20547.patch > > > The registry/configs/some code is based on a private patch by [~prasanth_j]. > The patch refactors tez pool session to use composition instead of > implementation inheritance from TezSessionState, to allow for two > implementations of TezSessionState. > For now it's blocked on getClient API in Tez that will be available after > 0.9.3 release; however I commented out that path to check that refactoring > passes tests. > When 0.9.3 becomes available, we can uncomment and commit. > In part 2, we may add some tests, and also consider other changes that are > required for external sessions (e.g. KillQuery, where we cannot assume YARN > is present). > We may also consider a WM change that allows for proportional session > distribution when the number of external sessions and the number of > admin-specified sessions doesn't match, or at least some validation to see > that the external sessions are available when applying a RP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20553) more acid stats tests
[ https://issues.apache.org/jira/browse/HIVE-20553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20553: -- Status: Patch Available (was: Open) > more acid stats tests > - > > Key: HIVE-20553 > URL: https://issues.apache.org/jira/browse/HIVE-20553 > Project: Hive > Issue Type: Improvement > Components: Statistics, Transactions >Affects Versions: 4.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-20553.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18583) Enable DateRangeRules
[ https://issues.apache.org/jira/browse/HIVE-18583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-18583: Attachment: (was: HIVE-18583.5.patch) > Enable DateRangeRules > -- > > Key: HIVE-18583 > URL: https://issues.apache.org/jira/browse/HIVE-18583 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-18583.2.patch, HIVE-18583.3.patch, > HIVE-18583.4.patch, HIVE-18583.patch > > > Enable DateRangeRules to translate druid filters to date ranges. > Need calcite version to upgrade to 0.16.0 before merging this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20524) Schema Evolution checking is broken in going from Hive version 2 to version 3 for ALTER TABLE VARCHAR to DECIMAL
[
https://issues.apache.org/jira/browse/HIVE-20524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613874#comment-16613874
]
Hive QA commented on HIVE-20524:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939475/HIVE-20524.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14940 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=107)
org.apache.hive.service.auth.TestCustomAuthentication.org.apache.hive.service.auth.TestCustomAuthentication
(batchId=247)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13761/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13761/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13761/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939475 - PreCommit-HIVE-Build
> Schema Evolution checking is broken in going from Hive version 2 to version 3
> for ALTER TABLE VARCHAR to DECIMAL
>
>
> Key: HIVE-20524
> URL: https://issues.apache.org/jira/browse/HIVE-20524
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20524.01.patch, HIVE-20524.02.patch
>
>
> Issue that started this JIRA:
> {code}
> create external table varchar_decimal (c1 varchar(25));
> alter table varchar_decimal change c1 c1 decimal(31,0);
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following
> columns have types incompatible with the existing columns in their respective
> positions :
> c1
> {code}
> There appear to be 2 issues here:
> 1) When hive.metastore.disallow.incompatible.col.type.changes is true (the
> default) we only allow StringFamily (STRING, CHAR, VARCHAR) conversion to a
> number that can hold the largest numbers. The theory being we don't want
> data loss you would get by converting the StringFamily field into integers,
> etc. In Hive version 2 the hierarchy of numbers had DECIMAL at the top. At
> some point during Hive version 2 we realized this was incorrect and put
> DOUBLE the top.
> However, the Hive2 Hive version 2 TypeInfoUtils.implicitConversion method
> allows StringFamily to either DOUBLE or DECIMAL conversion.
> The new org.apache.hadoop.hive.metastore.ColumnType class under Hive version
> 3 hive-standalone-metadata-server method checkColTypeChangeCompatible only
> allows DOUBLE.
> This JIRA fixes that problem.
> 2) Also, the checkColTypeChangeCompatible method lost a version 2 series bug
> fix that drops CHAR/VARCHAR (and DECIMAL I think) type decorations when
> checking for Schema Evolution compatibility. So, when that code is checking
> if a data type "varchar(25)" is StringFamily it fails because the "(25)"
> didn't get removed properly.
> This JIRA fixes issue #2 also.
> NOTE: Hive1 version 2 did undecoratedTypeName(oldType) and Hive2 version
> performed the logic in TypeInfoUtils.implicitConvertible on the
> PrimitiveCategory not the raw type string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20526) Add test case for HIVE-20489
[ https://issues.apache.org/jira/browse/HIVE-20526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-20526: --- Attachment: HIVE-20526.1.patch > Add test case for HIVE-20489 > > > Key: HIVE-20526 > URL: https://issues.apache.org/jira/browse/HIVE-20526 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20526.1.patch, HIVE-20526.1.patch > > > Add a test case for the issue discussed in HIVE-20489. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20524) Schema Evolution checking is broken in going from Hive version 2 to version 3 for ALTER TABLE VARCHAR to DECIMAL
[
https://issues.apache.org/jira/browse/HIVE-20524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613844#comment-16613844
]
Hive QA commented on HIVE-20524:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 5m
54s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13761/dev-support/hive-personality.sh
|
| git revision | master / 265 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13761/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13761/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: itests ql standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13761/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Schema Evolution checking is broken in going from Hive version 2 to version 3
> for ALTER TABLE VARCHAR to DECIMAL
>
>
> Key: HIVE-20524
> URL: https://issues.apache.org/jira/browse/HIVE-20524
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20524.01.patch, HIVE-20524.02.patch
>
>
> Issue that started this JIRA:
> {code}
> create external table varchar_decimal (c1 varchar(25));
> alter table varchar_decimal change c1 c1 decimal(31,0);
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following
> columns have types incompatible with the existing columns in their
[jira] [Commented] (HIVE-18908) FULL OUTER JOIN to MapJoin
[
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613841#comment-16613841
]
Hive QA commented on HIVE-18908:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} serde in master has 195 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 13m
33s{color} | {color:red} branch/itests/hive-jmh cannot run convertXmlToText
from findbugs {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} The patch storage-api passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
31s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} The patch serde passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} The patch . passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} itests/hive-jmh: The patch generated 0 new + 11
unchanged - 6 fixed = 11 total (was 17) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
51s{color} | {color:red} ql: The patch generated 312 new + 3561 unchanged - 174
fixed = 3873 total (was 3735) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
29s{color} | {color:red} patch/storage-api cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
56s{color} | {color:red} patch/serde cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
15s{color} | {color:red} patch/itests/hive-jmh cannot run setBugDatabaseInfo
from findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 6m
46s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 10m
59s{color} | {color:red} root generated 2 new + 386 unchanged - 0 fixed = 388
total (was 386) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} |
[jira] [Resolved] (HIVE-19190) Improve Logging for SemanticException Handling
[
https://issues.apache.org/jira/browse/HIVE-19190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan resolved HIVE-19190.
--
Resolution: Duplicate
> Improve Logging for SemanticException Handling
> --
>
> Key: HIVE-19190
> URL: https://issues.apache.org/jira/browse/HIVE-19190
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Major
> Attachments: table_not_found_example.txt
>
>
> Please improve the logging for queries that fail with SemanticException. For
> example, when performing an action on a table that does not exist. The most
> common reason why this happens is that someone fat fingers the table/column
> name. It is not a system error pe se, but a user validation error. This is
> not something the cluster administrator should have to worry about. Yet,
> Hive performs some pretty extreme logging on the matter. I have attached to
> this JIRA the logging produced by a single submission of the following query:
> {code:sql}
> select * from madeup;
> {code}
>
> For SemanticException exceptions, please print the Exception {{getMessage()}}
> to the server INFO logging so that the query's life-cycle can be traced, but
> do not blast ERRORs and stack traces to the log file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19302) Logging Too Verbose For TableNotFound
[ https://issues.apache.org/jira/browse/HIVE-19302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan reassigned HIVE-19302: Assignee: Alice Fan > Logging Too Verbose For TableNotFound > - > > Key: HIVE-19302 > URL: https://issues.apache.org/jira/browse/HIVE-19302 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 2.2.0, 3.0.0 >Reporter: BELUGA BEHR >Assignee: Alice Fan >Priority: Minor > Attachments: table_not_found_cdh6.txt > > > There is way too much logging when a user submits a query against a table > which does not exist. In an ad-hoc setting, it is quite normal that a user > fat-fingers a table name. Yet, from the perspective of the Hive > administrator, there was perhaps a major issue based on the volume and > severity of logging. Please change the logging to INFO level, and do not > present a stack trace, for such a trivial error. > > See the attached file for a sample of what logging a single "table not found" > query generates. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613811#comment-16613811 ] Sahil Takiar commented on HIVE-20440: - Makes sense to me. > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19847) Create Separate getInputSummary Service
[
https://issues.apache.org/jira/browse/HIVE-19847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-19847:
---
Release Note: (was: Fix FindBugs and CheckStyle Errors)
Status: Patch Available (was: Open)
Trying this again... patch didn't build but I also didn't change anything.
> Create Separate getInputSummary Service
> ---
>
> Key: HIVE-19847
> URL: https://issues.apache.org/jira/browse/HIVE-19847
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-19847.1.patch, HIVE-19847.2.patch,
> HIVE-19847.3.patch, HIVE-19847.4.patch, HIVE-19847.5.patch, HIVE-19847.6.patch
>
>
> The Hive {{org.apache.hadoop.hive.ql.exec.Utilities.java}} file has taken on
> a life of its own. We should consider separating out the various components
> into their own classes. For this ticket, I propose separating out the
> {{getInputSummary}} functionality into its own class.
> There are several issues with the current implementation:
> # It is
> [synchronized|https://github.com/apache/hive/blob/f27c38ff55902827499192a4f8cf8ed37d6fd967/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2383].
> Only one query can get file input summary at a time. For a query which
> deals with a large data set with a large number of files, this can block
> other queries for a long period of time. This is especially painful when
> most queries use a small data set, but a large data set is submitted on
> occasion.
> # For each query, time is spend setting up and tearing down a ThreadPool
> # It uses deprecated code
> I propose breaking it out into its own class and creating a single thread
> pool that all queries pull from. In this way, the bottle neck will be one
> the number of available threads, not on a single query and if a big query is
> running and a small query is also submitted, the smaller query will be able
> to proceed.
> In regards to setup/teardown... if a query uses 15 threads to perform this
> summary action, then finishes, it will tear down the threads, the next query
> may immediate create 15 new threads for processing. With a single pool,
> those threads are never performing tear down and setup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19847) Create Separate getInputSummary Service
[
https://issues.apache.org/jira/browse/HIVE-19847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-19847:
---
Status: Open (was: Patch Available)
> Create Separate getInputSummary Service
> ---
>
> Key: HIVE-19847
> URL: https://issues.apache.org/jira/browse/HIVE-19847
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-19847.1.patch, HIVE-19847.2.patch,
> HIVE-19847.3.patch, HIVE-19847.4.patch, HIVE-19847.5.patch, HIVE-19847.6.patch
>
>
> The Hive {{org.apache.hadoop.hive.ql.exec.Utilities.java}} file has taken on
> a life of its own. We should consider separating out the various components
> into their own classes. For this ticket, I propose separating out the
> {{getInputSummary}} functionality into its own class.
> There are several issues with the current implementation:
> # It is
> [synchronized|https://github.com/apache/hive/blob/f27c38ff55902827499192a4f8cf8ed37d6fd967/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2383].
> Only one query can get file input summary at a time. For a query which
> deals with a large data set with a large number of files, this can block
> other queries for a long period of time. This is especially painful when
> most queries use a small data set, but a large data set is submitted on
> occasion.
> # For each query, time is spend setting up and tearing down a ThreadPool
> # It uses deprecated code
> I propose breaking it out into its own class and creating a single thread
> pool that all queries pull from. In this way, the bottle neck will be one
> the number of available threads, not on a single query and if a big query is
> running and a small query is also submitted, the smaller query will be able
> to proceed.
> In regards to setup/teardown... if a query uses 15 threads to perform this
> summary action, then finishes, it will tear down the threads, the next query
> may immediate create 15 new threads for processing. With a single pool,
> those threads are never performing tear down and setup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19847) Create Separate getInputSummary Service
[
https://issues.apache.org/jira/browse/HIVE-19847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-19847:
---
Attachment: HIVE-19847.6.patch
> Create Separate getInputSummary Service
> ---
>
> Key: HIVE-19847
> URL: https://issues.apache.org/jira/browse/HIVE-19847
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-19847.1.patch, HIVE-19847.2.patch,
> HIVE-19847.3.patch, HIVE-19847.4.patch, HIVE-19847.5.patch, HIVE-19847.6.patch
>
>
> The Hive {{org.apache.hadoop.hive.ql.exec.Utilities.java}} file has taken on
> a life of its own. We should consider separating out the various components
> into their own classes. For this ticket, I propose separating out the
> {{getInputSummary}} functionality into its own class.
> There are several issues with the current implementation:
> # It is
> [synchronized|https://github.com/apache/hive/blob/f27c38ff55902827499192a4f8cf8ed37d6fd967/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2383].
> Only one query can get file input summary at a time. For a query which
> deals with a large data set with a large number of files, this can block
> other queries for a long period of time. This is especially painful when
> most queries use a small data set, but a large data set is submitted on
> occasion.
> # For each query, time is spend setting up and tearing down a ThreadPool
> # It uses deprecated code
> I propose breaking it out into its own class and creating a single thread
> pool that all queries pull from. In this way, the bottle neck will be one
> the number of available threads, not on a single query and if a big query is
> running and a small query is also submitted, the smaller query will be able
> to proceed.
> In regards to setup/teardown... if a query uses 15 threads to perform this
> summary action, then finishes, it will tear down the threads, the next query
> may immediate create 15 new threads for processing. With a single pool,
> those threads are never performing tear down and setup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18908) FULL OUTER JOIN to MapJoin
[
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613797#comment-16613797
]
Hive QA commented on HIVE-18908:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939473/HIVE-18908.09995.patch
{color:green}SUCCESS:{color} +1 due to 64 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14964 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
(batchId=237)
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
(batchId=237)
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMethodCounts
(batchId=237)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13760/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13760/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13760/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939473 - PreCommit-HIVE-Build
> FULL OUTER JOIN to MapJoin
> --
>
> Key: HIVE-18908
> URL: https://issues.apache.org/jira/browse/HIVE-18908
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: FULL OUTER MapJoin Code Changes.docx,
> HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch,
> HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch,
> HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch,
> HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch,
> HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch,
> HIVE-18908.0991.patch, HIVE-18908.0992.patch, HIVE-18908.0993.patch,
> HIVE-18908.0994.patch, HIVE-18908.0995.patch, HIVE-18908.0996.patch,
> HIVE-18908.0997.patch, HIVE-18908.0998.patch, HIVE-18908.0999.patch,
> HIVE-18908.09991.patch, HIVE-18908.09992.patch, HIVE-18908.09993.patch,
> HIVE-18908.09994.patch, HIVE-18908.09995.patch, JOIN to MAPJOIN
> Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf
>
>
> Currently, we do not support FULL OUTER JOIN in MapJoin.
> Rough TPC-DS timings run on laptop:
> (NOTE: Query 51 has PTF as a bigger serial portion -- Amdahl's law at play)
> FULL OUTER MapJoin OFF = MergeJoin
> Query 51:
> o Vectorization OFF
> • FULL OUTER MapJoin OFF: 4:30 minutes
> • FULL OUTER MapJoin ON: 4:37 minutes
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 2:35 minutes
> • FULL OUTER MapJoin ON: 1:47 minutes
> Query 97:
> o Vectorization OFF
> • FULL OUTER MapJoin OFF: 2:37 minutes
> • FULL OUTER MapJoin ON: 2:42 minutes
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 1:17 minutes
> • FULL OUTER MapJoin ON: 0:06 minutes
> FULL OUTER Join 10,000,000 rows against 323,910 small table keys
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 14:56 minutes
> • FULL OUTER MapJoin ON: 1:45 minutes
> FULL OUTER Join 10,000,000 rows against 1,000 small table keys
> o Vectorization ON
> • FULL OUTER MapJoin OFF: 12:37 minutes
> • FULL OUTER MapJoin ON: 1:38 minutes
> Hopefully, someone will do large scale cluster testing.
> [DynamicPartitionedHashJoin] MapJoin should scale dramatically better than
> [Sort] MergeJoin reduce-shuffle.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20548) Can not start llp via yarn service
[
https://issues.apache.org/jira/browse/HIVE-20548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613760#comment-16613760
]
Dharmik Thakkar commented on HIVE-20548:
[~zhangbutao] You can follow the article on
[llap-sizing-and-setup|https://community.hortonworks.com/articles/149486/llap-sizing-and-setup.html].
> Can not start llp via yarn service
> --
>
> Key: HIVE-20548
> URL: https://issues.apache.org/jira/browse/HIVE-20548
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.1.0
>Reporter: zhangbutao
>Priority: Major
>
> We start llap through yarn service instead of slider, and some problems
> happen as follows:
> {code:java}
> 2018-09-12 19:32:48,629 - LLAP start command:
> /usr/bch/current/hive-server2/bin/hive --service llap --size 10930m
> --startImmediately --name llap0 --cache 0m --xmx 8m --loglevel INFO --output
> /var/lib/ambari-agent/tmp/llap-yarn-service_2018-09-12_11-32-48
> --service-placement 4 --skiphadoopversion --skiphbasecp --instances 1
> --logger query-routing --args " -XX:+AlwaysPreTouch -XX:+UseG1GC
> -XX:TLABSize=8m -XX:+ResizeTLAB -XX:+UseNUMA -XX:+AggressiveOpts
> -XX:InitiatingHeapOccupancyPercent=70 -XX:+UnlockExperimentalVMOptions
> -XX:G1MaxNewSizePercent=40 -XX:G1ReservePercent=20 -XX:MaxGCPauseMillis=200
> -XX:MetaspaceSize=1024m"
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/bch/3.0.0/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/bch/3.0.0/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> WARN conf.HiveConf: HiveConf of name hive.hook.proto.base-directory does not
> exist
> WARN conf.HiveConf: HiveConf of name hive.strict.managed.tables does not exist
> WARN conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does
> not exist
> WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist
> WARN conf.HiveConf: HiveConf of name hive.druid.select.distribute does not
> exist
> WARN cli.LlapServiceDriver: Ignoring unknown llap server parameter:
> [hive.aux.jars.path]
> WARN cli.LlapServiceDriver: Java versions might not match :
> JAVA_HOME=[/usr/jdk64/jdk1.8.0_112],process jre=[/usr/jdk64/jdk1.8.0_112/jre]
> WARN conf.HiveConf: HiveConf of name hive.hook.proto.base-directory does not
> exist
> WARN conf.HiveConf: HiveConf of name hive.strict.managed.tables does not exist
> WARN conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does
> not exist
> WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist
> WARN conf.HiveConf: HiveConf of name hive.druid.select.distribute does not
> exist
> WARN conf.HiveConf: HiveConf of name hive.hook.proto.base-directory does not
> exist
> WARN conf.HiveConf: HiveConf of name hive.strict.managed.tables does not exist
> WARN conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does
> not exist
> WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist
> WARN conf.HiveConf: HiveConf of name hive.druid.select.distribute does not
> exist
> 11:32:54 Running as a child of LlapServiceDriver
> 11:32:54 Prepared the files
> 11:33:13 Packaged the files
> WARN curator.CuratorZookeeperClient: session timeout [1] is less than
> connection timeout [15000]
> ERROR client.ServiceClient: Error on destroy 'llap0': not found.
> WARN client.ServiceClient: Property yarn.service.framework.path has a value
> /bch/apps/3.0.0/yarn/service-dep.tar.gz, but is not a valid file
> 2018-09-12 19:33:17,385 -
> 2018-09-12 19:33:17,385 - LLAP status command :
> /usr/bch/current/hive-server2/bin/hive --service llapstatus -w -r 0.8 -i 2 -t
> 400
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/bch/3.0.0/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/bch/3.0.0/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> WARN conf.HiveConf: HiveConf of name hive.hook.proto.base-directory does not
> exist
> WARN conf.HiveConf: HiveConf of name hive.strict.managed.tables does not exist
> WARN conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does
> not exist
> WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist
> WARN conf.HiveConf: HiveConf of name hive.druid.select.distribute
[jira] [Resolved] (HIVE-19897) Add more tests for parallel compilation
[ https://issues.apache.org/jira/browse/HIVE-19897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen resolved HIVE-19897. - Resolution: Fixed > Add more tests for parallel compilation > > > Key: HIVE-19897 > URL: https://issues.apache.org/jira/browse/HIVE-19897 > Project: Hive > Issue Type: Test > Components: HiveServer2 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-19897.1.patch, HIVE-19897.3.patch > > > The two parallel compilation tests in > org.apache.hive.jdbc.TestJdbcWithMiniHS2 do not real cover the case of > queries compile concurrently from different connections. No sure it is on > purpose or by mistake. Add more tests to cover the case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20535) Add new configuration to set the size of the global compile lock
[ https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] denys kuzmenko updated HIVE-20535: -- Attachment: HIVE-20535.2.patch > Add new configuration to set the size of the global compile lock > > > Key: HIVE-20535 > URL: https://issues.apache.org/jira/browse/HIVE-20535 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20535.1.patch, HIVE-20535.2.patch > > > When removing the compile lock, it is quite risky to remove it entirely. > It would be good to provide a pool size for the concurrent compilation, so > the administrator can limit the load -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20527) Intern table descriptors from spark task
[ https://issues.apache.org/jira/browse/HIVE-20527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-20527: --- Attachment: HIVE-20527.1.patch > Intern table descriptors from spark task > > > Key: HIVE-20527 > URL: https://issues.apache.org/jira/browse/HIVE-20527 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20527.1.patch, HIVE-20527.1.patch, > HIVE-20527.1.patch > > > Table descriptors from MR tasks and Tez tasks are interned. This fix is to > intern table desc from spark tasks as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-8446) Failing to delete data after dropping a table or database should result in error
[ https://issues.apache.org/jira/browse/HIVE-8446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613743#comment-16613743 ] Bharathkrishna Guruvayoor Murali commented on HIVE-8446: Hi, I am interested in working on this issue. Please let me know if I can assign it to myself. > Failing to delete data after dropping a table or database should result in > error > > > Key: HIVE-8446 > URL: https://issues.apache.org/jira/browse/HIVE-8446 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Priority: Major > > Currently if we drop a table and it fails to delete the data, the command > completes successfully. We should instead return an error to the user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[
https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613731#comment-16613731
]
Antal Sinkovits commented on HIVE-20440:
Yes, this is a good idea.
But I would not bother with the eviction handler, instead fill both caches.
Here is the idea:
There are two caches. L1 is a time based one (with a maintenance thread
cleaning it), and L2 is a soft referenced (I don't think I need a cleanup
thread here).
{noformat}
At first I check L1.
If its a miss, i will check L2.
If I miss again, I load the data in both caches.
If its a hit, I would take it from L2 to return (I think it would
make sense to put it back into L1 as well)
If its a hit in L1, I return it.
{noformat}
If the data is in L1, I can be sure that its in L2 as well, because L1 contains
strong references to the object.
Since these are references, its cheap to store them in two places.
Do you see any flaw in this approach?
> Create better cache eviction policy for SmallTableCache
> ---
>
> Key: HIVE-20440
> URL: https://issues.apache.org/jira/browse/HIVE-20440
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch,
> HIVE-20440.03.patch, HIVE-20440.04.patch
>
>
> Enhance the SmallTableCache, to use guava cache with soft references, so that
> we evict when there is memory pressure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20527) Intern table descriptors from spark task
[ https://issues.apache.org/jira/browse/HIVE-20527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613726#comment-16613726 ] Janaki Lahorani commented on HIVE-20527: These failures are not related to this patch. These tests are failing without the patch in my local run. > Intern table descriptors from spark task > > > Key: HIVE-20527 > URL: https://issues.apache.org/jira/browse/HIVE-20527 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20527.1.patch, HIVE-20527.1.patch > > > Table descriptors from MR tasks and Tez tasks are interned. This fix is to > intern table desc from spark tasks as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20536) Add Surrogate Keys function to Hive
[
https://issues.apache.org/jira/browse/HIVE-20536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613688#comment-16613688
]
Hive QA commented on HIVE-20536:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939467/HIVE-20536.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13759/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13759/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13759/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12939467/HIVE-20536.02.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939467 - PreCommit-HIVE-Build
> Add Surrogate Keys function to Hive
> ---
>
> Key: HIVE-20536
> URL: https://issues.apache.org/jira/browse/HIVE-20536
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-20536.01.patch, HIVE-20536.02.patch
>
>
> Surrogate keys is an ability to generate and use unique integers for each row
> in a table. If we have that ability then in conjunction with default clause
> we can get surrogate keys functionality. Consider following ddl:
> create table t1 (a string, b bigint default unique_long());
> We already have default clause wherein you can specify a function to provide
> values. So, what we need is udf which can generate unique longs for each row
> across queries for a table.
> Idea is to use write_id . This is a column in metastore table TXN_COMPONENTS
> whose value is determined at compile time to be used during query execution.
> Each query execution generates a new write_id. So, we can seed udf with this
> value during compilation.
> Then we statically allocate ranges for each task from which it can draw next
> long. So, lets say 64-bit write_id we divy up such that last 24 bits belong
> to original usage of it that is txns. Next 16 bits are used for task_attempts
> and last 24 bits to generate new long for each row. This implies we can allow
> 17M txns, 65K tasks and 17M rows in a task. If you hit any of those limits we
> can fail the query.
> Implementation wise: serialize write_id in initialize() of udf. Then during
> execute() we find out what task_attempt current task is and use it along with
> write_id() to get starting long and give a new value on each invocation of
> execute().
> Here we are assuming write_id can be determined at compile time, which should
> be the case but we need to figure out how to get handle to it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20536) Add Surrogate Keys function to Hive
[
https://issues.apache.org/jira/browse/HIVE-20536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613686#comment-16613686
]
Hive QA commented on HIVE-20536:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939467/HIVE-20536.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14939 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_functions]
(batchId=78)
org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsBinary.testOpenConnectionMetrics
(batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13758/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13758/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13758/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939467 - PreCommit-HIVE-Build
> Add Surrogate Keys function to Hive
> ---
>
> Key: HIVE-20536
> URL: https://issues.apache.org/jira/browse/HIVE-20536
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-20536.01.patch, HIVE-20536.02.patch
>
>
> Surrogate keys is an ability to generate and use unique integers for each row
> in a table. If we have that ability then in conjunction with default clause
> we can get surrogate keys functionality. Consider following ddl:
> create table t1 (a string, b bigint default unique_long());
> We already have default clause wherein you can specify a function to provide
> values. So, what we need is udf which can generate unique longs for each row
> across queries for a table.
> Idea is to use write_id . This is a column in metastore table TXN_COMPONENTS
> whose value is determined at compile time to be used during query execution.
> Each query execution generates a new write_id. So, we can seed udf with this
> value during compilation.
> Then we statically allocate ranges for each task from which it can draw next
> long. So, lets say 64-bit write_id we divy up such that last 24 bits belong
> to original usage of it that is txns. Next 16 bits are used for task_attempts
> and last 24 bits to generate new long for each row. This implies we can allow
> 17M txns, 65K tasks and 17M rows in a task. If you hit any of those limits we
> can fail the query.
> Implementation wise: serialize write_id in initialize() of udf. Then during
> execute() we find out what task_attempt current task is and use it along with
> write_id() to get starting long and give a new value on each invocation of
> execute().
> Here we are assuming write_id can be determined at compile time, which should
> be the case but we need to figure out how to get handle to it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20536) Add Surrogate Keys function to Hive
[
https://issues.apache.org/jira/browse/HIVE-20536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613622#comment-16613622
]
Hive QA commented on HIVE-20536:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 4 new + 639 unchanged - 0
fixed = 643 total (was 639) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 12 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13758/dev-support/hive-personality.sh
|
| git revision | master / 265 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13758/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13758/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13758/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add Surrogate Keys function to Hive
> ---
>
> Key: HIVE-20536
> URL: https://issues.apache.org/jira/browse/HIVE-20536
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-20536.01.patch, HIVE-20536.02.patch
>
>
> Surrogate keys is an ability to generate and use unique integers for each row
> in a table. If we have that ability then in conjunction with default clause
> we can get surrogate keys functionality. Consider following ddl:
> create table t1 (a string, b bigint default unique_long());
> We already have default clause wherein you can specify a function to provide
> values. So, what we need is udf which can generate unique longs for each row
> across queries for a table.
> Idea is to use write_id . This is a column in metastore table TXN_COMPONENTS
> whose value is determined at compile time to be used during query execution.
> Each query execution generates a new write_id. So, we can seed udf with this
> value during compilation.
> Then we statically allocate ranges for each task from which it can draw next
> long. So, lets say 64-bit write_id we divy up such that last 24 bits belong
> to original usage of it that is
[jira] [Commented] (HIVE-20527) Intern table descriptors from spark task
[
https://issues.apache.org/jira/browse/HIVE-20527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613583#comment-16613583
]
Hive QA commented on HIVE-20527:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939453/HIVE-20527.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 14939 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert]
(batchId=193)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13757/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13757/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13757/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939453 - PreCommit-HIVE-Build
> Intern table descriptors from spark task
>
>
> Key: HIVE-20527
> URL: https://issues.apache.org/jira/browse/HIVE-20527
> Project: Hive
> Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-20527.1.patch, HIVE-20527.1.patch
>
>
> Table descriptors from MR tasks and Tez tasks are interned. This fix is to
> intern table desc from spark tasks as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20527) Intern table descriptors from spark task
[
https://issues.apache.org/jira/browse/HIVE-20527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613540#comment-16613540
]
Hive QA commented on HIVE-20527:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13757/dev-support/hive-personality.sh
|
| git revision | master / 265 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13757/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Intern table descriptors from spark task
>
>
> Key: HIVE-20527
> URL: https://issues.apache.org/jira/browse/HIVE-20527
> Project: Hive
> Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-20527.1.patch, HIVE-20527.1.patch
>
>
> Table descriptors from MR tasks and Tez tasks are interned. This fix is to
> intern table desc from spark tasks as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-20474) SASL negotiation failure
[
https://issues.apache.org/jira/browse/HIVE-20474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kevin resolved HIVE-20474.
--
Resolution: Fixed
> SASL negotiation failure
> -
>
> Key: HIVE-20474
> URL: https://issues.apache.org/jira/browse/HIVE-20474
> Project: Hive
> Issue Type: Bug
> Components: Authentication
>Affects Versions: 1.1.1
>Reporter: kevin
>Priority: Blocker
>
> error when running hiveserver2 behind the load balancer, When running
> through beeline it is working fine but getting error when running through
> oozie
>
> [HiveServer2-Handler-Pool: Thread-59]: SASL negotiation failure
> {color:#00}javax.security.sasl.SaslException: DIGEST-MD5: IO error
> acquiring password [Caused by
> org.apache.hadoop.security.token.SecretManager$InvalidToken: token expired or
> does not exist: HIVE_DELEGATION_TOKEN owner=hadoop, renewer=hive,
> realUser=hive/namenodestandby.lakala@lakala.com, issueDate=1535046422518,
> maxDate=1535651222518, sequenceNumber=7, masterKeyId=2]
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
> at
> org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:793)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:790)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:790)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: token
> expired or does not exist: HIVE_DELEGATION_TOKEN owner=hadoop, renewer=hive,
> realUser=hive/namenodestandby.lakala@lakala.com, issueDate=1535046422518,
> maxDate=1535651222518, sequenceNumber=7, masterKeyId=2
> at
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:114)
> at
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:56)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.getPassword(HadoopThriftAuthBridge.java:607)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.handle(HadoopThriftAuthBridge.java:638)
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
> ... 15 more
> [HiveServer2-Handler-Pool: Thread-59]: Error occurred during processing of
> message.
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException:
> DIGEST-MD5: IO error acquiring password
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:793)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:790)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:790)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused
[jira] [Commented] (HIVE-20474) SASL negotiation failure
[
https://issues.apache.org/jira/browse/HIVE-20474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613522#comment-16613522
]
kevin commented on HIVE-20474:
--
hiveserver2
set
hive.cluster.delegation.token.store.class=org.apache.hadoop.hive.thrift.ZooKeeperTokenStore
working fine
> SASL negotiation failure
> -
>
> Key: HIVE-20474
> URL: https://issues.apache.org/jira/browse/HIVE-20474
> Project: Hive
> Issue Type: Bug
> Components: Authentication
>Affects Versions: 1.1.1
>Reporter: kevin
>Priority: Blocker
>
> error when running hiveserver2 behind the load balancer, When running
> through beeline it is working fine but getting error when running through
> oozie
>
> [HiveServer2-Handler-Pool: Thread-59]: SASL negotiation failure
> {color:#00}javax.security.sasl.SaslException: DIGEST-MD5: IO error
> acquiring password [Caused by
> org.apache.hadoop.security.token.SecretManager$InvalidToken: token expired or
> does not exist: HIVE_DELEGATION_TOKEN owner=hadoop, renewer=hive,
> realUser=hive/namenodestandby.lakala@lakala.com, issueDate=1535046422518,
> maxDate=1535651222518, sequenceNumber=7, masterKeyId=2]
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
> at
> org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:793)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:790)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:790)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: token
> expired or does not exist: HIVE_DELEGATION_TOKEN owner=hadoop, renewer=hive,
> realUser=hive/namenodestandby.lakala@lakala.com, issueDate=1535046422518,
> maxDate=1535651222518, sequenceNumber=7, masterKeyId=2
> at
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:114)
> at
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:56)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.getPassword(HadoopThriftAuthBridge.java:607)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.handle(HadoopThriftAuthBridge.java:638)
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
> ... 15 more
> [HiveServer2-Handler-Pool: Thread-59]: Error occurred during processing of
> message.
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException:
> DIGEST-MD5: IO error acquiring password
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:793)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:790)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:790)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[jira] [Commented] (HIVE-20163) Simplify StringSubstrColStart Initialization
[
https://issues.apache.org/jira/browse/HIVE-20163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613509#comment-16613509
]
Naveen Gangam commented on HIVE-20163:
--
Looks good to me. +1
> Simplify StringSubstrColStart Initialization
>
>
> Key: HIVE-20163
> URL: https://issues.apache.org/jira/browse/HIVE-20163
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20163.1.patch, HIVE-20163.2.patch
>
>
> * Remove code
> * Remove exception handling
> * Remove {{printStackTrace}} call
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20537) Multi-column joins estimates with uncorrelated columns different in CBO and Hive
[ https://issues.apache.org/jira/browse/HIVE-20537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20537: --- Attachment: HIVE-20537.01.patch > Multi-column joins estimates with uncorrelated columns different in CBO and > Hive > > > Key: HIVE-20537 > URL: https://issues.apache.org/jira/browse/HIVE-20537 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20537.01.patch, HIVE-20537.01.patch, > HIVE-20537.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20537) Multi-column joins estimates with uncorrelated columns different in CBO and Hive
[
https://issues.apache.org/jira/browse/HIVE-20537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613496#comment-16613496
]
Hive QA commented on HIVE-20537:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12939449/HIVE-20537.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14939 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSparkSessionTimeout
(batchId=245)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13756/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13756/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13756/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12939449 - PreCommit-HIVE-Build
> Multi-column joins estimates with uncorrelated columns different in CBO and
> Hive
>
>
> Key: HIVE-20537
> URL: https://issues.apache.org/jira/browse/HIVE-20537
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20537.01.patch, HIVE-20537.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20095) Fix jdbc external table feature
[ https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613484#comment-16613484 ] Jesus Camacho Rodriguez commented on HIVE-20095: +1 > Fix jdbc external table feature > --- > > Key: HIVE-20095 > URL: https://issues.apache.org/jira/browse/HIVE-20095 > Project: Hive > Issue Type: Bug >Reporter: Jonathan Doron >Assignee: Jonathan Doron >Priority: Major > Attachments: HIVE-20095.1.patch, HIVE-20095.2.patch, > HIVE-20095.3.patch, HIVE-20095.4.patch, HIVE-20095.5.patch, > HIVE-20095.6.patch, HIVE-20095.7.patch > > > It seems like the committed code for HIVE-19161 > (7584b3276bebf64aa006eaa162c0a6264d8fcb56) reverted some of HIVE-18423 > updates, and therefore some of the external table queries are not working > correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20423) Set NULLS LAST as the default null ordering
[ https://issues.apache.org/jira/browse/HIVE-20423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613468#comment-16613468 ] Jesus Camacho Rodriguez commented on HIVE-20423: +1 > Set NULLS LAST as the default null ordering > --- > > Key: HIVE-20423 > URL: https://issues.apache.org/jira/browse/HIVE-20423 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20423.1.patch, HIVE-20423.2.patch, > HIVE-20423.3.patch, HIVE-20423.4.patch, HIVE-20423.4.patch, > HIVE-20423.5.patch, HIVE-20423.6.patch, HIVE-20423.7.patch, HIVE-20423.8.patch > > > HIVE-20150 TopNKeyOperator pushdown can be more efficient if NULLS LAST > becomes the default null ordering. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20537) Multi-column joins estimates with uncorrelated columns different in CBO and Hive
[
https://issues.apache.org/jira/browse/HIVE-20537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613441#comment-16613441
]
Hive QA commented on HIVE-20537:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13756/dev-support/hive-personality.sh
|
| git revision | master / 265 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13756/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Multi-column joins estimates with uncorrelated columns different in CBO and
> Hive
>
>
> Key: HIVE-20537
> URL: https://issues.apache.org/jira/browse/HIVE-20537
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20537.01.patch, HIVE-20537.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
