[jira] [Updated] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22240:
--
Status: Patch Available (was: In Progress)
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:793)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 11 more
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast
> to org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.iterate(GenericUDAFPercentileCont.java:259)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:214)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:639)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:814)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:720)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:788)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22240:
--
Attachment: HIVE-22240.1.patch
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:793)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 11 more
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast
> to org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.iterate(GenericUDAFPercentileCont.java:259)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:214)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:639)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:814)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:720)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:788)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22275) OperationManager.queryIdOperation does not properly clean up multiple queryIds
[
https://issues.apache.org/jira/browse/HIVE-22275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942500#comment-16942500
]
Hive QA commented on HIVE-22275:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} service in master has 49 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} service: The patch generated 3 new + 25 unchanged - 0
fixed = 28 total (was 25) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18822/dev-support/hive-personality.sh
|
| git revision | master / 5717cde |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18822/yetus/diff-checkstyle-service.txt
|
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18822/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> OperationManager.queryIdOperation does not properly clean up multiple queryIds
> --
>
> Key: HIVE-22275
> URL: https://issues.apache.org/jira/browse/HIVE-22275
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-22275.1.patch, HIVE-22275.2.patch
>
>
> In the case that multiple statements are run by a single Session before being
> cleaned up, it appears that OperationManager.queryIdOperation is not cleaned
> up properly.
> See the log statements below - with the exception of the first "Removed
> queryId:" log line, the queryId listed during cleanup is the same, when each
> of these handles should have their own queryId. Looks like only the last
> queryId executed is being cleaned up.
> As a result, HS2 can run out of memory as OperationManager.queryIdOperation
> grows and never cleans these queryIds/Operations up.
> {noformat}
> 2019-09-13T08:37:36,785 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=dfed4c18-a284-4640-9f4a-1a20527105f9]
> 2019-09-13T08:37:38,432 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed
[jira] [Updated] (HIVE-22267) Support password based authentication in HMS
[ https://issues.apache.org/jira/browse/HIVE-22267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-22267: -- Attachment: HIVE-22267.05.patch Status: Patch Available (was: Open) > Support password based authentication in HMS > > > Key: HIVE-22267 > URL: https://issues.apache.org/jira/browse/HIVE-22267 > Project: Hive > Issue Type: New Feature >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22267.00.patch, HIVE-22267.01.patch, > HIVE-22267.02.patch, HIVE-22267.03.patch, HIVE-22267.04.patch, > HIVE-22267.05.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Similar to HS2, support password based authentication in HMS. > Right now we provide LDAP and CONFIG based options. The later allows to set > user and password in config and is used only for testing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22267) Support password based authentication in HMS
[ https://issues.apache.org/jira/browse/HIVE-22267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-22267: -- Status: Open (was: Patch Available) > Support password based authentication in HMS > > > Key: HIVE-22267 > URL: https://issues.apache.org/jira/browse/HIVE-22267 > Project: Hive > Issue Type: New Feature >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22267.00.patch, HIVE-22267.01.patch, > HIVE-22267.02.patch, HIVE-22267.03.patch, HIVE-22267.04.patch, > HIVE-22267.05.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Similar to HS2, support password based authentication in HMS. > Right now we provide LDAP and CONFIG based options. The later allows to set > user and password in config and is used only for testing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-10328) Enable new return path for cbo
[
https://issues.apache.org/jira/browse/HIVE-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942467#comment-16942467
]
Hive QA commented on HIVE-10328:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} common: The patch generated 1 new + 366 unchanged - 1
fixed = 367 total (was 367) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18821/dev-support/hive-personality.sh
|
| git revision | master / 5717cde |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18821/yetus/diff-checkstyle-common.txt
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18821/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enable new return path for cbo
> --
>
> Key: HIVE-10328
> URL: https://issues.apache.org/jira/browse/HIVE-10328
> Project: Hive
> Issue Type: Task
> Components: CBO
>Reporter: Ashutosh Chauhan
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-10328.1.patch, HIVE-10328.10.patch,
> HIVE-10328.11.patch, HIVE-10328.12.patch, HIVE-10328.13.patch,
> HIVE-10328.14.patch, HIVE-10328.15.patch, HIVE-10328.2.patch,
> HIVE-10328.3.patch, HIVE-10328.4.patch, HIVE-10328.4.patch,
> HIVE-10328.5.patch, HIVE-10328.6.patch, HIVE-10328.7.patch,
> HIVE-10328.8.patch, HIVE-10328.9.patch, HIVE-10328.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
[
https://issues.apache.org/jira/browse/HIVE-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942459#comment-16942459
]
Hive QA commented on HIVE-22252:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981912/HIVE-22252.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17017 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=363)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18820/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18820/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18820/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981912 - PreCommit-HIVE-Build
> Fix caught NullPointerExceptions generated during EXPLAIN
> -
>
> Key: HIVE-22252
> URL: https://issues.apache.org/jira/browse/HIVE-22252
> Project: Hive
> Issue Type: Bug
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-22252.1.patch, HIVE-22252.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While debugging an issue I noticed that during EXPLAIN the following methods
> throw a NullPointerException:
> VectorColumnOutputMapping#finalize
> AbstractOperatorDesc#getUserLevelStatistics
> AbstractOperatorDesc#getColumnExprMapForExplain
> The exceptions do end up getting caught but we should add null checks and
> gracefully to be less wasteful and to aid future debugging.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
[
https://issues.apache.org/jira/browse/HIVE-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942440#comment-16942440
]
Hive QA commented on HIVE-22252:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
29s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} ql: The patch generated 0 new + 1 unchanged - 1
fixed = 1 total (was 2) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18820/dev-support/hive-personality.sh
|
| git revision | master / 5717cde |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18820/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix caught NullPointerExceptions generated during EXPLAIN
> -
>
> Key: HIVE-22252
> URL: https://issues.apache.org/jira/browse/HIVE-22252
> Project: Hive
> Issue Type: Bug
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-22252.1.patch, HIVE-22252.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While debugging an issue I noticed that during EXPLAIN the following methods
> throw a NullPointerException:
> VectorColumnOutputMapping#finalize
> AbstractOperatorDesc#getUserLevelStatistics
> AbstractOperatorDesc#getColumnExprMapForExplain
> The exceptions do end up getting caught but we should add null checks and
> gracefully to be less wasteful and to aid future debugging.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22278) Upgrade log4j to 2.12.1
[
https://issues.apache.org/jira/browse/HIVE-22278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942435#comment-16942435
]
Hive QA commented on HIVE-22278:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
39s{color} | {color:blue} standalone-metastore/metastore-common in master has
32 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
24s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} testutils/ptest2 in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 15m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 1m
21s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18819/dev-support/hive-personality.sh
|
| git revision | master / 5717cde |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18819/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore standalone-metastore/metastore-common ql .
testutils/ptest2 U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18819/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade log4j to 2.12.1
> ---
>
> Key: HIVE-22278
> URL: https://issues.apache.org/jira/browse/HIVE-22278
> Project: Hive
> Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22278.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive's currently using log4j 2.10.0 and according to HIVE-21273, a number of
> issues are present in it, which can be resolved by upgrading to 2.12.1:
> Curly braces in parameters are treated as placeholders
> affectsVersions:2.8.2;2.9.0;2.10.0
>
>
[jira] [Commented] (HIVE-22278) Upgrade log4j to 2.12.1
[
https://issues.apache.org/jira/browse/HIVE-22278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942431#comment-16942431
]
Hive QA commented on HIVE-22278:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981910/HIVE-22278.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17017 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testHushableRandomAccessFileAppender
(batchId=281)
org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout
(batchId=281)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18819/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18819/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18819/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981910 - PreCommit-HIVE-Build
> Upgrade log4j to 2.12.1
> ---
>
> Key: HIVE-22278
> URL: https://issues.apache.org/jira/browse/HIVE-22278
> Project: Hive
> Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22278.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive's currently using log4j 2.10.0 and according to HIVE-21273, a number of
> issues are present in it, which can be resolved by upgrading to 2.12.1:
> Curly braces in parameters are treated as placeholders
> affectsVersions:2.8.2;2.9.0;2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2032?filter=allopenissues]
> Remove Log4J API dependency on Management APIs
> affectsVersions:2.9.1;2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2126?filter=allopenissues]
> Log4j2 throws NoClassDefFoundError in Java 9
> affectsVersions:2.10.0;2.11.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2129?filter=allopenissues]
> ThreadContext map is cleared => entries are only available for one log event
> affectsVersions:2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2158?filter=allopenissues]
> Objects held in SortedArrayStringMap cannot be filtered during serialization
> affectsVersions:2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2163?filter=allopenissues]
> NullPointerException at
> org.apache.logging.log4j.util.Activator.loadProvider(Activator.java:81) in
> log4j 2.10.0
> affectsVersions:2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2182?filter=allopenissues]
> MarkerFilter onMismatch invalid attribute in .properties
> affectsVersions:2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2202?filter=allopenissues]
> Configuration builder classes should look for "onMismatch"; not "onMisMatch".
>
> affectsVersions:2.4;2.4.1;2.5;2.6;2.6.1;2.6.2;2.7;2.8;2.8.1;2.8.2;2.9.0;2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2219?filter=allopenissues]
> Empty Automatic-Module-Name Header
> affectsVersions:2.10.0;2.11.0;3.0.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2254?filter=allopenissues]
> ConcurrentModificationException from
> org.apache.logging.log4j.status.StatusLogger.(StatusLogger.java:71)
> affectsVersions:2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2276?filter=allopenissues]
> Allow SystemPropertiesPropertySource to run with a SecurityManager that
> rejects system property access
> affectsVersions:2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2279?filter=allopenissues]
> ParserConfigurationException when using Log4j with
> oracle.xml.jaxp.JXDocumentBuilderFactory
> affectsVersions:2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2283?filter=allopenissues]
> Log4j 2.10+not working with SLF4J 1.8 in OSGI environment
> affectsVersions:2.10.0;2.11.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2305?filter=allopenissues]
> fix the CacheEntry map in ThrowableProxy#toExtendedStackTrace to be put and
> gotten with same key
> affectsVersions:2.6.2;2.7;2.8;2.8.1;2.8.2;2.9.0;2.9.1;2.10.0;2.11.0
>
>
[jira] [Commented] (HIVE-22267) Support password based authentication in HMS
[
https://issues.apache.org/jira/browse/HIVE-22267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942405#comment-16942405
]
Hive QA commented on HIVE-22267:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981907/HIVE-22267.04.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18818/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18818/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18818/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-10-02 00:41:38.769
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-18818/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-10-02 00:41:38.773
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
8111ede..5717cde master -> origin/master
+ git reset --hard HEAD
HEAD is now at 8111ede HIVE-21987: Hive is unable to read Parquet int32
annotated with decimal (Marta Kuczora reviewed by Peter Vary)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 5717cde HIVE-21449: Implement 'WITHIN GROUP' clause (Krisztian
Kasa, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-10-02 00:41:39.759
+ rm -rf ../yetus_PreCommit-HIVE-Build-18818
+ mkdir ../yetus_PreCommit-HIVE-Build-18818
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-18818
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-18818/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: git diff header lacks filename information when removing 0 leading
pathname components (line 6803)
error: cannot apply binary patch to
'standalone-metastore/metastore-server/src/test/resources/creds/test.jceks'
without full index line
Falling back to three-way merge...
error: cannot apply binary patch to
'standalone-metastore/metastore-server/src/test/resources/creds/test.jceks'
without full index line
error:
standalone-metastore/metastore-server/src/test/resources/creds/test.jceks:
patch does not apply
error: hive-minikdc/src/test/java/org/apache/hive/minikdc/MiniHiveKdc.java:
does not exist in index
error:
hive-unit-hadoop2/src/test/java/org/apache/hadoop/hive/metastore/security/TestHadoopAuthBridge23.java:
does not exist in index
error: metastore-common/pom.xml: does not exist in index
error:
metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
does not exist in index
error:
metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
does not exist in index
error:
metastore-common/src/main/java/org/apache/hadoop/hive/metastore/security/HadoopThriftAuthBridge.java:
does not exist in index
error: metastore-server/pom.xml: does not exist in index
error:
metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
does not exist in index
error:
metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
does not exist in index
error:
metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStoreZK.java:
does not exist in index
error:
metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStoreZKBindHost.java:
does not exist in index
error: cannot
[jira] [Commented] (HIVE-21146) Enforce TransactionBatch size=1 for blob stores
[
https://issues.apache.org/jira/browse/HIVE-21146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942404#comment-16942404
]
Hive QA commented on HIVE-21146:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981893/HIVE-21146.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 17015 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactAfterAbortNew
(batchId=247)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactWhileStreaming
(batchId=247)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactWhileStreamingForSplitUpdateNew
(batchId=247)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactAfterAbortNew
(batchId=247)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactWhileStreaming
(batchId=247)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactWhileStreamingWithSplitUpdate
(batchId=247)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactWhileStreamingWithSplitUpdateNew
(batchId=247)
org.apache.hive.streaming.TestStreaming.testAllTypesDelimitedWriter
(batchId=360)
org.apache.hive.streaming.TestStreaming.testAllTypesDelimitedWriterInputStream
(batchId=360)
org.apache.hive.streaming.TestStreaming.testAutoRollTransactionBatch
(batchId=360)
org.apache.hive.streaming.TestStreaming.testCommitWithKeyValue (batchId=360)
org.apache.hive.streaming.TestStreaming.testErrorHandling (batchId=360)
org.apache.hive.streaming.TestStreaming.testFileDumpCorruptDataFiles
(batchId=360)
org.apache.hive.streaming.TestStreaming.testFileDumpCorruptSideFiles
(batchId=360)
org.apache.hive.streaming.TestStreaming.testHeartbeat (batchId=360)
org.apache.hive.streaming.TestStreaming.testInterleavedTransactionBatchCommits
(batchId=360)
org.apache.hive.streaming.TestStreaming.testMultipleTransactionBatchCommits
(batchId=360)
org.apache.hive.streaming.TestStreaming.testNoBuckets (batchId=360)
org.apache.hive.streaming.TestStreaming.testTransactionBatchAbortAndCommit
(batchId=360)
org.apache.hive.streaming.TestStreaming.testTransactionBatchCommitDelimited
(batchId=360)
org.apache.hive.streaming.TestStreaming.testTransactionBatchCommitJson
(batchId=360)
org.apache.hive.streaming.TestStreaming.testTransactionBatchCommitRegex
(batchId=360)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18817/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18817/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18817/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981893 - PreCommit-HIVE-Build
> Enforce TransactionBatch size=1 for blob stores
> ---
>
> Key: HIVE-21146
> URL: https://issues.apache.org/jira/browse/HIVE-21146
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21146.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Streaming Ingest API supports a concept of {{TransactionBatch}} where N
> transactions can be opened at once and the data in all of them will be
> written to the same delta_x_y directory where each transaction in the batch
> can be committed/aborted independently. The implementation relies on
> {{FSDataOutputStream.hflush()}} (called from OrcRecordUpdater}} which is
> available on HDFS but is often implemented as no-op in Blob store backed
> {{FileSystem}} objects.
> Need to add a check to {{HiveStreamingConnection()}} constructor to raise an
> error if {{builder.transactionBatchSize > 1}} and the target table/partitions
> are backed by something that doesn't support {{hflush()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21924) Split text files even if header/footer exists
[
https://issues.apache.org/jira/browse/HIVE-21924?focusedWorklogId=321636=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321636
]
ASF GitHub Bot logged work on HIVE-21924:
-
Author: ASF GitHub Bot
Created on: 02/Oct/19 00:23
Start Date: 02/Oct/19 00:23
Worklog Time Spent: 10m
Work Description: mustafaiman commented on issue #791: HIVE-21924
URL: https://github.com/apache/hive/pull/791#issuecomment-537284007
@sankarh Can you review this again?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 321636)
Time Spent: 1.5h (was: 1h 20m)
> Split text files even if header/footer exists
> -
>
> Key: HIVE-21924
> URL: https://issues.apache.org/jira/browse/HIVE-21924
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Affects Versions: 2.4.0, 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21924.2.patch, HIVE-21924.3.patch,
> HIVE-21924.4.patch, HIVE-21924.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> https://github.com/apache/hive/blob/967a1cc98beede8e6568ce750ebeb6e0d048b8ea/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java#L494-L503
>
> {code}
> int headerCount = 0;
> int footerCount = 0;
> if (table != null) {
> headerCount = Utilities.getHeaderCount(table);
> footerCount = Utilities.getFooterCount(table, conf);
> if (headerCount != 0 || footerCount != 0) {
> // Input file has header or footer, cannot be splitted.
> HiveConf.setLongVar(conf, ConfVars.MAPREDMINSPLITSIZE,
> Long.MAX_VALUE);
> }
> }
> {code}
> this piece of code makes the CSV (or any text files with header/footer) files
> not splittable if header or footer is present.
> If only header is present, we can find the offset after first line break and
> use that to split. Similarly for footer, may be read few KB's of data at the
> end and find the last line break offset. Use that to determine the data range
> which can be used for splitting. Few reads during split generation are
> cheaper than not splitting the file at all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22275) OperationManager.queryIdOperation does not properly clean up multiple queryIds
[
https://issues.apache.org/jira/browse/HIVE-22275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-22275:
--
Attachment: HIVE-22275.2.patch
> OperationManager.queryIdOperation does not properly clean up multiple queryIds
> --
>
> Key: HIVE-22275
> URL: https://issues.apache.org/jira/browse/HIVE-22275
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-22275.1.patch, HIVE-22275.2.patch
>
>
> In the case that multiple statements are run by a single Session before being
> cleaned up, it appears that OperationManager.queryIdOperation is not cleaned
> up properly.
> See the log statements below - with the exception of the first "Removed
> queryId:" log line, the queryId listed during cleanup is the same, when each
> of these handles should have their own queryId. Looks like only the last
> queryId executed is being cleaned up.
> As a result, HS2 can run out of memory as OperationManager.queryIdOperation
> grows and never cleans these queryIds/Operations up.
> {noformat}
> 2019-09-13T08:37:36,785 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=dfed4c18-a284-4640-9f4a-1a20527105f9]
> 2019-09-13T08:37:38,432 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083736_c49cf3cc-cfe8-48a1-bd22-8b924dfb0396
> corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=dfed4c18-a284-4640-9f4a-1a20527105f9] with tag: null
> 2019-09-13T08:37:38,469 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=24d0030c-0e49-45fb-a918-2276f0941cfb]
> 2019-09-13T08:37:52,662 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=b983802c-1dec-4fa0-8680-d05ab555321b]
> 2019-09-13T08:37:56,239 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=75dbc531-2964-47b2-84d7-85b59f88999c]
> 2019-09-13T08:38:02,551 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=72c79076-9d67-4894-a526-c233fa5450b2]
> 2019-09-13T08:38:10,558 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=17b30a62-612d-4b70-9ba7-4287d2d9229b]
> 2019-09-13T08:38:16,930 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=ea97e99d-cc77-470b-b49a-b869c73a4615]
> 2019-09-13T08:38:20,440 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=a277b789-ebb8-4925-878f-6728d3e8c5fb]
> 2019-09-13T08:38:26,303 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=9a023ab8-aa80-45db-af88-94790cc83033]
> 2019-09-13T08:38:30,791 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=b697c801-7da0-4544-bcfa-442eb1d3bd77]
> 2019-09-13T08:39:10,187 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=bda93c8f-0822-4592-a61c-4701720a1a5c]
> 2019-09-13T08:39:15,471 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
> corresponding to operation:
[jira] [Commented] (HIVE-21146) Enforce TransactionBatch size=1 for blob stores
[
https://issues.apache.org/jira/browse/HIVE-21146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942376#comment-16942376
]
Hive QA commented on HIVE-21146:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18817/dev-support/hive-personality.sh
|
| git revision | master / 8111ede |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: streaming U: streaming |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18817/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enforce TransactionBatch size=1 for blob stores
> ---
>
> Key: HIVE-21146
> URL: https://issues.apache.org/jira/browse/HIVE-21146
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21146.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Streaming Ingest API supports a concept of {{TransactionBatch}} where N
> transactions can be opened at once and the data in all of them will be
> written to the same delta_x_y directory where each transaction in the batch
> can be committed/aborted independently. The implementation relies on
> {{FSDataOutputStream.hflush()}} (called from OrcRecordUpdater}} which is
> available on HDFS but is often implemented as no-op in Blob store backed
> {{FileSystem}} objects.
> Need to add a check to {{HiveStreamingConnection()}} constructor to raise an
> error if {{builder.transactionBatchSize > 1}} and the target table/partitions
> are backed by something that doesn't support {{hflush()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21449) implement 'WITHIN GROUP' clause
[ https://issues.apache.org/jira/browse/HIVE-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21449: --- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master, thanks [~kkasa]! > implement 'WITHIN GROUP' clause > --- > > Key: HIVE-21449 > URL: https://issues.apache.org/jira/browse/HIVE-21449 > Project: Hive > Issue Type: Improvement > Components: Parser, UDF >Reporter: László Bodor >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21449.1.patch, HIVE-21449.2.patch, > HIVE-21449.3.patch, HIVE-21449.4.patch, HIVE-21449.5.patch, > HIVE-21449.5.patch, HIVE-21449.6.patch, HIVE-21449.6.patch, > HIVE-21449.6.patch, HIVE-21449.6.patch, HIVE-21449.7.patch, > HIVE-21449.7.patch, HIVE-21449.8.patch, HIVE-21449.9.patch, HIVE-21449.9.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21449) implement 'WITHIN GROUP' clause
[
https://issues.apache.org/jira/browse/HIVE-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942367#comment-16942367
]
Hive QA commented on HIVE-21449:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981887/HIVE-21449.9.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17017 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18816/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18816/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18816/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981887 - PreCommit-HIVE-Build
> implement 'WITHIN GROUP' clause
> ---
>
> Key: HIVE-21449
> URL: https://issues.apache.org/jira/browse/HIVE-21449
> Project: Hive
> Issue Type: Improvement
> Components: Parser, UDF
>Reporter: László Bodor
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21449.1.patch, HIVE-21449.2.patch,
> HIVE-21449.3.patch, HIVE-21449.4.patch, HIVE-21449.5.patch,
> HIVE-21449.5.patch, HIVE-21449.6.patch, HIVE-21449.6.patch,
> HIVE-21449.6.patch, HIVE-21449.6.patch, HIVE-21449.7.patch,
> HIVE-21449.7.patch, HIVE-21449.8.patch, HIVE-21449.9.patch, HIVE-21449.9.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21449) implement 'WITHIN GROUP' clause
[
https://issues.apache.org/jira/browse/HIVE-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942351#comment-16942351
]
Hive QA commented on HIVE-21449:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
27s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} ql: The patch generated 0 new + 494 unchanged - 2
fixed = 494 total (was 496) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18816/dev-support/hive-personality.sh
|
| git revision | master / 8111ede |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18816/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> implement 'WITHIN GROUP' clause
> ---
>
> Key: HIVE-21449
> URL: https://issues.apache.org/jira/browse/HIVE-21449
> Project: Hive
> Issue Type: Improvement
> Components: Parser, UDF
>Reporter: László Bodor
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21449.1.patch, HIVE-21449.2.patch,
> HIVE-21449.3.patch, HIVE-21449.4.patch, HIVE-21449.5.patch,
> HIVE-21449.5.patch, HIVE-21449.6.patch, HIVE-21449.6.patch,
> HIVE-21449.6.patch, HIVE-21449.6.patch, HIVE-21449.7.patch,
> HIVE-21449.7.patch, HIVE-21449.8.patch, HIVE-21449.9.patch, HIVE-21449.9.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22273) Access check is failed when a temporary directory is removed
[
https://issues.apache.org/jira/browse/HIVE-22273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942336#comment-16942336
]
Hive QA commented on HIVE-22273:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981884/HIVE-22273.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17014 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18815/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18815/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18815/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981884 - PreCommit-HIVE-Build
> Access check is failed when a temporary directory is removed
>
>
> Key: HIVE-22273
> URL: https://issues.apache.org/jira/browse/HIVE-22273
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22273.1.patch, HIVE-22273.2.patch,
> HIVE-22273.3.patch, HIVE-22273.patch
>
>
> The following exception is thrown if a temporary file is deleted during the
> access checks:
> {code}
> 2019-09-24T16:12:59,611 ERROR [7e491237-1505-4388-afb9-5ec2a688b0dc
> HiveServer2-HttpHandler-Pool: Thread-11941]: authorizer.RangerHiveAuthorizer
> (:()) - Error getting permissions for hdfs://HDFS_FOLDER/TABLE_NAME
> java.io.FileNotFoundException: File
> hdfs://HDFS_FOLDER/TABLE_NAME/.hive-staging_hive_2019-09-24_16-12-48_899_7291847300113791212-245/_tmp.-ext-10001
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1059)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1119)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1116)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1126)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hive.common.FileUtils.checkIsOwnerOfFileHierarchy(FileUtils.java:561)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.hadoop.hive.common.FileUtils.checkIsOwnerOfFileHierarchy(FileUtils.java:564)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.hadoop.hive.common.FileUtils.checkIsOwnerOfFileHierarchy(FileUtils.java:564)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at org.apache.hadoop.hive.common.FileUtils$4.run(FileUtils.java:540)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at org.apache.hadoop.hive.common.FileUtils$4.run(FileUtils.java:536)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_171]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_171]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> ~[hadoop-common-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hive.common.FileUtils.isOwnerOfFileHierarchy(FileUtils.java:536)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.hadoop.hive.common.FileUtils.isOwnerOfFileHierarchy(FileUtils.java:527)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.isURIAccessAllowed(RangerHiveAuthorizer.java:1420)
> ~[?:?]
> at
> org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.checkPrivileges(RangerHiveAuthorizer.java:287)
> ~[?:?]
> at org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:1336)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:1100)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:709)
>
[jira] [Commented] (HIVE-22273) Access check is failed when a temporary directory is removed
[
https://issues.apache.org/jira/browse/HIVE-22273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942299#comment-16942299
]
Hive QA commented on HIVE-22273:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18815/dev-support/hive-personality.sh
|
| git revision | master / 8111ede |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18815/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Access check is failed when a temporary directory is removed
>
>
> Key: HIVE-22273
> URL: https://issues.apache.org/jira/browse/HIVE-22273
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22273.1.patch, HIVE-22273.2.patch,
> HIVE-22273.3.patch, HIVE-22273.patch
>
>
> The following exception is thrown if a temporary file is deleted during the
> access checks:
> {code}
> 2019-09-24T16:12:59,611 ERROR [7e491237-1505-4388-afb9-5ec2a688b0dc
> HiveServer2-HttpHandler-Pool: Thread-11941]: authorizer.RangerHiveAuthorizer
> (:()) - Error getting permissions for hdfs://HDFS_FOLDER/TABLE_NAME
> java.io.FileNotFoundException: File
> hdfs://HDFS_FOLDER/TABLE_NAME/.hive-staging_hive_2019-09-24_16-12-48_899_7291847300113791212-245/_tmp.-ext-10001
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1059)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1119)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1116)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>
[jira] [Commented] (HIVE-22270) Upgrade commons-io to 2.6
[
https://issues.apache.org/jira/browse/HIVE-22270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942292#comment-16942292
]
Hive QA commented on HIVE-22270:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981873/HIVE-22270.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17014 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcWithMiniHS2ErasureCoding.testExplainErasureCoding
(batchId=286)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18814/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18814/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18814/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981873 - PreCommit-HIVE-Build
> Upgrade commons-io to 2.6
> -
>
> Key: HIVE-22270
> URL: https://issues.apache.org/jira/browse/HIVE-22270
> Project: Hive
> Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22270.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive's currently using commons-io 2.4 and according to HIVE-21273, a number
> of issues are present in it, which can be resolved by upgrading to 2.6:
> IOUtils copyLarge() and skip() methods are performance hogs
> affectsVersions:2.3;2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-355?filter=allopenissues]
> CharSequenceInputStream#reset() behaves incorrectly in case when buffer size
> is not dividable by data size
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-356?filter=allopenissues]
> [Tailer] InterruptedException while the thead is sleeping is silently ignored
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-357?filter=allopenissues]
> IOUtils.contentEquals* methods returns false if input1 == input2; should
> return true
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-362?filter=allopenissues]
> Apache Commons - standard links for documents are failing
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-369?filter=allopenissues]
> FileUtils.sizeOfDirectoryAsBigInteger can overflow
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-390?filter=allopenissues]
> Regression in FileUtils.readFileToString from 2.0.1
> affectsVersions:2.1;2.2;2.3;2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-453?filter=allopenissues]
> Correct exception message in FileUtils.getFile(File; String...)
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-479?filter=allopenissues]
> org.apache.commons.io.FileUtils#waitFor waits too long
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-481?filter=allopenissues]
> FilenameUtils should handle embedded null bytes
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-484?filter=allopenissues]
> Exceptions are suppressed incorrectly when copying files.
> affectsVersions:2.4;2.5
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-502?filter=allopenissues]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22270) Upgrade commons-io to 2.6
[
https://issues.apache.org/jira/browse/HIVE-22270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942270#comment-16942270
]
Hive QA commented on HIVE-22270:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18814/dev-support/hive-personality.sh
|
| git revision | master / 8111ede |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18814/yetus/patch-asflicense-problems.txt
|
| modules | C: . testutils/ptest2 U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18814/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade commons-io to 2.6
> -
>
> Key: HIVE-22270
> URL: https://issues.apache.org/jira/browse/HIVE-22270
> Project: Hive
> Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22270.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive's currently using commons-io 2.4 and according to HIVE-21273, a number
> of issues are present in it, which can be resolved by upgrading to 2.6:
> IOUtils copyLarge() and skip() methods are performance hogs
> affectsVersions:2.3;2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-355?filter=allopenissues]
> CharSequenceInputStream#reset() behaves incorrectly in case when buffer size
> is not dividable by data size
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-356?filter=allopenissues]
> [Tailer] InterruptedException while the thead is sleeping is silently ignored
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-357?filter=allopenissues]
> IOUtils.contentEquals* methods returns false if input1 == input2; should
> return true
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-362?filter=allopenissues]
> Apache Commons - standard links for documents are failing
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-369?filter=allopenissues]
> FileUtils.sizeOfDirectoryAsBigInteger can overflow
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-390?filter=allopenissues]
> Regression in
[jira] [Commented] (HIVE-7145) Remove dependence on apache commons-lang
[
https://issues.apache.org/jira/browse/HIVE-7145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942238#comment-16942238
]
Hive QA commented on HIVE-7145:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981871/HIVE-7145.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17014 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18813/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18813/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18813/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981871 - PreCommit-HIVE-Build
> Remove dependence on apache commons-lang
>
>
> Key: HIVE-7145
> URL: https://issues.apache.org/jira/browse/HIVE-7145
> Project: Hive
> Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-7145.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We currently depend on both Apache commons-lang and commons-lang3. They are
> the same project, just at version 2.x vs 3.x. I propose that we move all of
> the references in Hive to commons-lang3 and remove the v2 usage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-7145) Remove dependence on apache commons-lang
[
https://issues.apache.org/jira/browse/HIVE-7145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942233#comment-16942233
]
Hive QA commented on HIVE-7145:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 11m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
16s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18813/dev-support/hive-personality.sh
|
| git revision | master / 8111ede |
| Default Java | 1.8.0_111 |
| modules | C: storage-api shims/common shims/0.23 common serde metastore
vector-code-gen ql llap-server service accumulo-handler beeline cli
druid-handler hbase-handler hplsql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18813/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove dependence on apache commons-lang
>
>
> Key: HIVE-7145
> URL: https://issues.apache.org/jira/browse/HIVE-7145
> Project: Hive
> Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-7145.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We currently depend on both Apache commons-lang and commons-lang3. They are
> the same project, just at version 2.x vs 3.x. I propose that we move all of
> the references in Hive to commons-lang3 and remove the v2 usage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-10328) Enable new return path for cbo
[ https://issues.apache.org/jira/browse/HIVE-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-10328: -- Assignee: Jesus Camacho Rodriguez (was: Vineet Garg) > Enable new return path for cbo > -- > > Key: HIVE-10328 > URL: https://issues.apache.org/jira/browse/HIVE-10328 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Ashutosh Chauhan >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-10328.1.patch, HIVE-10328.10.patch, > HIVE-10328.11.patch, HIVE-10328.12.patch, HIVE-10328.13.patch, > HIVE-10328.14.patch, HIVE-10328.15.patch, HIVE-10328.2.patch, > HIVE-10328.3.patch, HIVE-10328.4.patch, HIVE-10328.4.patch, > HIVE-10328.5.patch, HIVE-10328.6.patch, HIVE-10328.7.patch, > HIVE-10328.8.patch, HIVE-10328.9.patch, HIVE-10328.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-10328) Enable new return path for cbo
[ https://issues.apache.org/jira/browse/HIVE-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-10328: --- Attachment: HIVE-10328.15.patch > Enable new return path for cbo > -- > > Key: HIVE-10328 > URL: https://issues.apache.org/jira/browse/HIVE-10328 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Ashutosh Chauhan >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-10328.1.patch, HIVE-10328.10.patch, > HIVE-10328.11.patch, HIVE-10328.12.patch, HIVE-10328.13.patch, > HIVE-10328.14.patch, HIVE-10328.15.patch, HIVE-10328.2.patch, > HIVE-10328.3.patch, HIVE-10328.4.patch, HIVE-10328.4.patch, > HIVE-10328.5.patch, HIVE-10328.6.patch, HIVE-10328.7.patch, > HIVE-10328.8.patch, HIVE-10328.9.patch, HIVE-10328.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-10328) Enable new return path for cbo
[ https://issues.apache.org/jira/browse/HIVE-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-10328: -- Assignee: Vineet Garg (was: Jesus Camacho Rodriguez) > Enable new return path for cbo > -- > > Key: HIVE-10328 > URL: https://issues.apache.org/jira/browse/HIVE-10328 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Ashutosh Chauhan >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-10328.1.patch, HIVE-10328.10.patch, > HIVE-10328.11.patch, HIVE-10328.12.patch, HIVE-10328.13.patch, > HIVE-10328.14.patch, HIVE-10328.15.patch, HIVE-10328.2.patch, > HIVE-10328.3.patch, HIVE-10328.4.patch, HIVE-10328.4.patch, > HIVE-10328.5.patch, HIVE-10328.6.patch, HIVE-10328.7.patch, > HIVE-10328.8.patch, HIVE-10328.9.patch, HIVE-10328.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-9132) CBO: Calcite Operator To Hive Operator (Calcite Return Path)
[ https://issues.apache.org/jira/browse/HIVE-9132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-9132: - Assignee: Jesus Camacho Rodriguez (was: Laljo John Pullokkaran) > CBO: Calcite Operator To Hive Operator (Calcite Return Path) > > > Key: HIVE-9132 > URL: https://issues.apache.org/jira/browse/HIVE-9132 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Laljo John Pullokkaran >Assignee: Jesus Camacho Rodriguez >Priority: Major > > This is the umbrella issue that captures conversion of Calcite Operator Tree > to Hive Operator Tree. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22136) Turn on tez.bucket.pruning
[ https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22136: --- Status: Open (was: Patch Available) > Turn on tez.bucket.pruning > --- > > Key: HIVE-22136 > URL: https://issues.apache.org/jira/browse/HIVE-22136 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch, > HIVE-22136.3.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22136) Turn on tez.bucket.pruning
[ https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22136: --- Status: Patch Available (was: Open) > Turn on tez.bucket.pruning > --- > > Key: HIVE-22136 > URL: https://issues.apache.org/jira/browse/HIVE-22136 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch, > HIVE-22136.3.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22137) Implement alter/rename partition related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942178#comment-16942178
]
Hive QA commented on HIVE-22137:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981866/HIVE-22137.08.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17166 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18812/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18812/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18812/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981866 - PreCommit-HIVE-Build
> Implement alter/rename partition related methods on temporary tables
>
>
> Key: HIVE-22137
> URL: https://issues.apache.org/jira/browse/HIVE-22137
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22137.01.patch, HIVE-22137.02.patch,
> HIVE-22137.03.patch, HIVE-22137.04.patch, HIVE-22137.05.patch,
> HIVE-22137.06.patch, HIVE-22137.07.patch, HIVE-22137.08.patch
>
>
> IMetaStoreClient exposes the following methods related to altering of
> partitions:
> {code:java}
> void alter_partition(String dbName, String tblName, Partition newPart);
> void alter_partition(String catName, String dbName, String tblName, Partition
> newPart);
> void alter_partition(String dbName, String tblName, Partition newPart,
> EnvironmentContext environmentContext);
> void alter_partition(String catName, String dbName, String tblName, Partition
> newPart, EnvironmentContext environmentContext, String writeIdList);
> void alter_partition(String catName, String dbName, String tblName, Partition
> newPart, EnvironmentContext environmentContext);
> void alter_partitions(String dbName, String tblName, List
> newParts);
> void alter_partitions(String dbName, String tblName, List
> newParts, EnvironmentContext environmentContext);
> void alter_partitions(String dbName, String tblName, List
> newParts, EnvironmentContext environmentContext,String writeIdList, long
> writeId);
> void alter_partitions(String catName, String dbName, String tblName,
> List newParts);
> void alter_partitions(String catName, String dbName, String tblName,
> List newParts, EnvironmentContext environmentContext, String
> writeIdList, long writeId);
> void renamePartition(final String dbname, final String tableName, final
> List part_vals, final Partition newPart);
> void renamePartition(String catName, String dbname, String tableName,
> List part_vals, Partition newPart, String validWriteIds){code}
> These should be implemented, in order to completely support partition on
> temporary tables.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
[ https://issues.apache.org/jira/browse/HIVE-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-22252: -- Labels: pull-request-available (was: ) > Fix caught NullPointerExceptions generated during EXPLAIN > - > > Key: HIVE-22252 > URL: https://issues.apache.org/jira/browse/HIVE-22252 > Project: Hive > Issue Type: Bug >Reporter: John Sherman >Assignee: John Sherman >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-22252.1.patch, HIVE-22252.2.patch > > > While debugging an issue I noticed that during EXPLAIN the following methods > throw a NullPointerException: > VectorColumnOutputMapping#finalize > AbstractOperatorDesc#getUserLevelStatistics > AbstractOperatorDesc#getColumnExprMapForExplain > The exceptions do end up getting caught but we should add null checks and > gracefully to be less wasteful and to aid future debugging. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
[ https://issues.apache.org/jira/browse/HIVE-22252?focusedWorklogId=321382=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321382 ] ASF GitHub Bot logged work on HIVE-22252: - Author: ASF GitHub Bot Created on: 01/Oct/19 17:28 Start Date: 01/Oct/19 17:28 Worklog Time Spent: 10m Work Description: jfsii commented on pull request #799: HIVE-22252: Fix caught NullPointerExceptions generated during EXPLAIN URL: https://github.com/apache/hive/pull/799 Waiting for a green pass from CI. (Last run failed due to flaky MTT test) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321382) Remaining Estimate: 0h Time Spent: 10m > Fix caught NullPointerExceptions generated during EXPLAIN > - > > Key: HIVE-22252 > URL: https://issues.apache.org/jira/browse/HIVE-22252 > Project: Hive > Issue Type: Bug >Reporter: John Sherman >Assignee: John Sherman >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-22252.1.patch, HIVE-22252.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > While debugging an issue I noticed that during EXPLAIN the following methods > throw a NullPointerException: > VectorColumnOutputMapping#finalize > AbstractOperatorDesc#getUserLevelStatistics > AbstractOperatorDesc#getColumnExprMapForExplain > The exceptions do end up getting caught but we should add null checks and > gracefully to be less wasteful and to aid future debugging. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
[ https://issues.apache.org/jira/browse/HIVE-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-22252: Attachment: HIVE-22252.2.patch > Fix caught NullPointerExceptions generated during EXPLAIN > - > > Key: HIVE-22252 > URL: https://issues.apache.org/jira/browse/HIVE-22252 > Project: Hive > Issue Type: Bug >Reporter: John Sherman >Assignee: John Sherman >Priority: Minor > Attachments: HIVE-22252.1.patch, HIVE-22252.2.patch > > > While debugging an issue I noticed that during EXPLAIN the following methods > throw a NullPointerException: > VectorColumnOutputMapping#finalize > AbstractOperatorDesc#getUserLevelStatistics > AbstractOperatorDesc#getColumnExprMapForExplain > The exceptions do end up getting caught but we should add null checks and > gracefully to be less wasteful and to aid future debugging. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[
https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942166#comment-16942166
]
Rajkumar Singh commented on HIVE-22255:
---
[~pvary] with "mark the table compacted " I was referring to
txnHandler.markCompacted which put the table in "Ready to Clean" state, we can
use the same method if we encounter any table with n base and 0 deltas.
> Hive don't trigger Major Compaction automatically if table contains only base
> files
>
>
> Key: HIVE-22255
> URL: https://issues.apache.org/jira/browse/HIVE-22255
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 3.1.2
> Environment: Hive-3.1.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> user may run into the issue if the table consists of all base files but no
> delta, then the following condition will yield false and automatic major
> compaction will be skipped.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313]
>
> Steps to Reproduce:
> # create Acid table
> {code:java}
> // create table myacid(id int);
> {code}
> # Run multiple insert table
> {code:java}
> // insert overwrite table myacid values(1);insert overwrite table myacid
> values(2),(3),(4){code}
> # DFS ls output
> {code:java}
> // dfs -ls -R /warehouse/tablespace/managed/hive/myacid;
> ++
> | DFS Output |
> ++
> | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001 |
> | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001/_orc_acid_version |
> | -rw-rw+ 3 hive hadoop 610 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001/bucket_0 |
> | drwxrwx---+ - hive hadoop 0 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002 |
> | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002/_orc_acid_version |
> | -rw-rw+ 3 hive hadoop 633 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002/bucket_0 |
> ++{code}
>
> you will see that Major compaction will not be trigger until you run alter
> table compact MAJOR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22277) TestBuddyAllocator.testMTT[2] is flaky (direct=true, mmap=true)
[
https://issues.apache.org/jira/browse/HIVE-22277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942164#comment-16942164
]
John Sherman commented on HIVE-22277:
-
This is a duplicate of HIVE-22175
> TestBuddyAllocator.testMTT[2] is flaky (direct=true, mmap=true)
> ---
>
> Key: HIVE-22277
> URL: https://issues.apache.org/jira/browse/HIVE-22277
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Priority: Major
> Attachments: hive.log.tar.gz
>
>
> It's more than flaky. Recently, in most of the cases it's the only failing
> test in otherwise green runs.
> full hive logs: [^hive.log.tar.gz]
> {code}
> Error Message
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> Stacktrace
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at
> org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:145)
> ... 33 more
> Caused by:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at
>
[jira] [Commented] (HIVE-21273) Your project apache/hive is using buggy third-party libraries [WARNING]
[
https://issues.apache.org/jira/browse/HIVE-21273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942159#comment-16942159
]
David Lavati commented on HIVE-21273:
-
{color:#172b4d}Thank you for the feedback! This was on my radar for a while,
but finally got around to check it. I'm just gonna mention the related
HADOOP-16113 you opened in the same fashion, as in a number of places this
project's dependendent on it. Here are my findings with apache/hive HEAD being
at 33ccc9bef:{color}
* {color:#172b4d}httpcomponents httpclient: DONE{color}
** {color:#172b4d}reviewed version: 4.5.2{color}
** {color:#172b4d}current version: 4.5.6 HIVE-21306
[https://github.com/apache/hive/commit/11b8c2ce7a544df2147485318e3f15adc203214a]{color}
* {color:#172b4d}commons-cli: TODO, critical?{color}
** {color:#172b4d}reviewed version: 1.2{color}
** {color:#172b4d}this seems to be depending on Hadoop{color}
* {color:#172b4d}commons-io: WIP{color}
** {color:#172b4d}reviewed version: 2.4{color}
** {color:#172b4d}feasable upgrade to: 2.6, see HIVE-22270 (hadoop is at 2.5
by HADOOP-15261 in 3.1, so this is probably not bound by it){color}
* {color:#172b4d}log4j2: WIP{color}
** {color:#172b4d}reviewed version: 2.10.0{color}
** {color:#172b4d}feasable upgrade to 2.12.1, see HIVE-22278{color}
* {color:#172b4d}commons-lang3: DONE for both counted cases
{color}
** {color:#172b4d}reviewed version: 3.3.2{color}
** {color:#172b4d}current version: 3.9 HIVE-22132
[https://github.com/apache/hive/commit/41770e9ce19a4730b2f96545f7e6d697f2354be8]{color}
* {color:#172b4d}commons-lang: WIP{color}
** {color:#172b4d}version: 2.6{color}
** {color:#172b4d}looks feasable, see HIVE-7145{color}
> Your project apache/hive is using buggy third-party libraries [WARNING]
> ---
>
> Key: HIVE-21273
> URL: https://issues.apache.org/jira/browse/HIVE-21273
> Project: Hive
> Issue Type: Bug
>Reporter: Kaifeng Huang
>Priority: Major
>
> Hi, there!
> We are a research team working on third-party library analysis. We have
> found that some widely-used third-party libraries in your project have
> major/critical bugs, which will degrade the quality of your project. We
> highly recommend you to update those libraries to new versions.
> We have attached the buggy third-party libraries and corresponding jira
> issue links below for you to have more detailed information.
> 1. org.apache.httpcomponents httpclient(pom.xml)
> version: 4.5.2
> Jira issues:
>
> org.apache.http.impl.client.AbstractHttpClient#createClientConnectionManager
> Does not account for context class loader
> affectsVersions:4.4.1;4.5;4.5.1;4.5.2
>
> https://issues.apache.org/jira/projects/HTTPCLIENT/issues/HTTPCLIENT-1727?filter=allopenissues
> Memory Leak in OSGi support
> affectsVersions:4.4.1;4.5.2
>
> https://issues.apache.org/jira/projects/HTTPCLIENT/issues/HTTPCLIENT-1749?filter=allopenissues
> SystemDefaultRoutePlanner: Possible null pointer dereference
> affectsVersions:4.5.2
>
> https://issues.apache.org/jira/projects/HTTPCLIENT/issues/HTTPCLIENT-1766?filter=allopenissues
> Null pointer dereference in EofSensorInputStream and ResponseEntityProxy
> affectsVersions:4.5.2
>
> https://issues.apache.org/jira/projects/HTTPCLIENT/issues/HTTPCLIENT-1767?filter=allopenissues
> [OSGi] WeakList needs to support "clear" method
> affectsVersions:4.5.2;5.0 Alpha1
>
> https://issues.apache.org/jira/projects/HTTPCLIENT/issues/HTTPCLIENT-1772?filter=allopenissues
> [OSGi] HttpProxyConfigurationActivator does not unregister
> HttpClientBuilderFactory
> affectsVersions:4.5.2
>
> https://issues.apache.org/jira/projects/HTTPCLIENT/issues/HTTPCLIENT-1773?filter=allopenissues
> Why is Retry around Redirect and not the other way round
> affectsVersions:4.5.2
>
> https://issues.apache.org/jira/projects/HTTPCLIENT/issues/HTTPCLIENT-1800?filter=allopenissues
> 2. commons-cli
> commons-cli(pom.xml,testutils/ptest2/pom.xml,upgrade-acid/pre-upgrade/pom.xml)
> version: 1.2
> Jira issues:
> Unable to select a pure long option in a group
> affectsVersions:1.0;1.1;1.2
>
> https://issues.apache.org/jira/projects/CLI/issues/CLI-182?filter=allopenissues
> Clear the selection from the groups before parsing
> affectsVersions:1.0;1.1;1.2
>
> https://issues.apache.org/jira/projects/CLI/issues/CLI-183?filter=allopenissues
> Commons CLI incorrectly stripping leading and trailing quotes
> affectsVersions:1.1;1.2
>
> https://issues.apache.org/jira/projects/CLI/issues/CLI-185?filter=allopenissues
> Coding error: OptionGroup.setSelected causes
>
[jira] [Updated] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
[ https://issues.apache.org/jira/browse/HIVE-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-22252: Description: While debugging an issue I noticed that during EXPLAIN the following methods throw a NullPointerException: VectorColumnOutputMapping#finalize AbstractOperatorDesc#getUserLevelStatistics AbstractOperatorDesc#getColumnExprMapForExplain The exceptions do end up getting caught but we should add null checks and gracefully to be less wasteful and to aid future debugging. was: While debugging an issue I noticed that during EXPLAIN the following methods throw a NullPointerException: VectorColumnOutputMapping::finalize AbstractOperatorDesc::getUserLevelStatistics AbstractOperatorDesc::getColumnExprMapForExplain The exceptions do end up getting caught but we should add null checks and gracefully to be less wasteful and to aid future debugging. > Fix caught NullPointerExceptions generated during EXPLAIN > - > > Key: HIVE-22252 > URL: https://issues.apache.org/jira/browse/HIVE-22252 > Project: Hive > Issue Type: Bug >Reporter: John Sherman >Assignee: John Sherman >Priority: Minor > Attachments: HIVE-22252.1.patch > > > While debugging an issue I noticed that during EXPLAIN the following methods > throw a NullPointerException: > VectorColumnOutputMapping#finalize > AbstractOperatorDesc#getUserLevelStatistics > AbstractOperatorDesc#getColumnExprMapForExplain > The exceptions do end up getting caught but we should add null checks and > gracefully to be less wasteful and to aid future debugging. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22278) Upgrade log4j to 2.12.1
[ https://issues.apache.org/jira/browse/HIVE-22278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-22278: Attachment: HIVE-22278.patch Status: Patch Available (was: Open) The following had to be addressed: * SlidingFilenameRolloverStrategy had to implement a new method, but since this strategy's goal is to ignore renaming, this probably isn't required by us (See LOG4J2-1906) * log4j-core had to be explicitly added in a few places as a new dependency, as it became optional for log4j-1.2-api (See LOG4J2-2556) > Upgrade log4j to 2.12.1 > --- > > Key: HIVE-22278 > URL: https://issues.apache.org/jira/browse/HIVE-22278 > Project: Hive > Issue Type: Improvement >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22278.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Hive's currently using log4j 2.10.0 and according to HIVE-21273, a number of > issues are present in it, which can be resolved by upgrading to 2.12.1: > Curly braces in parameters are treated as placeholders > affectsVersions:2.8.2;2.9.0;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2032?filter=allopenissues] > Remove Log4J API dependency on Management APIs > affectsVersions:2.9.1;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2126?filter=allopenissues] > Log4j2 throws NoClassDefFoundError in Java 9 > affectsVersions:2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2129?filter=allopenissues] > ThreadContext map is cleared => entries are only available for one log event > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2158?filter=allopenissues] > Objects held in SortedArrayStringMap cannot be filtered during serialization > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2163?filter=allopenissues] > NullPointerException at > org.apache.logging.log4j.util.Activator.loadProvider(Activator.java:81) in > log4j 2.10.0 > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2182?filter=allopenissues] > MarkerFilter onMismatch invalid attribute in .properties > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2202?filter=allopenissues] > Configuration builder classes should look for "onMismatch"; not "onMisMatch". > > affectsVersions:2.4;2.4.1;2.5;2.6;2.6.1;2.6.2;2.7;2.8;2.8.1;2.8.2;2.9.0;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2219?filter=allopenissues] > Empty Automatic-Module-Name Header > affectsVersions:2.10.0;2.11.0;3.0.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2254?filter=allopenissues] > ConcurrentModificationException from > org.apache.logging.log4j.status.StatusLogger.(StatusLogger.java:71) > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2276?filter=allopenissues] > Allow SystemPropertiesPropertySource to run with a SecurityManager that > rejects system property access > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2279?filter=allopenissues] > ParserConfigurationException when using Log4j with > oracle.xml.jaxp.JXDocumentBuilderFactory > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2283?filter=allopenissues] > Log4j 2.10+not working with SLF4J 1.8 in OSGI environment > affectsVersions:2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2305?filter=allopenissues] > fix the CacheEntry map in ThrowableProxy#toExtendedStackTrace to be put and > gotten with same key > affectsVersions:2.6.2;2.7;2.8;2.8.1;2.8.2;2.9.0;2.9.1;2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2389?filter=allopenissues] > NullPointerException when closing never used RollingRandomAccessFileAppender > affectsVersions:2.10.0;2.11.1 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2418?filter=allopenissues] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
[ https://issues.apache.org/jira/browse/HIVE-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942151#comment-16942151 ] Jesus Camacho Rodriguez commented on HIVE-22252: +1 (pending tests) > Fix caught NullPointerExceptions generated during EXPLAIN > - > > Key: HIVE-22252 > URL: https://issues.apache.org/jira/browse/HIVE-22252 > Project: Hive > Issue Type: Bug >Reporter: John Sherman >Assignee: John Sherman >Priority: Minor > Attachments: HIVE-22252.1.patch > > > While debugging an issue I noticed that during EXPLAIN the following methods > throw a NullPointerException: > VectorColumnOutputMapping::finalize > AbstractOperatorDesc::getUserLevelStatistics > AbstractOperatorDesc::getColumnExprMapForExplain > The exceptions do end up getting caught but we should add null checks and > gracefully to be less wasteful and to aid future debugging. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22278) Upgrade log4j to 2.12.1
[ https://issues.apache.org/jira/browse/HIVE-22278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-22278: -- Labels: pull-request-available (was: ) > Upgrade log4j to 2.12.1 > --- > > Key: HIVE-22278 > URL: https://issues.apache.org/jira/browse/HIVE-22278 > Project: Hive > Issue Type: Improvement >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > > Hive's currently using log4j 2.10.0 and according to HIVE-21273, a number of > issues are present in it, which can be resolved by upgrading to 2.12.1: > Curly braces in parameters are treated as placeholders > affectsVersions:2.8.2;2.9.0;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2032?filter=allopenissues] > Remove Log4J API dependency on Management APIs > affectsVersions:2.9.1;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2126?filter=allopenissues] > Log4j2 throws NoClassDefFoundError in Java 9 > affectsVersions:2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2129?filter=allopenissues] > ThreadContext map is cleared => entries are only available for one log event > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2158?filter=allopenissues] > Objects held in SortedArrayStringMap cannot be filtered during serialization > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2163?filter=allopenissues] > NullPointerException at > org.apache.logging.log4j.util.Activator.loadProvider(Activator.java:81) in > log4j 2.10.0 > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2182?filter=allopenissues] > MarkerFilter onMismatch invalid attribute in .properties > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2202?filter=allopenissues] > Configuration builder classes should look for "onMismatch"; not "onMisMatch". > > affectsVersions:2.4;2.4.1;2.5;2.6;2.6.1;2.6.2;2.7;2.8;2.8.1;2.8.2;2.9.0;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2219?filter=allopenissues] > Empty Automatic-Module-Name Header > affectsVersions:2.10.0;2.11.0;3.0.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2254?filter=allopenissues] > ConcurrentModificationException from > org.apache.logging.log4j.status.StatusLogger.(StatusLogger.java:71) > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2276?filter=allopenissues] > Allow SystemPropertiesPropertySource to run with a SecurityManager that > rejects system property access > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2279?filter=allopenissues] > ParserConfigurationException when using Log4j with > oracle.xml.jaxp.JXDocumentBuilderFactory > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2283?filter=allopenissues] > Log4j 2.10+not working with SLF4J 1.8 in OSGI environment > affectsVersions:2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2305?filter=allopenissues] > fix the CacheEntry map in ThrowableProxy#toExtendedStackTrace to be put and > gotten with same key > affectsVersions:2.6.2;2.7;2.8;2.8.1;2.8.2;2.9.0;2.9.1;2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2389?filter=allopenissues] > NullPointerException when closing never used RollingRandomAccessFileAppender > affectsVersions:2.10.0;2.11.1 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2418?filter=allopenissues] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22278) Upgrade log4j to 2.12.1
[ https://issues.apache.org/jira/browse/HIVE-22278?focusedWorklogId=321368=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321368 ] ASF GitHub Bot logged work on HIVE-22278: - Author: ASF GitHub Bot Created on: 01/Oct/19 16:58 Start Date: 01/Oct/19 16:58 Worklog Time Spent: 10m Work Description: dlavati commented on pull request #798: HIVE-22278: Upgrade log4j to 2.12.1 URL: https://github.com/apache/hive/pull/798 Change-Id: Ieced853bd4b2dd3e46291abec17bb73cba0536f4 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321368) Remaining Estimate: 0h Time Spent: 10m > Upgrade log4j to 2.12.1 > --- > > Key: HIVE-22278 > URL: https://issues.apache.org/jira/browse/HIVE-22278 > Project: Hive > Issue Type: Improvement >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Hive's currently using log4j 2.10.0 and according to HIVE-21273, a number of > issues are present in it, which can be resolved by upgrading to 2.12.1: > Curly braces in parameters are treated as placeholders > affectsVersions:2.8.2;2.9.0;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2032?filter=allopenissues] > Remove Log4J API dependency on Management APIs > affectsVersions:2.9.1;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2126?filter=allopenissues] > Log4j2 throws NoClassDefFoundError in Java 9 > affectsVersions:2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2129?filter=allopenissues] > ThreadContext map is cleared => entries are only available for one log event > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2158?filter=allopenissues] > Objects held in SortedArrayStringMap cannot be filtered during serialization > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2163?filter=allopenissues] > NullPointerException at > org.apache.logging.log4j.util.Activator.loadProvider(Activator.java:81) in > log4j 2.10.0 > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2182?filter=allopenissues] > MarkerFilter onMismatch invalid attribute in .properties > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2202?filter=allopenissues] > Configuration builder classes should look for "onMismatch"; not "onMisMatch". > > affectsVersions:2.4;2.4.1;2.5;2.6;2.6.1;2.6.2;2.7;2.8;2.8.1;2.8.2;2.9.0;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2219?filter=allopenissues] > Empty Automatic-Module-Name Header > affectsVersions:2.10.0;2.11.0;3.0.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2254?filter=allopenissues] > ConcurrentModificationException from > org.apache.logging.log4j.status.StatusLogger.(StatusLogger.java:71) > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2276?filter=allopenissues] > Allow SystemPropertiesPropertySource to run with a SecurityManager that > rejects system property access > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2279?filter=allopenissues] > ParserConfigurationException when using Log4j with > oracle.xml.jaxp.JXDocumentBuilderFactory > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2283?filter=allopenissues] > Log4j 2.10+not working with SLF4J 1.8 in OSGI environment > affectsVersions:2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2305?filter=allopenissues] > fix the CacheEntry map in ThrowableProxy#toExtendedStackTrace to be put and > gotten with same key > affectsVersions:2.6.2;2.7;2.8;2.8.1;2.8.2;2.9.0;2.9.1;2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2389?filter=allopenissues] > NullPointerException when closing never used RollingRandomAccessFileAppender > affectsVersions:2.10.0;2.11.1 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2418?filter=allopenissues] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22137) Implement alter/rename partition related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942143#comment-16942143
]
Hive QA commented on HIVE-22137:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
52s{color} | {color:blue} standalone-metastore/metastore-common in master has
32 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
170 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
22s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 4 unchanged - 2 fixed = 4 total (was 6) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18812/dev-support/hive-personality.sh
|
| git revision | master / 8111ede |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18812/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement alter/rename partition related methods on temporary tables
>
>
> Key: HIVE-22137
> URL: https://issues.apache.org/jira/browse/HIVE-22137
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22137.01.patch, HIVE-22137.02.patch,
> HIVE-22137.03.patch, HIVE-22137.04.patch,
[jira] [Assigned] (HIVE-22278) Upgrade log4j to 2.12.1
[ https://issues.apache.org/jira/browse/HIVE-22278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati reassigned HIVE-22278: --- > Upgrade log4j to 2.12.1 > --- > > Key: HIVE-22278 > URL: https://issues.apache.org/jira/browse/HIVE-22278 > Project: Hive > Issue Type: Improvement >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > > Hive's currently using log4j 2.10.0 and according to HIVE-21273, a number of > issues are present in it, which can be resolved by upgrading to 2.12.1: > Curly braces in parameters are treated as placeholders > affectsVersions:2.8.2;2.9.0;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2032?filter=allopenissues] > Remove Log4J API dependency on Management APIs > affectsVersions:2.9.1;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2126?filter=allopenissues] > Log4j2 throws NoClassDefFoundError in Java 9 > affectsVersions:2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2129?filter=allopenissues] > ThreadContext map is cleared => entries are only available for one log event > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2158?filter=allopenissues] > Objects held in SortedArrayStringMap cannot be filtered during serialization > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2163?filter=allopenissues] > NullPointerException at > org.apache.logging.log4j.util.Activator.loadProvider(Activator.java:81) in > log4j 2.10.0 > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2182?filter=allopenissues] > MarkerFilter onMismatch invalid attribute in .properties > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2202?filter=allopenissues] > Configuration builder classes should look for "onMismatch"; not "onMisMatch". > > affectsVersions:2.4;2.4.1;2.5;2.6;2.6.1;2.6.2;2.7;2.8;2.8.1;2.8.2;2.9.0;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2219?filter=allopenissues] > Empty Automatic-Module-Name Header > affectsVersions:2.10.0;2.11.0;3.0.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2254?filter=allopenissues] > ConcurrentModificationException from > org.apache.logging.log4j.status.StatusLogger.(StatusLogger.java:71) > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2276?filter=allopenissues] > Allow SystemPropertiesPropertySource to run with a SecurityManager that > rejects system property access > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2279?filter=allopenissues] > ParserConfigurationException when using Log4j with > oracle.xml.jaxp.JXDocumentBuilderFactory > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2283?filter=allopenissues] > Log4j 2.10+not working with SLF4J 1.8 in OSGI environment > affectsVersions:2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2305?filter=allopenissues] > fix the CacheEntry map in ThrowableProxy#toExtendedStackTrace to be put and > gotten with same key > affectsVersions:2.6.2;2.7;2.8;2.8.1;2.8.2;2.9.0;2.9.1;2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2389?filter=allopenissues] > NullPointerException when closing never used RollingRandomAccessFileAppender > affectsVersions:2.10.0;2.11.1 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2418?filter=allopenissues] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22267) Support password based authentication in HMS
[ https://issues.apache.org/jira/browse/HIVE-22267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-22267: -- Status: Open (was: Patch Available) > Support password based authentication in HMS > > > Key: HIVE-22267 > URL: https://issues.apache.org/jira/browse/HIVE-22267 > Project: Hive > Issue Type: New Feature >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22267.00.patch, HIVE-22267.01.patch, > HIVE-22267.02.patch, HIVE-22267.03.patch, HIVE-22267.04.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Similar to HS2, support password based authentication in HMS. > Right now we provide LDAP and CONFIG based options. The later allows to set > user and password in config and is used only for testing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22267) Support password based authentication in HMS
[ https://issues.apache.org/jira/browse/HIVE-22267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-22267: -- Attachment: HIVE-22267.04.patch Status: Patch Available (was: Open) > Support password based authentication in HMS > > > Key: HIVE-22267 > URL: https://issues.apache.org/jira/browse/HIVE-22267 > Project: Hive > Issue Type: New Feature >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22267.00.patch, HIVE-22267.01.patch, > HIVE-22267.02.patch, HIVE-22267.03.patch, HIVE-22267.04.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Similar to HS2, support password based authentication in HMS. > Right now we provide LDAP and CONFIG based options. The later allows to set > user and password in config and is used only for testing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22097) Incompatible java.util.ArrayList for java 11
[
https://issues.apache.org/jira/browse/HIVE-22097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942106#comment-16942106
]
Hive QA commented on HIVE-22097:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981869/HIVE-22097.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17014 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=363)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18811/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18811/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18811/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981869 - PreCommit-HIVE-Build
> Incompatible java.util.ArrayList for java 11
>
>
> Key: HIVE-22097
> URL: https://issues.apache.org/jira/browse/HIVE-22097
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0, 3.1.1
>Reporter: Yuming Wang
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22097.1.patch, JDK1.8.png, JDK11.png
>
>
> {noformat}
> export JAVA_HOME=/usr/lib/jdk-11.0.3
> export PATH=${JAVA_HOME}/bin:${PATH}
> hive> create table t(id int);
> Time taken: 0.035 seconds
> hive> insert into t values(1);
> Query ID = root_20190811155400_7c0e0494-eecb-4c54-a9fd-942ab52a0794
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> java.lang.RuntimeException: java.lang.NoSuchFieldException: parentOffset
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:390)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$1.create(SerializationUtilities.java:235)
> at
> org.apache.hive.com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.borrowKryo(SerializationUtilities.java:280)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:595)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapWork(Utilities.java:587)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:579)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:357)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:159)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2317)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1969)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1636)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1396)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1390)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:189)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:838)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:777)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:696)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
[jira] [Commented] (HIVE-22097) Incompatible java.util.ArrayList for java 11
[
https://issues.apache.org/jira/browse/HIVE-22097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942099#comment-16942099
]
Prasanth Jayachandran commented on HIVE-22097:
--
lgtm, +1
> Incompatible java.util.ArrayList for java 11
>
>
> Key: HIVE-22097
> URL: https://issues.apache.org/jira/browse/HIVE-22097
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0, 3.1.1
>Reporter: Yuming Wang
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22097.1.patch, JDK1.8.png, JDK11.png
>
>
> {noformat}
> export JAVA_HOME=/usr/lib/jdk-11.0.3
> export PATH=${JAVA_HOME}/bin:${PATH}
> hive> create table t(id int);
> Time taken: 0.035 seconds
> hive> insert into t values(1);
> Query ID = root_20190811155400_7c0e0494-eecb-4c54-a9fd-942ab52a0794
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> java.lang.RuntimeException: java.lang.NoSuchFieldException: parentOffset
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:390)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$1.create(SerializationUtilities.java:235)
> at
> org.apache.hive.com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.borrowKryo(SerializationUtilities.java:280)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:595)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapWork(Utilities.java:587)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:579)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:357)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:159)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2317)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1969)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1636)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1396)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1390)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:189)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:838)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:777)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:696)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.NoSuchFieldException: parentOffset
> at java.base/java.lang.Class.getDeclaredField(Class.java:2412)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:384)
> ... 29 more
> Job Submission failed with exception
> 'java.lang.RuntimeException(java.lang.NoSuchFieldException: parentOffset)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask. java.lang.NoSuchFieldException:
> parentOffset
> {noformat}
> The reason is Java removed {{parentOffset}}:
> !JDK1.8.png!
> !JDK11.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22097) Incompatible java.util.ArrayList for java 11
[
https://issues.apache.org/jira/browse/HIVE-22097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942058#comment-16942058
]
Hive QA commented on HIVE-22097:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
20s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 25 new + 13 unchanged - 81
fixed = 38 total (was 94) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18811/dev-support/hive-personality.sh
|
| git revision | master / 8111ede |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18811/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18811/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Incompatible java.util.ArrayList for java 11
>
>
> Key: HIVE-22097
> URL: https://issues.apache.org/jira/browse/HIVE-22097
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0, 3.1.1
>Reporter: Yuming Wang
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22097.1.patch, JDK1.8.png, JDK11.png
>
>
> {noformat}
> export JAVA_HOME=/usr/lib/jdk-11.0.3
> export PATH=${JAVA_HOME}/bin:${PATH}
> hive> create table t(id int);
> Time taken: 0.035 seconds
> hive> insert into t values(1);
> Query ID = root_20190811155400_7c0e0494-eecb-4c54-a9fd-942ab52a0794
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> java.lang.RuntimeException: java.lang.NoSuchFieldException: parentOffset
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:390)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$1.create(SerializationUtilities.java:235)
> at
> org.apache.hive.com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
>
[jira] [Commented] (HIVE-22276) Break up DDLSemanticAnalyzer - extract View related analyzers
[
https://issues.apache.org/jira/browse/HIVE-22276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942018#comment-16942018
]
Hive QA commented on HIVE-22276:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981867/HIVE-22276.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 17014 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_cluster]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_distribute_sort]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partition_cluster]
(batchId=184)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_transactional_rewrite_2]
(batchId=102)
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=363)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18810/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18810/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18810/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981867 - PreCommit-HIVE-Build
> Break up DDLSemanticAnalyzer - extract View related analyzers
> -
>
> Key: HIVE-22276
> URL: https://issues.apache.org/jira/browse/HIVE-22276
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Attachments: HIVE-22276.01.patch
>
>
> DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is
> to refactor it in order to have everything cut into more handleable classes
> under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each analyzers
> * have a package for each operation, containing an analyzer, a description,
> and an operation, so the amount of classes under a package is more manageable
> Step #6: extract all the vire related analyzers from DDLSemanticAnalyzer, and
> move them under the new package.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21146) Enforce TransactionBatch size=1 for blob stores
[
https://issues.apache.org/jira/browse/HIVE-21146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Lavati updated HIVE-21146:
Attachment: HIVE-21146.patch
Status: Patch Available (was: In Progress)
I'm not sure of the ideal way to validate the outstream, or whether I'm using a
proper path/convention by creating a temp file under the given location.
> Enforce TransactionBatch size=1 for blob stores
> ---
>
> Key: HIVE-21146
> URL: https://issues.apache.org/jira/browse/HIVE-21146
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21146.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Streaming Ingest API supports a concept of {{TransactionBatch}} where N
> transactions can be opened at once and the data in all of them will be
> written to the same delta_x_y directory where each transaction in the batch
> can be committed/aborted independently. The implementation relies on
> {{FSDataOutputStream.hflush()}} (called from OrcRecordUpdater}} which is
> available on HDFS but is often implemented as no-op in Blob store backed
> {{FileSystem}} objects.
> Need to add a check to {{HiveStreamingConnection()}} constructor to raise an
> error if {{builder.transactionBatchSize > 1}} and the target table/partitions
> are backed by something that doesn't support {{hflush()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21146) Enforce TransactionBatch size=1 for blob stores
[
https://issues.apache.org/jira/browse/HIVE-21146?focusedWorklogId=321282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321282
]
ASF GitHub Bot logged work on HIVE-21146:
-
Author: ASF GitHub Bot
Created on: 01/Oct/19 14:06
Start Date: 01/Oct/19 14:06
Worklog Time Spent: 10m
Work Description: dlavati commented on pull request #797: HIVE-21146
Enforce TransactionBatch size=1 for blob stores
URL: https://github.com/apache/hive/pull/797
Change-Id: Ia5f94c34a044c2990e95204de03b661d162874c7
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 321282)
Remaining Estimate: 0h
Time Spent: 10m
> Enforce TransactionBatch size=1 for blob stores
> ---
>
> Key: HIVE-21146
> URL: https://issues.apache.org/jira/browse/HIVE-21146
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Streaming Ingest API supports a concept of {{TransactionBatch}} where N
> transactions can be opened at once and the data in all of them will be
> written to the same delta_x_y directory where each transaction in the batch
> can be committed/aborted independently. The implementation relies on
> {{FSDataOutputStream.hflush()}} (called from OrcRecordUpdater}} which is
> available on HDFS but is often implemented as no-op in Blob store backed
> {{FileSystem}} objects.
> Need to add a check to {{HiveStreamingConnection()}} constructor to raise an
> error if {{builder.transactionBatchSize > 1}} and the target table/partitions
> are backed by something that doesn't support {{hflush()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21146) Enforce TransactionBatch size=1 for blob stores
[
https://issues.apache.org/jira/browse/HIVE-21146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-21146:
--
Labels: pull-request-available (was: )
> Enforce TransactionBatch size=1 for blob stores
> ---
>
> Key: HIVE-21146
> URL: https://issues.apache.org/jira/browse/HIVE-21146
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
>
> Streaming Ingest API supports a concept of {{TransactionBatch}} where N
> transactions can be opened at once and the data in all of them will be
> written to the same delta_x_y directory where each transaction in the batch
> can be committed/aborted independently. The implementation relies on
> {{FSDataOutputStream.hflush()}} (called from OrcRecordUpdater}} which is
> available on HDFS but is often implemented as no-op in Blob store backed
> {{FileSystem}} objects.
> Need to add a check to {{HiveStreamingConnection()}} constructor to raise an
> error if {{builder.transactionBatchSize > 1}} and the target table/partitions
> are backed by something that doesn't support {{hflush()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22276) Break up DDLSemanticAnalyzer - extract View related analyzers
[
https://issues.apache.org/jira/browse/HIVE-22276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941865#comment-16941865
]
Hive QA commented on HIVE-22276:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 9 new + 727 unchanged - 13
fixed = 736 total (was 740) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18810/dev-support/hive-personality.sh
|
| git revision | master / 8111ede |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18810/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18810/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18810/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Break up DDLSemanticAnalyzer - extract View related analyzers
> -
>
> Key: HIVE-22276
> URL: https://issues.apache.org/jira/browse/HIVE-22276
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Attachments: HIVE-22276.01.patch
>
>
> DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is
> to refactor it in order to have everything cut into more handleable classes
> under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each analyzers
> * have a package for each operation, containing an analyzer, a description,
> and an operation, so the amount of classes under a package is more manageable
> Step #6: extract all the vire related analyzers from DDLSemanticAnalyzer, and
> move them under the new package.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22208) Column name with reserved keyword is unescaped when query including join on table with mask column is re-written
[
https://issues.apache.org/jira/browse/HIVE-22208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-22208:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
pushed to master. Thank you [~zmatyus]!
> Column name with reserved keyword is unescaped when query including join on
> table with mask column is re-written
>
>
> Key: HIVE-22208
> URL: https://issues.apache.org/jira/browse/HIVE-22208
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Riju Trivedi
>Assignee: Zoltan Matyus
>Priority: Critical
> Fix For: 4.0.0
>
> Attachments: HIVE-22208.01.patch, HIVE-22208.02.patch
>
>
> Join query involving table with mask column and other having reserved
> keyword as column name fails with SemanticException during parsing re-written
> query :
> Original Query :
> {code:java}
> select a.`date`, b.nm
> from sample_keyword a
> join sample_mask b
> on b.id = a.id;
> {code}
> Re-written Query :
>
> {code:java}
> select a.date, b.nm
> from sample_keyword a
> join (SELECT `id`, CAST(mask_hash(nm) AS string) AS `nm`,
> BLOCK__OFFSET__INSIDE__FILE, INPUT__FILE__NAME, ROW__ID FROM
> `default`.`sample_mask` )`b`
> on b.id = a.id;
> {code}
> Re-written query does not have escape quotes for date column which cause
> SemanticException while parsing :
> {code:java}
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near 'a' '.' 'date' in selection target
>
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(
> SemanticAnalyzer.java:12084)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(
> SemanticAnalyzer.java:12298)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(
> CalcitePlanner.java:360)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
> BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22245) Make qtest feature parser reuseable
[
https://issues.apache.org/jira/browse/HIVE-22245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-22245:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
pushed to master. Thank you Laszlo for reviewing the changes!
> Make qtest feature parser reuseable
> ---
>
> Key: HIVE-22245
> URL: https://issues.apache.org/jira/browse/HIVE-22245
> Project: Hive
> Issue Type: Improvement
> Components: Test
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22245.01.patch, HIVE-22245.02.patch,
> HIVE-22245.03.patch, HIVE-22245.03.patch, HIVE-22245.04.patch
>
>
> right now we have a parser for {{--! qt:dataset}} ; to enable further
> addition of things (I would like to run scheduled query service for some
> qtests )
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21449) implement 'WITHIN GROUP' clause
[ https://issues.apache.org/jira/browse/HIVE-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21449: -- Attachment: HIVE-21449.9.patch > implement 'WITHIN GROUP' clause > --- > > Key: HIVE-21449 > URL: https://issues.apache.org/jira/browse/HIVE-21449 > Project: Hive > Issue Type: Improvement > Components: Parser, UDF >Reporter: László Bodor >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21449.1.patch, HIVE-21449.2.patch, > HIVE-21449.3.patch, HIVE-21449.4.patch, HIVE-21449.5.patch, > HIVE-21449.5.patch, HIVE-21449.6.patch, HIVE-21449.6.patch, > HIVE-21449.6.patch, HIVE-21449.6.patch, HIVE-21449.7.patch, > HIVE-21449.7.patch, HIVE-21449.8.patch, HIVE-21449.9.patch, HIVE-21449.9.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21449) implement 'WITHIN GROUP' clause
[ https://issues.apache.org/jira/browse/HIVE-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21449: -- Status: Patch Available (was: Open) > implement 'WITHIN GROUP' clause > --- > > Key: HIVE-21449 > URL: https://issues.apache.org/jira/browse/HIVE-21449 > Project: Hive > Issue Type: Improvement > Components: Parser, UDF >Reporter: László Bodor >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21449.1.patch, HIVE-21449.2.patch, > HIVE-21449.3.patch, HIVE-21449.4.patch, HIVE-21449.5.patch, > HIVE-21449.5.patch, HIVE-21449.6.patch, HIVE-21449.6.patch, > HIVE-21449.6.patch, HIVE-21449.6.patch, HIVE-21449.7.patch, > HIVE-21449.7.patch, HIVE-21449.8.patch, HIVE-21449.9.patch, HIVE-21449.9.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21449) implement 'WITHIN GROUP' clause
[ https://issues.apache.org/jira/browse/HIVE-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21449: -- Status: Open (was: Patch Available) > implement 'WITHIN GROUP' clause > --- > > Key: HIVE-21449 > URL: https://issues.apache.org/jira/browse/HIVE-21449 > Project: Hive > Issue Type: Improvement > Components: Parser, UDF >Reporter: László Bodor >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21449.1.patch, HIVE-21449.2.patch, > HIVE-21449.3.patch, HIVE-21449.4.patch, HIVE-21449.5.patch, > HIVE-21449.5.patch, HIVE-21449.6.patch, HIVE-21449.6.patch, > HIVE-21449.6.patch, HIVE-21449.6.patch, HIVE-21449.7.patch, > HIVE-21449.7.patch, HIVE-21449.8.patch, HIVE-21449.9.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21449) implement 'WITHIN GROUP' clause
[
https://issues.apache.org/jira/browse/HIVE-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941814#comment-16941814
]
Hive QA commented on HIVE-21449:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981852/HIVE-21449.9.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 17017 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=363)
org.apache.hive.jdbc.TestJdbcWithMiniLlapRow.testComplexQuery (batchId=283)
org.apache.hive.jdbc.TestJdbcWithMiniLlapRow.testDataTypes (batchId=283)
org.apache.hive.jdbc.TestJdbcWithMiniLlapRow.testEscapedStrings (batchId=283)
org.apache.hive.jdbc.TestJdbcWithMiniLlapRow.testKillQuery (batchId=283)
org.apache.hive.jdbc.TestJdbcWithMiniLlapRow.testLlapInputFormatEndToEnd
(batchId=283)
org.apache.hive.jdbc.TestJdbcWithMiniLlapRow.testNonAsciiStrings (batchId=283)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18809/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18809/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18809/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981852 - PreCommit-HIVE-Build
> implement 'WITHIN GROUP' clause
> ---
>
> Key: HIVE-21449
> URL: https://issues.apache.org/jira/browse/HIVE-21449
> Project: Hive
> Issue Type: Improvement
> Components: Parser, UDF
>Reporter: László Bodor
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21449.1.patch, HIVE-21449.2.patch,
> HIVE-21449.3.patch, HIVE-21449.4.patch, HIVE-21449.5.patch,
> HIVE-21449.5.patch, HIVE-21449.6.patch, HIVE-21449.6.patch,
> HIVE-21449.6.patch, HIVE-21449.6.patch, HIVE-21449.7.patch,
> HIVE-21449.7.patch, HIVE-21449.8.patch, HIVE-21449.9.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-8838) Support Parquet through HCatalog
[ https://issues.apache.org/jira/browse/HIVE-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941805#comment-16941805 ] Mass Dosage commented on HIVE-8838: --- [~szita] - thanks, e've opened a new patch and are in the process of getting it merged at HIVE-22249 (also linked above). > Support Parquet through HCatalog > > > Key: HIVE-8838 > URL: https://issues.apache.org/jira/browse/HIVE-8838 > Project: Hive > Issue Type: New Feature >Reporter: Brock Noland >Assignee: Ádám Szita >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-8838.0.patch, HIVE-8838.1.patch, HIVE-8838.2.patch, > HIVE-8838.3.patch, HIVE-8838.4.patch > > > Similar to HIVE-8687 for Avro we need to fix Parquet with HCatalog. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22097) Incompatible java.util.ArrayList for java 11
[
https://issues.apache.org/jira/browse/HIVE-22097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941791#comment-16941791
]
László Bodor commented on HIVE-22097:
-
+1, pending tests
> Incompatible java.util.ArrayList for java 11
>
>
> Key: HIVE-22097
> URL: https://issues.apache.org/jira/browse/HIVE-22097
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0, 3.1.1
>Reporter: Yuming Wang
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22097.1.patch, JDK1.8.png, JDK11.png
>
>
> {noformat}
> export JAVA_HOME=/usr/lib/jdk-11.0.3
> export PATH=${JAVA_HOME}/bin:${PATH}
> hive> create table t(id int);
> Time taken: 0.035 seconds
> hive> insert into t values(1);
> Query ID = root_20190811155400_7c0e0494-eecb-4c54-a9fd-942ab52a0794
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> java.lang.RuntimeException: java.lang.NoSuchFieldException: parentOffset
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:390)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$1.create(SerializationUtilities.java:235)
> at
> org.apache.hive.com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.borrowKryo(SerializationUtilities.java:280)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:595)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapWork(Utilities.java:587)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:579)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:357)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:159)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2317)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1969)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1636)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1396)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1390)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:189)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:838)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:777)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:696)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.NoSuchFieldException: parentOffset
> at java.base/java.lang.Class.getDeclaredField(Class.java:2412)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:384)
> ... 29 more
> Job Submission failed with exception
> 'java.lang.RuntimeException(java.lang.NoSuchFieldException: parentOffset)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask. java.lang.NoSuchFieldException:
> parentOffset
> {noformat}
> The reason is Java removed {{parentOffset}}:
> !JDK1.8.png!
> !JDK11.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22273) Access check is failed when a temporary directory is removed
[
https://issues.apache.org/jira/browse/HIVE-22273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-22273:
--
Attachment: HIVE-22273.3.patch
> Access check is failed when a temporary directory is removed
>
>
> Key: HIVE-22273
> URL: https://issues.apache.org/jira/browse/HIVE-22273
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22273.1.patch, HIVE-22273.2.patch,
> HIVE-22273.3.patch, HIVE-22273.patch
>
>
> The following exception is thrown if a temporary file is deleted during the
> access checks:
> {code}
> 2019-09-24T16:12:59,611 ERROR [7e491237-1505-4388-afb9-5ec2a688b0dc
> HiveServer2-HttpHandler-Pool: Thread-11941]: authorizer.RangerHiveAuthorizer
> (:()) - Error getting permissions for hdfs://HDFS_FOLDER/TABLE_NAME
> java.io.FileNotFoundException: File
> hdfs://HDFS_FOLDER/TABLE_NAME/.hive-staging_hive_2019-09-24_16-12-48_899_7291847300113791212-245/_tmp.-ext-10001
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1059)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1119)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1116)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1126)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hive.common.FileUtils.checkIsOwnerOfFileHierarchy(FileUtils.java:561)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.hadoop.hive.common.FileUtils.checkIsOwnerOfFileHierarchy(FileUtils.java:564)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.hadoop.hive.common.FileUtils.checkIsOwnerOfFileHierarchy(FileUtils.java:564)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at org.apache.hadoop.hive.common.FileUtils$4.run(FileUtils.java:540)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at org.apache.hadoop.hive.common.FileUtils$4.run(FileUtils.java:536)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_171]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_171]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> ~[hadoop-common-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hive.common.FileUtils.isOwnerOfFileHierarchy(FileUtils.java:536)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.hadoop.hive.common.FileUtils.isOwnerOfFileHierarchy(FileUtils.java:527)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.isURIAccessAllowed(RangerHiveAuthorizer.java:1420)
> ~[?:?]
> at
> org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.checkPrivileges(RangerHiveAuthorizer.java:287)
> ~[?:?]
> at org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:1336)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:1100)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:709)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1816)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1811)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
> ~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:262)
> ~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
>
[jira] [Commented] (HIVE-22249) Support Parquet through HCatalog
[ https://issues.apache.org/jira/browse/HIVE-22249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941777#comment-16941777 ] Peter Vary commented on HIVE-22249: --- +1 - will commit tomorrow (if I do not forget :D) > Support Parquet through HCatalog > > > Key: HIVE-22249 > URL: https://issues.apache.org/jira/browse/HIVE-22249 > Project: Hive > Issue Type: New Feature >Reporter: Jay Green-Stevens >Assignee: Jay Green-Stevens >Priority: Major > Fix For: 2.3.6 > > Attachments: HIVE-22249.2.branch-2.3.patch, > HIVE-22249.branch-2.3.patch > > > HIVE-8838 added Parquet support to HCatalog for Hive 3.0.0. We would like to > backport this functionality to Hive 2.x (primarily 2.3.x) for users who are > currently unable to migrate to Hive 3. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21449) implement 'WITHIN GROUP' clause
[
https://issues.apache.org/jira/browse/HIVE-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941775#comment-16941775
]
Hive QA commented on HIVE-21449:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
21s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} ql: The patch generated 0 new + 494 unchanged - 2
fixed = 494 total (was 496) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18809/dev-support/hive-personality.sh
|
| git revision | master / 8111ede |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18809/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> implement 'WITHIN GROUP' clause
> ---
>
> Key: HIVE-21449
> URL: https://issues.apache.org/jira/browse/HIVE-21449
> Project: Hive
> Issue Type: Improvement
> Components: Parser, UDF
>Reporter: László Bodor
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21449.1.patch, HIVE-21449.2.patch,
> HIVE-21449.3.patch, HIVE-21449.4.patch, HIVE-21449.5.patch,
> HIVE-21449.5.patch, HIVE-21449.6.patch, HIVE-21449.6.patch,
> HIVE-21449.6.patch, HIVE-21449.6.patch, HIVE-21449.7.patch,
> HIVE-21449.7.patch, HIVE-21449.8.patch, HIVE-21449.9.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22273) Access check is failed when a temporary directory is removed
[
https://issues.apache.org/jira/browse/HIVE-22273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941763#comment-16941763
]
Hive QA commented on HIVE-22273:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981851/HIVE-22273.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17016 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=363)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18808/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18808/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18808/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981851 - PreCommit-HIVE-Build
> Access check is failed when a temporary directory is removed
>
>
> Key: HIVE-22273
> URL: https://issues.apache.org/jira/browse/HIVE-22273
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22273.1.patch, HIVE-22273.2.patch, HIVE-22273.patch
>
>
> The following exception is thrown if a temporary file is deleted during the
> access checks:
> {code}
> 2019-09-24T16:12:59,611 ERROR [7e491237-1505-4388-afb9-5ec2a688b0dc
> HiveServer2-HttpHandler-Pool: Thread-11941]: authorizer.RangerHiveAuthorizer
> (:()) - Error getting permissions for hdfs://HDFS_FOLDER/TABLE_NAME
> java.io.FileNotFoundException: File
> hdfs://HDFS_FOLDER/TABLE_NAME/.hive-staging_hive_2019-09-24_16-12-48_899_7291847300113791212-245/_tmp.-ext-10001
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1059)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1119)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1116)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1126)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hive.common.FileUtils.checkIsOwnerOfFileHierarchy(FileUtils.java:561)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.hadoop.hive.common.FileUtils.checkIsOwnerOfFileHierarchy(FileUtils.java:564)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.hadoop.hive.common.FileUtils.checkIsOwnerOfFileHierarchy(FileUtils.java:564)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at org.apache.hadoop.hive.common.FileUtils$4.run(FileUtils.java:540)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at org.apache.hadoop.hive.common.FileUtils$4.run(FileUtils.java:536)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_171]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_171]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> ~[hadoop-common-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hive.common.FileUtils.isOwnerOfFileHierarchy(FileUtils.java:536)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.hadoop.hive.common.FileUtils.isOwnerOfFileHierarchy(FileUtils.java:527)
> ~[hive-common-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
> org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.isURIAccessAllowed(RangerHiveAuthorizer.java:1420)
> ~[?:?]
> at
> org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.checkPrivileges(RangerHiveAuthorizer.java:287)
> ~[?:?]
> at org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:1336)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> at
[jira] [Commented] (HIVE-21987) Hive is unable to read Parquet int32 annotated with decimal
[
https://issues.apache.org/jira/browse/HIVE-21987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941756#comment-16941756
]
Marta Kuczora commented on HIVE-21987:
--
Pushed to master. Thanks a lot [~pvary] for the review.
> Hive is unable to read Parquet int32 annotated with decimal
> ---
>
> Key: HIVE-21987
> URL: https://issues.apache.org/jira/browse/HIVE-21987
> Project: Hive
> Issue Type: Improvement
>Reporter: Nándor Kollár
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21987.1.patch, HIVE-21987.2.patch,
> HIVE-21987.3.patch, HIVE-21987.4.patch, HIVE-21987.5.patch,
> part-0-e5287735-8dcf-4dda-9c6e-4d5c98dc15f2-c000.snappy.parquet
>
>
> When I tried to read a Parquet file from a Hive (with Tez execution engine)
> table with a small decimal column, I got the following exception:
> {code}
> Caused by: java.lang.UnsupportedOperationException:
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$1
> at
> org.apache.parquet.io.api.PrimitiveConverter.addInt(PrimitiveConverter.java:98)
> at
> org.apache.parquet.column.impl.ColumnReaderImpl$2$3.writeValue(ColumnReaderImpl.java:248)
> at
> org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:367)
> at
> org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:226)
> ... 28 more
> {code}
> Steps to reproduce:
> - Create a Hive table with a single decimal(4, 2) column
> - Create a Parquet file with int32 column annotated with decimal(4, 2)
> logical type, put it into the previously created table location (or use the
> attached parquet file, in this case the column should be named as 'd', to
> match the Hive schema with the Parquet schema in the file)
> - Execute a {{select *}} on this table
> Also, I'm afraid that similar problems can happen with int64 decimals too.
> [Parquet specification |
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md] allows
> both of these cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21987) Hive is unable to read Parquet int32 annotated with decimal
[
https://issues.apache.org/jira/browse/HIVE-21987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marta Kuczora updated HIVE-21987:
-
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Hive is unable to read Parquet int32 annotated with decimal
> ---
>
> Key: HIVE-21987
> URL: https://issues.apache.org/jira/browse/HIVE-21987
> Project: Hive
> Issue Type: Improvement
>Reporter: Nándor Kollár
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21987.1.patch, HIVE-21987.2.patch,
> HIVE-21987.3.patch, HIVE-21987.4.patch, HIVE-21987.5.patch,
> part-0-e5287735-8dcf-4dda-9c6e-4d5c98dc15f2-c000.snappy.parquet
>
>
> When I tried to read a Parquet file from a Hive (with Tez execution engine)
> table with a small decimal column, I got the following exception:
> {code}
> Caused by: java.lang.UnsupportedOperationException:
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$1
> at
> org.apache.parquet.io.api.PrimitiveConverter.addInt(PrimitiveConverter.java:98)
> at
> org.apache.parquet.column.impl.ColumnReaderImpl$2$3.writeValue(ColumnReaderImpl.java:248)
> at
> org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:367)
> at
> org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:226)
> ... 28 more
> {code}
> Steps to reproduce:
> - Create a Hive table with a single decimal(4, 2) column
> - Create a Parquet file with int32 column annotated with decimal(4, 2)
> logical type, put it into the previously created table location (or use the
> attached parquet file, in this case the column should be named as 'd', to
> match the Hive schema with the Parquet schema in the file)
> - Execute a {{select *}} on this table
> Also, I'm afraid that similar problems can happen with int64 decimals too.
> [Parquet specification |
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md] allows
> both of these cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21987) Hive is unable to read Parquet int32 annotated with decimal
[
https://issues.apache.org/jira/browse/HIVE-21987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941746#comment-16941746
]
Marta Kuczora commented on HIVE-21987:
--
Got +1 from [~pvary] on review board.
> Hive is unable to read Parquet int32 annotated with decimal
> ---
>
> Key: HIVE-21987
> URL: https://issues.apache.org/jira/browse/HIVE-21987
> Project: Hive
> Issue Type: Improvement
>Reporter: Nándor Kollár
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21987.1.patch, HIVE-21987.2.patch,
> HIVE-21987.3.patch, HIVE-21987.4.patch, HIVE-21987.5.patch,
> part-0-e5287735-8dcf-4dda-9c6e-4d5c98dc15f2-c000.snappy.parquet
>
>
> When I tried to read a Parquet file from a Hive (with Tez execution engine)
> table with a small decimal column, I got the following exception:
> {code}
> Caused by: java.lang.UnsupportedOperationException:
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$1
> at
> org.apache.parquet.io.api.PrimitiveConverter.addInt(PrimitiveConverter.java:98)
> at
> org.apache.parquet.column.impl.ColumnReaderImpl$2$3.writeValue(ColumnReaderImpl.java:248)
> at
> org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:367)
> at
> org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:226)
> ... 28 more
> {code}
> Steps to reproduce:
> - Create a Hive table with a single decimal(4, 2) column
> - Create a Parquet file with int32 column annotated with decimal(4, 2)
> logical type, put it into the previously created table location (or use the
> attached parquet file, in this case the column should be named as 'd', to
> match the Hive schema with the Parquet schema in the file)
> - Execute a {{select *}} on this table
> Also, I'm afraid that similar problems can happen with int64 decimals too.
> [Parquet specification |
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md] allows
> both of these cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22273) Access check is failed when a temporary directory is removed
[
https://issues.apache.org/jira/browse/HIVE-22273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941728#comment-16941728
]
Hive QA commented on HIVE-22273:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18808/dev-support/hive-personality.sh
|
| git revision | master / 3dc5926 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18808/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Access check is failed when a temporary directory is removed
>
>
> Key: HIVE-22273
> URL: https://issues.apache.org/jira/browse/HIVE-22273
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22273.1.patch, HIVE-22273.2.patch, HIVE-22273.patch
>
>
> The following exception is thrown if a temporary file is deleted during the
> access checks:
> {code}
> 2019-09-24T16:12:59,611 ERROR [7e491237-1505-4388-afb9-5ec2a688b0dc
> HiveServer2-HttpHandler-Pool: Thread-11941]: authorizer.RangerHiveAuthorizer
> (:()) - Error getting permissions for hdfs://HDFS_FOLDER/TABLE_NAME
> java.io.FileNotFoundException: File
> hdfs://HDFS_FOLDER/TABLE_NAME/.hive-staging_hive_2019-09-24_16-12-48_899_7291847300113791212-245/_tmp.-ext-10001
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1059)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1119)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1116)
> ~[hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.1.1.3.1.0.0-78.jar:?]
>
[jira] [Commented] (HIVE-22207) Tez: SplitGenerator throws NumberFormatException when "dfs.block.size" on cluster is "128m"
[
https://issues.apache.org/jira/browse/HIVE-22207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941718#comment-16941718
]
Hive QA commented on HIVE-22207:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981847/HIVE-22207.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17016 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18807/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18807/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18807/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981847 - PreCommit-HIVE-Build
> Tez: SplitGenerator throws NumberFormatException when "dfs.block.size" on
> cluster is "128m"
> ---
>
> Key: HIVE-22207
> URL: https://issues.apache.org/jira/browse/HIVE-22207
> Project: Hive
> Issue Type: Bug
> Components: Hive, Tez
>Affects Versions: 2.1.0, 3.1.0
>Reporter: Praneet Sharma
>Assignee: Naresh P R
>Priority: Major
> Attachments: HIVE-22207.1.patch, HIVE-22207.2.patch, HIVE-22207.patch
>
>
> When the value of "dfs.block.size" on the cluster is "128m" in place of
> actual bytes as Long, then Tez job fails with NumberFormatException:
> {code:java}
> java.lang.NumberFormatException: For input string: "128m" at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Long.parseLong(Long.java:589) at
> java.lang.Long.parseLong(Long.java:631) at
> org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1539) at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:194)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[
https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941682#comment-16941682
]
Peter Vary commented on HIVE-22255:
---
[~Rajkumar Singh]: "Compacting" old base files is very "cheap" compared to real
compaction, since we do not have to rewrite the data, we just have to delete
the directory. So I do not see any added value in allowing to define the
num_base_threashold (we still might want to be able to turn on/off this
feature). I would go for a change in Initiator where for this kind of directory
structure we should {{requestCleanUp}} instead of {{requestCompaction}}, but
still run as a {{MAJOR}} compaction. Also we can use the {{COMPACTION_QUEUE}}
table to store the data.
> Hive don't trigger Major Compaction automatically if table contains only base
> files
>
>
> Key: HIVE-22255
> URL: https://issues.apache.org/jira/browse/HIVE-22255
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 3.1.2
> Environment: Hive-3.1.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> user may run into the issue if the table consists of all base files but no
> delta, then the following condition will yield false and automatic major
> compaction will be skipped.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313]
>
> Steps to Reproduce:
> # create Acid table
> {code:java}
> // create table myacid(id int);
> {code}
> # Run multiple insert table
> {code:java}
> // insert overwrite table myacid values(1);insert overwrite table myacid
> values(2),(3),(4){code}
> # DFS ls output
> {code:java}
> // dfs -ls -R /warehouse/tablespace/managed/hive/myacid;
> ++
> | DFS Output |
> ++
> | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001 |
> | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001/_orc_acid_version |
> | -rw-rw+ 3 hive hadoop 610 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001/bucket_0 |
> | drwxrwx---+ - hive hadoop 0 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002 |
> | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002/_orc_acid_version |
> | -rw-rw+ 3 hive hadoop 633 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002/bucket_0 |
> ++{code}
>
> you will see that Major compaction will not be trigger until you run alter
> table compact MAJOR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22207) Tez: SplitGenerator throws NumberFormatException when "dfs.block.size" on cluster is "128m"
[
https://issues.apache.org/jira/browse/HIVE-22207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941679#comment-16941679
]
Hive QA commented on HIVE-22207:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18807/dev-support/hive-personality.sh
|
| git revision | master / 3dc5926 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18807/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Tez: SplitGenerator throws NumberFormatException when "dfs.block.size" on
> cluster is "128m"
> ---
>
> Key: HIVE-22207
> URL: https://issues.apache.org/jira/browse/HIVE-22207
> Project: Hive
> Issue Type: Bug
> Components: Hive, Tez
>Affects Versions: 2.1.0, 3.1.0
>Reporter: Praneet Sharma
>Assignee: Naresh P R
>Priority: Major
> Attachments: HIVE-22207.1.patch, HIVE-22207.2.patch, HIVE-22207.patch
>
>
> When the value of "dfs.block.size" on the cluster is "128m" in place of
> actual bytes as Long, then Tez job fails with NumberFormatException:
> {code:java}
> java.lang.NumberFormatException: For input string: "128m" at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Long.parseLong(Long.java:589) at
> java.lang.Long.parseLong(Long.java:631) at
> org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1539) at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:194)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22277) TestBuddyAllocator.testMTT[2] is flaky (direct=true, mmap=true)
[
https://issues.apache.org/jira/browse/HIVE-22277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941663#comment-16941663
]
László Bodor commented on HIVE-22277:
-
I'm not familiar with this test case, but we should choose a way to go:
1. @RunWith retry logic (best effort, doesn't solve the original problem) on
AllocatorOutOfMemoryException
2. review the resource needs for this job and change infra accordingly
3. disable test
what do you think [~bslim]?
> TestBuddyAllocator.testMTT[2] is flaky (direct=true, mmap=true)
> ---
>
> Key: HIVE-22277
> URL: https://issues.apache.org/jira/browse/HIVE-22277
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Priority: Major
> Attachments: hive.log.tar.gz
>
>
> It's more than flaky. Recently, in most of the cases it's the only failing
> test in otherwise green runs.
> full hive logs: [^hive.log.tar.gz]
> {code}
> Error Message
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> Stacktrace
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at
> org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:145)
> ... 33 more
> Caused by:
>
[jira] [Commented] (HIVE-22277) TestBuddyAllocator.testMTT[2] is flaky (direct=true, mmap=true)
[
https://issues.apache.org/jira/browse/HIVE-22277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941657#comment-16941657
]
Zoltan Haindrich commented on HIVE-22277:
-
yes; this test have interfered with some of my runs as well lately; should we
disable it?
> TestBuddyAllocator.testMTT[2] is flaky (direct=true, mmap=true)
> ---
>
> Key: HIVE-22277
> URL: https://issues.apache.org/jira/browse/HIVE-22277
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Priority: Major
> Attachments: hive.log.tar.gz
>
>
> It's more than flaky. Recently, in most of the cases it's the only failing
> test in otherwise green runs.
> full hive logs: [^hive.log.tar.gz]
> {code}
> Error Message
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> Stacktrace
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at
> org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:145)
> ... 33 more
> Caused by:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at
>
[jira] [Commented] (HIVE-22275) OperationManager.queryIdOperation does not properly clean up multiple queryIds
[
https://issues.apache.org/jira/browse/HIVE-22275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941646#comment-16941646
]
Hive QA commented on HIVE-22275:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981839/HIVE-22275.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17016 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=363)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18806/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18806/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18806/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981839 - PreCommit-HIVE-Build
> OperationManager.queryIdOperation does not properly clean up multiple queryIds
> --
>
> Key: HIVE-22275
> URL: https://issues.apache.org/jira/browse/HIVE-22275
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-22275.1.patch
>
>
> In the case that multiple statements are run by a single Session before being
> cleaned up, it appears that OperationManager.queryIdOperation is not cleaned
> up properly.
> See the log statements below - with the exception of the first "Removed
> queryId:" log line, the queryId listed during cleanup is the same, when each
> of these handles should have their own queryId. Looks like only the last
> queryId executed is being cleaned up.
> As a result, HS2 can run out of memory as OperationManager.queryIdOperation
> grows and never cleans these queryIds/Operations up.
> {noformat}
> 2019-09-13T08:37:36,785 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=dfed4c18-a284-4640-9f4a-1a20527105f9]
> 2019-09-13T08:37:38,432 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083736_c49cf3cc-cfe8-48a1-bd22-8b924dfb0396
> corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=dfed4c18-a284-4640-9f4a-1a20527105f9] with tag: null
> 2019-09-13T08:37:38,469 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=24d0030c-0e49-45fb-a918-2276f0941cfb]
> 2019-09-13T08:37:52,662 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=b983802c-1dec-4fa0-8680-d05ab555321b]
> 2019-09-13T08:37:56,239 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=75dbc531-2964-47b2-84d7-85b59f88999c]
> 2019-09-13T08:38:02,551 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=72c79076-9d67-4894-a526-c233fa5450b2]
> 2019-09-13T08:38:10,558 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=17b30a62-612d-4b70-9ba7-4287d2d9229b]
> 2019-09-13T08:38:16,930 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=ea97e99d-cc77-470b-b49a-b869c73a4615]
> 2019-09-13T08:38:20,440 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=a277b789-ebb8-4925-878f-6728d3e8c5fb]
>
[jira] [Commented] (HIVE-8838) Support Parquet through HCatalog
[ https://issues.apache.org/jira/browse/HIVE-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941644#comment-16941644 ] Ádám Szita commented on HIVE-8838: -- Hi Jay, I guess you could open a new Jira especially if the change for pre Hive 3.0 largely differs from the patch committed here. If you do so, please make sure to link this Jira as related. > Support Parquet through HCatalog > > > Key: HIVE-8838 > URL: https://issues.apache.org/jira/browse/HIVE-8838 > Project: Hive > Issue Type: New Feature >Reporter: Brock Noland >Assignee: Ádám Szita >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-8838.0.patch, HIVE-8838.1.patch, HIVE-8838.2.patch, > HIVE-8838.3.patch, HIVE-8838.4.patch > > > Similar to HIVE-8687 for Avro we need to fix Parquet with HCatalog. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22249) Support Parquet through HCatalog
[ https://issues.apache.org/jira/browse/HIVE-22249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941635#comment-16941635 ] Jay Green-Stevens edited comment on HIVE-22249 at 10/1/19 8:51 AM: --- Thanks [~pvary]. I had a look and realised there was a problem in the patch so I uploaded the right version. It seems like that has done the job, but there are 8 tests failures listed in the log above which I am not having an issue with locally. The only one which fails for me is _org.apache.hive.spark.client.rpc.TestRpc_ in _spark-client_. Any thoughts on this? Thanks! was (Author: jay_grn): Thanks Peter. I had a look and realised there was a problem in the patch so I uploaded the right version. It seems like that has done the job, but there are 8 tests failures listed in the log above which I am not having an issue with locally. The only one which fails for me is _org.apache.hive.spark.client.rpc.TestRpc_ in _spark-client_. Any thoughts on this? Thanks! > Support Parquet through HCatalog > > > Key: HIVE-22249 > URL: https://issues.apache.org/jira/browse/HIVE-22249 > Project: Hive > Issue Type: New Feature >Reporter: Jay Green-Stevens >Assignee: Jay Green-Stevens >Priority: Major > Fix For: 2.3.6 > > Attachments: HIVE-22249.2.branch-2.3.patch, > HIVE-22249.branch-2.3.patch > > > HIVE-8838 added Parquet support to HCatalog for Hive 3.0.0. We would like to > backport this functionality to Hive 2.x (primarily 2.3.x) for users who are > currently unable to migrate to Hive 3. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22277) TestBuddyAllocator.testMTT[2] is flaky (direct=true, mmap=true)
[
https://issues.apache.org/jira/browse/HIVE-22277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22277:
Summary: TestBuddyAllocator.testMTT[2] is flaky (direct=true, mmap=true)
(was: TestBuddyAllocator.testMTT[2] is flaky (direct, mmap))
> TestBuddyAllocator.testMTT[2] is flaky (direct=true, mmap=true)
> ---
>
> Key: HIVE-22277
> URL: https://issues.apache.org/jira/browse/HIVE-22277
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Priority: Major
> Attachments: hive.log.tar.gz
>
>
> It's more than flaky. Recently, in most of the cases it's the only failing
> test in otherwise green runs.
> full hive logs: [^hive.log.tar.gz]
> {code}
> Error Message
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> Stacktrace
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at
> org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:145)
> ... 33 more
> Caused by:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at
>
[jira] [Updated] (HIVE-22277) TestBuddyAllocator.testMTT[2] is flaky (direct, mmap)
[
https://issues.apache.org/jira/browse/HIVE-22277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22277:
Summary: TestBuddyAllocator.testMTT[2] is flaky (direct, mmap) (was:
TestBuddyAllocator.testMTT[2] is flaky)
> TestBuddyAllocator.testMTT[2] is flaky (direct, mmap)
> -
>
> Key: HIVE-22277
> URL: https://issues.apache.org/jira/browse/HIVE-22277
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Priority: Major
> Attachments: hive.log.tar.gz
>
>
> It's more than flaky. Recently, in most of the cases it's the only failing
> test in otherwise green runs.
> full hive logs: [^hive.log.tar.gz]
> {code}
> Error Message
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> Stacktrace
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at
> org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:145)
> ... 33 more
> Caused by:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at
>
[jira] [Commented] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[
https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941641#comment-16941641
]
Peter Vary commented on HIVE-22255:
---
[~nareshpr]: Hive full ACID is working with snapshot isolation. Here is an
example:
* We have {{table1}} and it is compacted. The directory structure is this:
{code:java}
table1/
table1/base_01/
table1/base_01/bucket_0
{code}
* Query A is issued: {{select * from table1}}. The directory structure does
not change, and it will read {{table1/base_01/bucket_0}}
* Later Query B is issues: {{insert overwrite table1 ...}}. It will create the
following directory structure:
{code:java}
table1/
table1/base_01/
table1/base_01/bucket_0
table1/base_02/
table1/base_02/bucket_0
{code}
* Query B is finished. It will mark writeId 02 as a committed
* Query C is issued: {{select * from table1}}. The directory structure does
not change, and it will read {{table1/base_02/bucket_0}}
* Query A is still working, an it is still reading {{base_01}} so it can
not be removed
* Query A is finished. We can remove the {{base_01}} directory
This means that we can have two base directories at a given time, and we are
not allowed to remove them until the last transaction reading the is not
finished.
> Hive don't trigger Major Compaction automatically if table contains only base
> files
>
>
> Key: HIVE-22255
> URL: https://issues.apache.org/jira/browse/HIVE-22255
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 3.1.2
> Environment: Hive-3.1.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> user may run into the issue if the table consists of all base files but no
> delta, then the following condition will yield false and automatic major
> compaction will be skipped.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313]
>
> Steps to Reproduce:
> # create Acid table
> {code:java}
> // create table myacid(id int);
> {code}
> # Run multiple insert table
> {code:java}
> // insert overwrite table myacid values(1);insert overwrite table myacid
> values(2),(3),(4){code}
> # DFS ls output
> {code:java}
> // dfs -ls -R /warehouse/tablespace/managed/hive/myacid;
> ++
> | DFS Output |
> ++
> | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001 |
> | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001/_orc_acid_version |
> | -rw-rw+ 3 hive hadoop 610 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001/bucket_0 |
> | drwxrwx---+ - hive hadoop 0 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002 |
> | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002/_orc_acid_version |
> | -rw-rw+ 3 hive hadoop 633 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002/bucket_0 |
> ++{code}
>
> you will see that Major compaction will not be trigger until you run alter
> table compact MAJOR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22249) Support Parquet through HCatalog
[ https://issues.apache.org/jira/browse/HIVE-22249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941635#comment-16941635 ] Jay Green-Stevens commented on HIVE-22249: -- Thanks Peter. I had a look and realised there was a problem in the patch so I uploaded the right version. It seems like that has done the job, but there are 8 tests failures listed in the log above which I am not having an issue with locally. The only one which fails for me is _org.apache.hive.spark.client.rpc.TestRpc_ in _spark-client_. Any thoughts on this? Thanks! > Support Parquet through HCatalog > > > Key: HIVE-22249 > URL: https://issues.apache.org/jira/browse/HIVE-22249 > Project: Hive > Issue Type: New Feature >Reporter: Jay Green-Stevens >Assignee: Jay Green-Stevens >Priority: Major > Fix For: 2.3.6 > > Attachments: HIVE-22249.2.branch-2.3.patch, > HIVE-22249.branch-2.3.patch > > > HIVE-8838 added Parquet support to HCatalog for Hive 3.0.0. We would like to > backport this functionality to Hive 2.x (primarily 2.3.x) for users who are > currently unable to migrate to Hive 3. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22270) Upgrade commons-io to 2.6
[ https://issues.apache.org/jira/browse/HIVE-22270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-22270: Attachment: HIVE-22270.patch Status: Patch Available (was: Open) > Upgrade commons-io to 2.6 > - > > Key: HIVE-22270 > URL: https://issues.apache.org/jira/browse/HIVE-22270 > Project: Hive > Issue Type: Improvement >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22270.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Hive's currently using commons-io 2.4 and according to HIVE-21273, a number > of issues are present in it, which can be resolved by upgrading to 2.6: > IOUtils copyLarge() and skip() methods are performance hogs > affectsVersions:2.3;2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-355?filter=allopenissues] > CharSequenceInputStream#reset() behaves incorrectly in case when buffer size > is not dividable by data size > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-356?filter=allopenissues] > [Tailer] InterruptedException while the thead is sleeping is silently ignored > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-357?filter=allopenissues] > IOUtils.contentEquals* methods returns false if input1 == input2; should > return true > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-362?filter=allopenissues] > Apache Commons - standard links for documents are failing > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-369?filter=allopenissues] > FileUtils.sizeOfDirectoryAsBigInteger can overflow > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-390?filter=allopenissues] > Regression in FileUtils.readFileToString from 2.0.1 > affectsVersions:2.1;2.2;2.3;2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-453?filter=allopenissues] > Correct exception message in FileUtils.getFile(File; String...) > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-479?filter=allopenissues] > org.apache.commons.io.FileUtils#waitFor waits too long > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-481?filter=allopenissues] > FilenameUtils should handle embedded null bytes > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-484?filter=allopenissues] > Exceptions are suppressed incorrectly when copying files. > affectsVersions:2.4;2.5 > > [https://issues.apache.org/jira/projects/IO/issues/IO-502?filter=allopenissues] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22277) TestBuddyAllocator.testMTT[2] is flaky
[
https://issues.apache.org/jira/browse/HIVE-22277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22277:
Description:
It's more than flaky. Recently, in most of the cases it's the only failing test
in otherwise green runs.
full hive logs: [^hive.log.tar.gz]
{code}
Error Message
java.util.concurrent.ExecutionException:
org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
or an internal issue)
Stacktrace
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
or an internal issue)
at
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:149)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
Caused by: java.util.concurrent.ExecutionException:
org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
or an internal issue)
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:145)
... 33 more
Caused by:
org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
or an internal issue)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:454)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:299)
at
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.allocateAndUseBuffer(TestBuddyAllocator.java:254)
at
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.allocateUp(TestBuddyAllocator.java:231)
at
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.access$000(TestBuddyAllocator.java:43)
at
[jira] [Updated] (HIVE-22270) Upgrade commons-io to 2.6
[ https://issues.apache.org/jira/browse/HIVE-22270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-22270: -- Labels: pull-request-available (was: ) > Upgrade commons-io to 2.6 > - > > Key: HIVE-22270 > URL: https://issues.apache.org/jira/browse/HIVE-22270 > Project: Hive > Issue Type: Improvement >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > > Hive's currently using commons-io 2.4 and according to HIVE-21273, a number > of issues are present in it, which can be resolved by upgrading to 2.6: > IOUtils copyLarge() and skip() methods are performance hogs > affectsVersions:2.3;2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-355?filter=allopenissues] > CharSequenceInputStream#reset() behaves incorrectly in case when buffer size > is not dividable by data size > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-356?filter=allopenissues] > [Tailer] InterruptedException while the thead is sleeping is silently ignored > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-357?filter=allopenissues] > IOUtils.contentEquals* methods returns false if input1 == input2; should > return true > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-362?filter=allopenissues] > Apache Commons - standard links for documents are failing > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-369?filter=allopenissues] > FileUtils.sizeOfDirectoryAsBigInteger can overflow > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-390?filter=allopenissues] > Regression in FileUtils.readFileToString from 2.0.1 > affectsVersions:2.1;2.2;2.3;2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-453?filter=allopenissues] > Correct exception message in FileUtils.getFile(File; String...) > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-479?filter=allopenissues] > org.apache.commons.io.FileUtils#waitFor waits too long > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-481?filter=allopenissues] > FilenameUtils should handle embedded null bytes > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-484?filter=allopenissues] > Exceptions are suppressed incorrectly when copying files. > affectsVersions:2.4;2.5 > > [https://issues.apache.org/jira/projects/IO/issues/IO-502?filter=allopenissues] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22270) Upgrade commons-io to 2.6
[ https://issues.apache.org/jira/browse/HIVE-22270?focusedWorklogId=321100=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321100 ] ASF GitHub Bot logged work on HIVE-22270: - Author: ASF GitHub Bot Created on: 01/Oct/19 08:12 Start Date: 01/Oct/19 08:12 Worklog Time Spent: 10m Work Description: dlavati commented on pull request #796: HIVE-22270 Upgrade commons-io to 2.6 URL: https://github.com/apache/hive/pull/796 Change-Id: I7231875d99b65089ed71a1227d8dd88e5c287cb3 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321100) Remaining Estimate: 0h Time Spent: 10m > Upgrade commons-io to 2.6 > - > > Key: HIVE-22270 > URL: https://issues.apache.org/jira/browse/HIVE-22270 > Project: Hive > Issue Type: Improvement >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Hive's currently using commons-io 2.4 and according to HIVE-21273, a number > of issues are present in it, which can be resolved by upgrading to 2.6: > IOUtils copyLarge() and skip() methods are performance hogs > affectsVersions:2.3;2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-355?filter=allopenissues] > CharSequenceInputStream#reset() behaves incorrectly in case when buffer size > is not dividable by data size > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-356?filter=allopenissues] > [Tailer] InterruptedException while the thead is sleeping is silently ignored > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-357?filter=allopenissues] > IOUtils.contentEquals* methods returns false if input1 == input2; should > return true > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-362?filter=allopenissues] > Apache Commons - standard links for documents are failing > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-369?filter=allopenissues] > FileUtils.sizeOfDirectoryAsBigInteger can overflow > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-390?filter=allopenissues] > Regression in FileUtils.readFileToString from 2.0.1 > affectsVersions:2.1;2.2;2.3;2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-453?filter=allopenissues] > Correct exception message in FileUtils.getFile(File; String...) > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-479?filter=allopenissues] > org.apache.commons.io.FileUtils#waitFor waits too long > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-481?filter=allopenissues] > FilenameUtils should handle embedded null bytes > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-484?filter=allopenissues] > Exceptions are suppressed incorrectly when copying files. > affectsVersions:2.4;2.5 > > [https://issues.apache.org/jira/projects/IO/issues/IO-502?filter=allopenissues] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22277) TestBuddyAllocator.testMTT[2] is flaky
[
https://issues.apache.org/jira/browse/HIVE-22277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22277:
Attachment: hive.log.tar.gz
> TestBuddyAllocator.testMTT[2] is flaky
> --
>
> Key: HIVE-22277
> URL: https://issues.apache.org/jira/browse/HIVE-22277
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Priority: Major
> Attachments: hive.log.tar.gz
>
>
> It's more than flaky. Recently, in most of the cases it's the only failing
> test in otherwise green runs.
> {code}
> Error Message
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> Stacktrace
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at
> org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:145)
> ... 33 more
> Caused by:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
> or an internal issue)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:454)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:299)
>
[jira] [Updated] (HIVE-7145) Remove dependence on apache commons-lang
[ https://issues.apache.org/jira/browse/HIVE-7145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-7145: --- Attachment: HIVE-7145.patch Status: Patch Available (was: In Progress) > Remove dependence on apache commons-lang > > > Key: HIVE-7145 > URL: https://issues.apache.org/jira/browse/HIVE-7145 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Attachments: HIVE-7145.patch > > Time Spent: 10m > Remaining Estimate: 0h > > We currently depend on both Apache commons-lang and commons-lang3. They are > the same project, just at version 2.x vs 3.x. I propose that we move all of > the references in Hive to commons-lang3 and remove the v2 usage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-7145) Remove dependence on apache commons-lang
[ https://issues.apache.org/jira/browse/HIVE-7145?focusedWorklogId=321096=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321096 ] ASF GitHub Bot logged work on HIVE-7145: Author: ASF GitHub Bot Created on: 01/Oct/19 08:11 Start Date: 01/Oct/19 08:11 Worklog Time Spent: 10m Work Description: dlavati commented on pull request #795: HIVE-7145 Remove dependence on apache commons-lang URL: https://github.com/apache/hive/pull/795 Change-Id: I0a71c77d736860ad2dbb9a89b10d92745c7c0137 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321096) Remaining Estimate: 0h Time Spent: 10m > Remove dependence on apache commons-lang > > > Key: HIVE-7145 > URL: https://issues.apache.org/jira/browse/HIVE-7145 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Attachments: HIVE-7145.patch > > Time Spent: 10m > Remaining Estimate: 0h > > We currently depend on both Apache commons-lang and commons-lang3. They are > the same project, just at version 2.x vs 3.x. I propose that we move all of > the references in Hive to commons-lang3 and remove the v2 usage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-7145) Remove dependence on apache commons-lang
[ https://issues.apache.org/jira/browse/HIVE-7145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-7145: - Labels: pull-request-available (was: ) > Remove dependence on apache commons-lang > > > Key: HIVE-7145 > URL: https://issues.apache.org/jira/browse/HIVE-7145 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > > We currently depend on both Apache commons-lang and commons-lang3. They are > the same project, just at version 2.x vs 3.x. I propose that we move all of > the references in Hive to commons-lang3 and remove the v2 usage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?focusedWorklogId=321093=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321093 ] ASF GitHub Bot logged work on HIVE-21884: - Author: ASF GitHub Bot Created on: 01/Oct/19 07:59 Start Date: 01/Oct/19 07:59 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on pull request #794: HIVE-21884 URL: https://github.com/apache/hive/pull/794 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321093) Remaining Estimate: 0h Time Spent: 10m > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22277) TestBuddyAllocator.testMTT[2] is flaky
[
https://issues.apache.org/jira/browse/HIVE-22277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22277:
Description:
It's more than flaky. Recently, in most of the cases it's the only failing test
in otherwise green runs.
{code}
Error Message
java.util.concurrent.ExecutionException:
org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
or an internal issue)
Stacktrace
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
or an internal issue)
at
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:149)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
Caused by: java.util.concurrent.ExecutionException:
org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
or an internal issue)
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:145)
... 33 more
Caused by:
org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked,
or an internal issue)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:454)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:299)
at
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.allocateAndUseBuffer(TestBuddyAllocator.java:254)
at
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.allocateUp(TestBuddyAllocator.java:231)
at
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.access$000(TestBuddyAllocator.java:43)
at
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator$1.call(TestBuddyAllocator.java:119)
at
[jira] [Updated] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21884: -- Labels: pull-request-available (was: ) > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch, > HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch, > HIVE-21884.01.patch, HIVE-21884.02.patch, HIVE-21884.03.patch, > HIVE-21884.09.patch, HIVE-21884.10.patch, HIVE-21884.10.patch, > HIVE-21884.11.patch, HIVE-21884.12.patch, HIVE-21884.13.patch, > HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.14.patch, > HIVE-21884.16.patch, HIVE-21884.17.patch, HIVE-21884.18.patch, > HIVE-21884.20.patch, Scheduled queries2.pdf > > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22277) TestBuddyAllocator.testMTT[2] is flaky
[ https://issues.apache.org/jira/browse/HIVE-22277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22277: Description: It's more than flaky. Recently, in most of the cases it's the only failing test in otherwise green runs. > TestBuddyAllocator.testMTT[2] is flaky > -- > > Key: HIVE-22277 > URL: https://issues.apache.org/jira/browse/HIVE-22277 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Priority: Major > > It's more than flaky. Recently, in most of the cases it's the only failing > test in otherwise green runs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22097) Incompatible java.util.ArrayList for java 11
[
https://issues.apache.org/jira/browse/HIVE-22097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22097:
-
Attachment: (was: HIVE-22097.1.patch)
> Incompatible java.util.ArrayList for java 11
>
>
> Key: HIVE-22097
> URL: https://issues.apache.org/jira/browse/HIVE-22097
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0, 3.1.1
>Reporter: Yuming Wang
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22097.1.patch, JDK1.8.png, JDK11.png
>
>
> {noformat}
> export JAVA_HOME=/usr/lib/jdk-11.0.3
> export PATH=${JAVA_HOME}/bin:${PATH}
> hive> create table t(id int);
> Time taken: 0.035 seconds
> hive> insert into t values(1);
> Query ID = root_20190811155400_7c0e0494-eecb-4c54-a9fd-942ab52a0794
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> java.lang.RuntimeException: java.lang.NoSuchFieldException: parentOffset
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:390)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$1.create(SerializationUtilities.java:235)
> at
> org.apache.hive.com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.borrowKryo(SerializationUtilities.java:280)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:595)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapWork(Utilities.java:587)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:579)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:357)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:159)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2317)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1969)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1636)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1396)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1390)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:189)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:838)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:777)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:696)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.NoSuchFieldException: parentOffset
> at java.base/java.lang.Class.getDeclaredField(Class.java:2412)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:384)
> ... 29 more
> Job Submission failed with exception
> 'java.lang.RuntimeException(java.lang.NoSuchFieldException: parentOffset)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask. java.lang.NoSuchFieldException:
> parentOffset
> {noformat}
> The reason is Java removed {{parentOffset}}:
> !JDK1.8.png!
> !JDK11.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22097) Incompatible java.util.ArrayList for java 11
[
https://issues.apache.org/jira/browse/HIVE-22097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22097:
-
Attachment: HIVE-22097.1.patch
> Incompatible java.util.ArrayList for java 11
>
>
> Key: HIVE-22097
> URL: https://issues.apache.org/jira/browse/HIVE-22097
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0, 3.1.1
>Reporter: Yuming Wang
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22097.1.patch, JDK1.8.png, JDK11.png
>
>
> {noformat}
> export JAVA_HOME=/usr/lib/jdk-11.0.3
> export PATH=${JAVA_HOME}/bin:${PATH}
> hive> create table t(id int);
> Time taken: 0.035 seconds
> hive> insert into t values(1);
> Query ID = root_20190811155400_7c0e0494-eecb-4c54-a9fd-942ab52a0794
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> java.lang.RuntimeException: java.lang.NoSuchFieldException: parentOffset
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:390)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$1.create(SerializationUtilities.java:235)
> at
> org.apache.hive.com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.borrowKryo(SerializationUtilities.java:280)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:595)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapWork(Utilities.java:587)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:579)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:357)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:159)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2317)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1969)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1636)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1396)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1390)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:189)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:838)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:777)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:696)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.NoSuchFieldException: parentOffset
> at java.base/java.lang.Class.getDeclaredField(Class.java:2412)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:384)
> ... 29 more
> Job Submission failed with exception
> 'java.lang.RuntimeException(java.lang.NoSuchFieldException: parentOffset)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask. java.lang.NoSuchFieldException:
> parentOffset
> {noformat}
> The reason is Java removed {{parentOffset}}:
> !JDK1.8.png!
> !JDK11.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22097) Incompatible java.util.ArrayList for java 11
[
https://issues.apache.org/jira/browse/HIVE-22097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22097:
-
Status: Open (was: Patch Available)
> Incompatible java.util.ArrayList for java 11
>
>
> Key: HIVE-22097
> URL: https://issues.apache.org/jira/browse/HIVE-22097
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.1.1, 3.0.0
>Reporter: Yuming Wang
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22097.1.patch, JDK1.8.png, JDK11.png
>
>
> {noformat}
> export JAVA_HOME=/usr/lib/jdk-11.0.3
> export PATH=${JAVA_HOME}/bin:${PATH}
> hive> create table t(id int);
> Time taken: 0.035 seconds
> hive> insert into t values(1);
> Query ID = root_20190811155400_7c0e0494-eecb-4c54-a9fd-942ab52a0794
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> java.lang.RuntimeException: java.lang.NoSuchFieldException: parentOffset
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:390)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$1.create(SerializationUtilities.java:235)
> at
> org.apache.hive.com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.borrowKryo(SerializationUtilities.java:280)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:595)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapWork(Utilities.java:587)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:579)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:357)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:159)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2317)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1969)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1636)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1396)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1390)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:189)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:838)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:777)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:696)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.NoSuchFieldException: parentOffset
> at java.base/java.lang.Class.getDeclaredField(Class.java:2412)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:384)
> ... 29 more
> Job Submission failed with exception
> 'java.lang.RuntimeException(java.lang.NoSuchFieldException: parentOffset)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask. java.lang.NoSuchFieldException:
> parentOffset
> {noformat}
> The reason is Java removed {{parentOffset}}:
> !JDK1.8.png!
> !JDK11.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22097) Incompatible java.util.ArrayList for java 11
[
https://issues.apache.org/jira/browse/HIVE-22097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22097:
-
Status: Patch Available (was: Open)
> Incompatible java.util.ArrayList for java 11
>
>
> Key: HIVE-22097
> URL: https://issues.apache.org/jira/browse/HIVE-22097
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.1.1, 3.0.0
>Reporter: Yuming Wang
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22097.1.patch, JDK1.8.png, JDK11.png
>
>
> {noformat}
> export JAVA_HOME=/usr/lib/jdk-11.0.3
> export PATH=${JAVA_HOME}/bin:${PATH}
> hive> create table t(id int);
> Time taken: 0.035 seconds
> hive> insert into t values(1);
> Query ID = root_20190811155400_7c0e0494-eecb-4c54-a9fd-942ab52a0794
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> java.lang.RuntimeException: java.lang.NoSuchFieldException: parentOffset
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:390)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$1.create(SerializationUtilities.java:235)
> at
> org.apache.hive.com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.borrowKryo(SerializationUtilities.java:280)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:595)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapWork(Utilities.java:587)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:579)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:357)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:159)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2317)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1969)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1636)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1396)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1390)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:189)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:838)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:777)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:696)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.NoSuchFieldException: parentOffset
> at java.base/java.lang.Class.getDeclaredField(Class.java:2412)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:384)
> ... 29 more
> Job Submission failed with exception
> 'java.lang.RuntimeException(java.lang.NoSuchFieldException: parentOffset)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask. java.lang.NoSuchFieldException:
> parentOffset
> {noformat}
> The reason is Java removed {{parentOffset}}:
> !JDK1.8.png!
> !JDK11.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22275) OperationManager.queryIdOperation does not properly clean up multiple queryIds
[
https://issues.apache.org/jira/browse/HIVE-22275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941613#comment-16941613
]
Hive QA commented on HIVE-22275:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} service in master has 49 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} service: The patch generated 3 new + 25 unchanged - 0
fixed = 28 total (was 25) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18806/dev-support/hive-personality.sh
|
| git revision | master / 3dc5926 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18806/yetus/diff-checkstyle-service.txt
|
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18806/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> OperationManager.queryIdOperation does not properly clean up multiple queryIds
> --
>
> Key: HIVE-22275
> URL: https://issues.apache.org/jira/browse/HIVE-22275
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-22275.1.patch
>
>
> In the case that multiple statements are run by a single Session before being
> cleaned up, it appears that OperationManager.queryIdOperation is not cleaned
> up properly.
> See the log statements below - with the exception of the first "Removed
> queryId:" log line, the queryId listed during cleanup is the same, when each
> of these handles should have their own queryId. Looks like only the last
> queryId executed is being cleaned up.
> As a result, HS2 can run out of memory as OperationManager.queryIdOperation
> grows and never cleans these queryIds/Operations up.
> {noformat}
> 2019-09-13T08:37:36,785 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=dfed4c18-a284-4640-9f4a-1a20527105f9]
> 2019-09-13T08:37:38,432 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId:
[jira] [Updated] (HIVE-22276) Break up DDLSemanticAnalyzer - extract View related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22276: -- Attachment: HIVE-22276.01.patch > Break up DDLSemanticAnalyzer - extract View related analyzers > - > > Key: HIVE-22276 > URL: https://issues.apache.org/jira/browse/HIVE-22276 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22276.01.patch > > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #6: extract all the vire related analyzers from DDLSemanticAnalyzer, and > move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
